- Home
- Interventions
- Making the Decision
- Study Questions
-
Questions to Ask About Research Studies on Autism Interventions
This page provides a list of questions you may like to ask when you are trying to work out how scientifically rigorous a specific research study is.
The more times you can answer 'Yes', the more likely the study will be valid and reliable - although this is only a very rough rule of thumb.
Please note: Scientists never draw firm conclusions from just one study or set of results. They consider the contribution it makes in the context of other work and their own experience.
Peer review
- Has the study been reviewed by other scientists who are independent of the original research team?
(The peer review process helps to ensure that the information presented in a research study is scientifically rigorous because it has been checked by other scientists.)
The hypothesis (theory)
- Does the study set out a clear hypothesis and clear predictions of what should happen?
(A well-defined and specific research question is more likely to provide useful information than a question that is vague and poorly-defined.)
The researchers
- Are the researchers objective and unbiased?
(Some researchers may be setting out to prove that an intervention works, irrespective of what the data in their research shows. Other researchers may stand to gain financially if, for example, they are providers of the intervention.)
- Is the study being run by several different research teams at several different sites at the same time?
(A study run by different research teams on different sites is less likely to be biased than a study run by a single team on the same site.)
The participants
- Does the study include enough participants to make the results meaningful?
(If the study does not include enough participants the results may not be statistically significant or applicable to other people - Please see Note 1 at the end of this page.)
- Does the study record important information about the participants, such as the specific diagnosis, any co-occurring conditions, their ages, their intellectual ability, and so on?
(It is important to record important information about the participants as this could affect the results of the intervention. For example, some interventions may not be effective for particular groups of people.)
- Does the study include a control group? That is, a group of participants who do not receive the same treatment as the experimental group or who receive a different form of the same treatment?
(A control group is important because it can help to show that any changes in the experimental group are likely to be as a result of the intervention.)
- Does the study record important information about the control group, so that you can tell if they were similar or not to the experimental group?
(If the control group is not the same as the experimental group any differences in the results may not be the result of the treatment.)
- Were the participants randomised between the experimental group and the control group?
(Randomisation ensures that no one is able to influence who is put in which group and this reduces the risk of bias.)
The interventions
- Does the study provide enough detail about the intervention and how it was delivered - such as who delivered it, where and when, for how long and how often, and any specific techniques used?
(This makes it easier for other researchers to replicate the study and check whether the findings are accurate.)
- Does the study provide enough information about the interventions received by the control group - if there is one?
In some studies, the control group may be accessing other types of interventions, which may be having their own effects. Not reporting these makes it difficult to compare the effects of the intervention given to the treatment group compared to what the control group is receiving.
- Are the participants (and any parents/carers) blind to the intervention? That is, are they unaware of which intervention they received?
(Blinding ensures that the participants (and any parent/carers) do not behave in a particular way because they know which treatment they are receiving, which could alter the results of the study.)
- Are the assessors blind to the intervention, that is, are they unaware of which intervention the participants received?
(Blinding ensures that the assessors do not make judgements about the success of the intervention based on their beliefs about which participants are receiving which treatments.)
The measures
- Does the study use widely recognised and relevant outcome measures?
(Using widely recognised and relevant outcome measures ensures that the results are more likely to be valid and that they can be interpreted by other researchers.)
- Does the study use the same measures before and after the intervention?
(If the study uses different measures before and after it can be difficult to compare the effect of the intervention.)
Statistical analysis
- Does the study use statistical techniques that are appropriate to the design of the study?
(Using appropriate and widely-accepted statistical techniques means that any results reported in the study are more likely to be valid and reliable.)
Other questions
The following questions do not necessarily tell you how scientifically rigorous a specific research study is. But they are important questions you may want to ask anyway.
Ethical approval
- Did the study receive ethical approval from an appropriate agency, such as a university?
(Ethical approval safeguards the rights of participants to be treated as openly and fairly as possible within the research, and to consent fully to taking part.)
Inclusion
- Does the research team include any autistic people?
(Including autistic people on the research team, rather than just using them as study participants, means that the study is more likely to be of relevance to other autistic people.)
Social validity
- Is the research study socially valid, that is, is it likely to produce any benefits in the real world?
(Many research studies ask purely theoretical questions that have no practical application to people on the autism spectrum, their parents/ carers or service providers.)
Note 1
There is no general agreement on how many participants is 'enough' for a research study because it depends on so many other factors. So some researchers may believe that three participants is enough - provided they are part of a study that uses a reasonably robust design, such as a multiple baseline design. Other researchers may believe that a minimum of 60 participants is required - provided they are part of a study that uses a very robust design, such as a randomised controlled trial.
Further Information
These questions are taken from 'Choosing Autism Interventions: A Research-Based Guide' which was published in April 2015.
Related Pages
- Quick link:
- https://informationautism.org/study-questions
- Updated
- 16 Jun 2022