A recent Cochrane review assessed the efficacy of methylphenidate for attention-deficit/hyperactivity disorder (ADHD) in children and adolescents. Notwithstanding the moderate-to-large effect sizes for ADHD symptom reduction found in the meta-analysis, the authors concluded that the quality of the evidence is low and therefore the true magnitude of these effects remains uncertain. We identified a number of major concerns with the review, in the domains of study inclusion, approaches to quality assessment and interpretation of data relating to serious adverse events as well as of the clinical implications of the reported effects. We also found errors in the extraction of data used to estimate the effect size of the primary outcome. Considering all the shortcomings, the conclusion in the Cochrane review that the status of the evidence is uncertain is misplaced. Professionals, parents and patients should refer to previous reviews and existing guidelines, which include methylphenidate as one of the safe and efficacious treatment strategies for ADHD.
Attention-deficit/hyperactivity disorder (ADHD) is a common disorder starting in childhood and frequently persisting across the lifespan. Current treatment guidelines, including those of the National Institute for Health and Care Excellence (NICE) identify methylphenidate (MPH) as a first-line treatment for ADHD. Furthermore, previous systematic reviews and meta-analyses (e.g, refs. 2 3) have reported large effect sizes for the efficacy of MPH in the treatment of ADHD, at least in the short term.
In November 2015, Storebø et al4 published a Cochrane review on the efficacy and tolerability of MPH for the treatment of ADHD in children and adolescents. This review challenged the conclusions of previous reviews and guidelines. While the meta-analysis found similar effect sizes in relation to efficacy to those previously reported, the authors concluded that the magnitude of this effect is uncertain due to the very low quality of the evidence. As such, the conclusions of Storebø et al could raise questions about the role of MPH as a core component of ADHD treatment. As an international group committed to the provision of evidence-based clinical guidance on the management of ADHD, we do not agree with the conclusions of Storebø et al. Rather, we argue that the Storebø et al review is flawed in a number of ways that lead to these incorrect conclusions:
Additionally, there are a number of errors in the calculation of standar- dised mean differences (SMDs) and meta-analytic weights.
As per their protocol, Storebø et al aimed to include randomised controlled trials (RCTs) comparing MPH with placebo or no intervention, allowing cointerventions provided that the compared intervention groups received the cointervention similarly.
In fact, Storebø et al included three studies, among them the large Multimodal Treatment of ADHD (MTA) study, where there was no placebo/no treatment arm and/or where MPH was used as an add-on intervention and a study (included only in secondary analyses) that was not randomised. Removing these studies increases the effect size for the primary outcome of teacher-reported ADHD symptom ratings from −0.77 (95% CIs −0.90, −0.64) to −0.83 (95% CIs −0.96,−0.70). The inclusion of the large MTA study5 has important implications for all subgroup ana- lyses of long-term (>6 months) versus short-term MPH administration.
Storebø et al reported a smaller effect for long-term administration according to teacher (but not parent or observer) reports. However, since the MTA is the only study meeting their definition of long-term administration, these analyses are misleading. We agree with Storebø et al that there is an absence of long-term data, but disagree that long-term placebo-controlled RCTs provide the solution. There are serious ethical problems in extending placebo-controlled RCTs into the longer term where there is evidence of strong benefit.
In our view, a more appropriate design would be ‘randomised discontinuation trials’ as proposed in the NICE guideline.1 These, along with longer term safety studies, are now a requirement of the regulatory development programme of the European Medicines Agency for new ADHD medications.
Storebø et al adopted the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach (http://handbook. cochrane.org/chapter_12/12_2_1_the_grade_approach.html), which includes the Cochrane risk of bias (RoB) tool (http:handbook.cochrane.orgchapter_8/8_assessing_risk_of_bias_in_included_studies.html).
Using GRADE with regard to the main outcome (teacher-rated ADHD symptoms), they downgraded the quality of evidence by one point for inconsistency of effects (heterogeneity) and by two points for RoB. Both these decisions are questionable. In relation to heterogeneity, I2 ndbook suggests that heterogeneity up to 40% ‘might not be import- ant’. This downgrading is not appropriate, especially considering that exclusion of the MTA study reduces heterogeneity to 25%.
With regard to the RoB assessment, we identified two major issues. First, Storebø et al4 added an additional domain, ‘vested interests’, which is not included in the current Cochrane RoB. The authors support their choice, citing a single work9 and the assessing the methodological quality of systematic reviews approach (AMSTAR) (http://amstar.ca/). While there remains controversy over the best method to account for vested interests,10 11 the Cochrane handbook suggests that it should be reported in the ‘characteristics of included studies’ table rather than in the RoB. Moreover, it is unclear how Storebø et al4 handled the assessment of vested interests in specific cases.
They defined a trial as being at ‘high RoB’ when it was ‘funded by parties that might have had a conflict of interest (eg, a manufacturer of MPH) or where poten- tial conflicts of interest were reported by trial authors’. However, there appears to be no further specification, for example, the number or role of authors with supposed conflict of interest, the time frame of the putative conflicts or the type of industry support that would lead to a study being rated as biased. This may partially explain why this domain was inconsistently rated across the included trials (see online supplementary appendix table 1 for example).
In a BMJ online reply to the concerns of others about their use of this domain, the authors stated ‘There were no trials with only the ‘vested interest bias’ domain assessed as ‘unclear RoB’ or ‘high RoB’. This is incorrect; there are actually seven studies in which this was the only RoB domain rated as unclear or high (see online supplementary appendix table 2).
Storebø et al rated studies as being at high RoB if any domain (includ- ing their additional domain of vested interests) was rated as high or unclear. This approach is problematic because domains were rated as unclear due to lack of detailed information.
However, there is no evidence that the authors routinely attempted to contact authors for clarification regarding RoB domains (including vested interest) rated as ‘unclear’ (see online supplementary appendix, table 3), although they did contact some authors for missing quantitative data. For example, in the study by Ashare et al five RoB domains were rated as ‘unclear’ due to a lack of information.
While Storebø et al wrote that they contacted the authors for quantitative data, there is no evidence they requested RoB information.
Storebø et al argue that the low quality of studies (their assessment) casts doubt on the accuracy of effect sizes. A more scientific approach is to test their opinion with the available data on RoB and study quality.
The authors compared RCTs at high versus low risk of all bias (their ratings) and found no significant difference in effect size (χ21=2.43, p=.12). Unfortunately, this finding is not included in their adjunct publications in the BMJ and JAMA.
Storebø et al4 went on to suggest that even those studies where no item of the RoB was rated as high/unclear were likely to be biased, due to unblinding. They assumed that ‘people in the trial might know which treatments the children were taking’ because of adverse effects associated with MPH. We deem this unlikely, at least with their primary outcome ratings from teachers.
The most common adverse effects reported are sleep difficulties and appetite reduction. There is no evidence to support the claim that teachers would be aware of these symptoms in their pupils and this is not our clinical experience.
ABSTRACT
Banaschewski and colleagues from the European Attention Deficit Hyperactivity Disorder (ADHD) Guideline Group make a number of critical comments regarding our systematic review on methylphenidate for children and adolescents. In our present article we demonstrate that our trial selection was not flawed and that inclusion/exclusion was undertaken with scientific justification. Similarly, our data collection and interpretation was systematic and correct. We have followed a sound methodology for assessing risk of bias and our conclusions are not misleading. Weacknowledge that different researchers might make risk of bias judgments at higher or lower
thresholds, but we have been consistent and transparent in applying our pre-published peer-reviewed protocol. Although we made minor errors, we have reviewed these and demonstrate that the effects are negligible and not affecting our conclusions. We are happy to correct such errors as well as to engage in debate on methodological and ethical issues. In terms of clinical implications, we are not advocating that clinicians or patients and their relatives should weight carefully risks and benefits. Clinical experience seems to suggest that there are people who benefit from this medication. Our systematic review does, however, raise questions regarding the overall quality of the methylphenidate trials.
Banaschewski and colleagues from the European Attention Deficit Hyperactivity Disorder (ADHD) Guideline Group make a number of critical comments regarding our systematic review on methylphenidate for children and adolescents.1,2,3 We thank for their continued interest in our work. We respond here to the critical points they raise, and we do not believe they have identified any reason to change the conclusions of our review
Re. inappropriate selection of studies for inclusion
Banaschewski et al. suggest that in our analyses we included trials that should have been excluded.
This is not correct. They highlight three trials, which they cite as having used “active controls” whereas these are actually co-interventions used in both the methylphenidate group and in the control group. Such trials are eligible for inclusion in accordance with our protocol because they still compare “methylphenidate” with “no methylphenidate”. The fact that both the “methylphenidate” and “no methylphenidate” arms receive an identical co-intervention does not invalidate the comparison of “methylphenidate” with “no methylphenidate”. If anything, the psychological co-intervention may reduce ADHD symptoms in each arm and this would add value to the comparison of the pharmacological treatment of the remaining symptoms, i.e., the “methylphenidate” versus “no methylphenidate” component of the intervention.
Moreover, excluding these trials from our review would have produced only a negligible change in our results (a difference of 0.06 points in the standardized mean difference). Banaschewski et al. also claim that we included a trial that was not randomised.8 This is not correct. We received information by email from Dr. Lufi September 17th 2013, clarifying that this trial was randomised.
The Multimodal Treatment of Attention Deficit Hyperactivity Disorder (MTA) trial is correctly included in accordance with our protocol. We raise doubts about the long-term benefit of methylphenidate. The MTA trial, which reported that beneficial effects may diminish over time, was the only trial that met our “long duration” inclusion criteria and was therefore correctly included. Our sub-group analysis of teacher-rated ADHD symptoms in “short duration” trials (up to 6 months) compared to “long duration” trials (over 6 months) showed a significant difference. The fact that there was only one trial included simply reflects the dearth of long-term trials of methylphenidate. In the clinical world, however, most patients receive methylphenidate for substantially longer periods than the length of existing trials. Randomised discontinuation trials are useful where the drug effect is known to be beneficial. However, we cannot yet be sure this is the case for methylphenidate so as the drug efficacy is still uncertain. The correct design for further trials would there be to undertake randomised placebo – or active placebo (nocebo)-controlled trials to be undertaken.
Re. assessment of study quality
Our assessment of the evidence as being ‘very low quality’ is not only based on the assessment of risk of bias in the included trials, but on additional factors encompassing heterogeneity, imprecision, indirectness of the evidence, and publication bias.7 12 13 We comment on this in detail in our review.
We downgraded the quality of the included trials in the teacher-reported ADHD symptoms meta-analysis to ‘very low quality’ because of high risk of bias and moderate heterogeneity. We have written in our published protocol that we would consider I values between 30% to 60% as moderate level of heterogeneity. The downgrading for heterogeneity might be considered borderline, but we chose to downgrade because we think that a heterogeneity score (I2) of 37% may affect our findings. Had we not done so, the downgrading would have become ‘low quality’, still signaling
that one ought not to have too much confidence in the findings. The short trial duration could be the
basis for further downgrading for indirectness according to GRADE. 12, 13 We did not downgrade for
this, but arguably could have done so.
For many years, there has been intense discussion within The Cochrane Collaboration, about the
risk of bias due to vested interests.14-16 Andreas Lundh and colleagues have shown that sponsorship
and conflict of interest affect outcomes through many subtle mechanisms.17
They also demonstrated that vested interests per se were enough to lead to overestimation of benefits and underestimation of harms, even when all other bias domains were assessed as being of low risk of bias.
Banaschewski et al. state that our vested interest domain was inconsistently rated across the included trials and show some examples of this in their appendix. Had there been inconsistencies regarding one domain of bias in a few trials this would not change the fact that these trials, overall, should still be considered as being at high risk of bias.
Regarding the three trials by Barkley 1991, Brown 1985, and Jensen 1999 in Table 11, we erroneously rated them as being of low risk of bias.
They should have been rated as “unclear” because of the vested interest bias and we will correct this in the next update of our review (again, it will not materially change our results or conclusions). It is not incorrect for us to state that none of the trials funded by the pharmaceutical industry showed a low risk of bias in all other areas as we considered all the trials as high risk of bias on the domain of blinding because the common and well recognised adverse effects of sleep difficulties and appetite suppression are easily recognisable by outcome assessors. We report this clearly in our review.
We contacted the authors of 161 trials twice for supplemental information, including information about vested interests. Only 92 responded. Based on the available evidence, we assessed 71 trials as having a high risk of bias in the vested interest domain as they were funded by the industry and/or the authors were affiliated with the industry.
We contacted Ashare et al. by email in 2013, but did not receive a response. However, considering that this trial was funded by National Institute of Mental Health and from the National Institute on Drug Abuse, we rated it as low risk of bias on the vested interest domain.
In our BMJ article, we reported that we had undertaken the risk of bias subgroup analysis, but did
not report the result in the BMJ article as we considered all trials to be high risk of bias trials.
As our BMJ article cross references our full Cochrane review, the readership will locate this information in the full review, should they wish to.
In placebo-controlled trials it is possible to discriminate between drugs inducing symptoms / adverse events and placebo on the basis of reported effects alone. This fact raises questions about the true blindness of such trials.18 Adverse events such as decreased appetite or sleep difficulties (and subsequent tiredness) are unsubtle and obvious, so easy detection by teachers is likely.
On the issue of nocebo, we acknowledge that there are substantial ethical dilemmas around their use. Several authors have underlined the importance of the use of ‘active placebo’ (nocebo) in clinical trials.18 This is a methodological issue and we would like to stress that nocebos would need to first be shown to be safe in adults, and methylphenidate versus nocebo trials also shown to favour methylphenidate in adults. Only then would nocebo controlled trials be ethically defensible in children.
Re. serious adverse effects of methylphenidate
We agree with Banaschewski et al. that there is certainly a weakness of the available evidence of serious adverse events from randomised clinical trials. Based on our other protocol on methylphenidate for ADHD, we are presently examining the reporting of adverse events in quasi-randomised and observational studies.19 This work is not yet complete. We are not sure that the article by Graham et al. showing almost no adverse events, accurately reflects the evidence in this field and it may reflect their bias or reflect the fact that adverse events are under-reported in
efficacy trials.
Re. effect sizes and clinical effectiveness
The problem is that one does not know the true magnitude of the effect size due to the very low quality of evidence. Therefore, it is difficult to state whether the effect size of methylphenidate should be favorably viewed.
Re. errors in our primary analyses of the teacher-rated ADHD symptoms outcome
As reported, all 19 trials were includable in accordance with our protocol.7 Banaschewski et al. state that they found errors in the imputation of data and/or sample size in seven trials. We have now checked these trials for errors and found that we included incorrect values in the placebo group of the Butter 1993 trial. We will correct this in our next review update. Again, this will not materially change our results or conclusions.
When reporting end-of-period data in crossover trials in meta-analyses, it is correct to count the data from the pre-crossover period as well as the post-crossover if such data are available. This is not “double counting” as separate data exists for exposure of the participants to both placebo and active drug. But we did erroneously count participants twice in two crossover trials (Moshe 2012 and Taylor 1987) in which we only had data from the first period. We re-analysed the data accordingly and the standardized mean difference is now -0.78, 95% CI -0.92 to -0.64 (rather than -0.77, -0.90 to -0.64, as originally reported); test for subgroup differences: Chi² = 0.01, df = 1 (P = 0.91), I² = 0%. None of the corrections leads to any noticeable changes in our results or conclusions. Anyway, we will correct these in an update of our review.
Regarding the Findling 2006 trial, the authors state that: “The primary efficacy population was the per-protocol (PP) population defined as those subjects who received study treatment and had at least one efficacy measurement after the first dose, with no major efficacy protocol deviations”. Yes, we used the per-protocol population, because – as stated in the review – we conducted the analyses using available data.
Many of the trials had several publications. In some of these there were participant numbers that differed, depending on which outcome was being reported. Moreover, in some trials we had to compute standard deviations from standard error values and in one trial we had to calculate the mean difference and standard deviations from total mean difference and p value. This was due to data not being presented in the publications themselves and authors not providing these data. We received data for the first period of the crossover trial by Moshe 2012. These data are not reported in the published articles and it is impossible to find them in the published articles. For this reason,
we reported in the review all the data we used for analyses.
We appreciate these small errors being highlighted as it has given us the opportunity to reflect on their meaning and retest and evaluate their impact; however, they do not lead to noticeable changes in our results or conclusions. Please do let us know if more errors are identified and we shall correct any mistakes in future updates.
Conclusions
We have demonstrated that trial selection in our review was not flawed and was undertaken with sufficient scientific justification. We have already responded to similar criticisms by members of the European ADHD Guidelines Group elswhere and hope that our point-by-point replies have helped clarify the issues further. Synthesising data from all identified 38 parallel and 147 crossover randomised trials comparing methylphenidate versus placebo demonstrates effects that are at best modest (based on external criteria for clinical significance) and limited to a very short window of time, i.e., less than 3 months. In terms of clinical implications, we are not advocating that clinicians or patients should weight if any potential benefits overpower any risks of harms. Clinical experience seems to suggest that there are people who benefit from this medication. Our study does, however, raise questions regarding the overall quality of the methylphenidate trials. These shortcomings have previously largely been ignored. Clinicians, parents, and children have the right to know this, in order to make decisions informed by the evidence.
This website uses cookies.
Tilføj din kommentar her - Feedback er altid velkomment!