Since 2015, Storebø et al. have tried to vilify Methylphenidate, by creating doubt around the validity of the scientific evidence which, for more than 50 years, have told us, that Methylphenidate was safe, effective, and had few serious side-effects. It have caused tremendous harm to patients, all over the world, as well as taken time, money and focus from 'real' science. In this article, I ask Storebø et al. (after advice from Cochrane Institute), to clarify their conclusions in their research, and ask them to explain why their science is so 'bias-fixated'.
Since 2015, Storebø et al. have tried to vilify Methylphenidate, by creating doubt around the validity of the scientific evidence which, for more than 50 years, have told us, that Methylphenidate was safe, effective, and had few serious side-effects. It have caused tremendous harm to patients, all over the world, as well as taken time, money and focus from ‘real’ science. In this article, I ask Storebø et al. (after advice from Cochrane Institute), to clarify their conclusions in their research, and ask them to explain why their science is so ‘bias-fixated’.
On April 11, 2019, I contacted the Press Office at Cochrane.org, to place a formal complaint over the way that Storebø et al., for more than 4 years now, have shown improper conduct in their research, by claiming one factual evidence in their scientific papers, but then interpret them in a very biased manner, based on assumptions which they do not have any credible peer-reviewed evidence, with which to back up their claims.
I received this reply, the very same day:
[…]
Dear Peter
I have been in touch with our Editor in Chief and Deputy Editor in Chief and have bought your concerns to their attention. Cochrane welcomes critical feedback as a means to improving our reviews. As such we would like to invite you to submit your concerns through our official channels. This is important as it ensures your feedback is collected and seen by the right people given that the concerns you describe bridge various matters.
Firstly, you are welcome to submit a comment on the review (https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009885.pub2/comment.) This will give the review authors the opportunity to respond to your comments as they relate to the evidence summarised in the review.
Secondly, you may wish to use the formal procedure to register a complaint (https://community.cochrane.org/editorial-and-publishing-policy-resource/overview-cochrane-library-and-related-content/cochrane-library-complaints-procedure.) This will enable us to follow our formal complaints procedure.
With kind regards
Katie Abbotts
Press Office
[…]
… and this is my reply …
Excerpt from [Swanson, 2018]
Storebø and colleagues provide a concise summary of two Cochrane reviews about benefits and harms of treatment of ADHD with methylphenidate: (1) a review of 185 randomized controlled trials (Storebø et al., 2015) that estimated benefits (e.g., moderate-to-large reduction in teacher-rated ADHD symptoms), and (2) a review of 260 non-randomized trials (Storebø et al., 2018) that estimated harms (e.g., infrequent serious but frequent non-serious adverse events). They also mention (without providing much detail) additional information from the structured Cochrane methodology about the risk-for-bias in the trials reviewed (assessed as being high) and quality-of-evidence of the outcomes considered (rated as being very low). This led to the conclusion that “the jury is still out on benefits and harms.”. A similar conclusion of the first review generated an extensive debate in the literature. Some details will be summarized and discussed here to provide context in case the recent review and the summary by Storebø and colleagues revive the debate.
Swanson, J. M. (2018). Risk of Bias and Quality of Evidence for Treatment of ADHD With Stimulant Medication. Clinical Pharmacology and Therapeutics, 104(4), 638–643. https://doi.org/10.1002/cpt.1186
Excerpt from [Cortese et al., 2018]
Importantly, previous systematic reviews or meta-analyses [(Storebø et al., 2015) and (Storebø et al., 2018) ] also attempted to address the comparative efficacy of some available drugs for ADHD, considering either different formulations of the same drug class or different classes and agents. Such systematic reviews/ meta-analyses have relied on comparison of effect sizes from individual RCTs or on the qualitative/quantitative summary of head-to-head studies; most of them focused on the comparison of two drugs only. Overall, evidence from these reviews is inconclusive and/or mixed. For example, while some of them found comparable efficacy between psychostimulants and atomoxetine, others pointed to significantly higher efficacy of psychostimulants compared with other drugs.
Cortese, S., Adamo, N., Del Giovane, C., Mohr-Jensen, C., Hayes, A. J., Carucci, S., … Cipriani, A. (2018). Comparative efficacy and tolerability of medications for attention-deficit hyperactivity disorder in children, adolescents, and adults: a systematic review and network meta-analysis. The Lancet Psychiatry, 5(9), 727–738. https://doi.org/10.1016/S2215-0366(18)30269-4
Excerpt from [Swanson J.M., 2018]
The general consensus is that ADHD is a one of the most prevalent psychiatric disorders of childhood and requires treatment, and recognition rates and treatment rates have been increasing for decades (Visser et al., 2014). Recent guidelines (see www.nice.org) recommend methylphenidate as the first-line pharmacological treatment for children and adolescents with ADHD. Storebø and colleagues summarized two reviews of methylphenidate based on methods of the Cochrane Collaboration (www.cochrane.org) and questioned the strength of this recommendation.
First, they summarized the well-publicized 2015 Cochrane review of 185 randomized controlled trials (Storebø et al., 2015). They emphasized the moderate-to-large benefits of methylphenidate reflected by the primary outcome specified for the review (teacher ratings of ADHD symptoms), which had a standardized mean difference between groups of -0.78 for 19 trials that qualified for this meta-analysis. They also explained that the analyses of adverse events were underpowered, calling for another review based on the larger number of non- randomized trials (despite the limitations this imposed).
Second, they summarized the just- published 2018 Cochrane review of 260 non-randomized trials (Storebø et al., 2018). They estimated a low percentage of participants had serious adverse events (1.2% with 1 or more) based on 7 qualifying studies, but a high percentage had non-serious adverse events (51.2% with 1 or more) based on 49 qualifying trials. Beyond some issues about selection of the trials to be included in the meta-analyses, these estimates of benefits and harms are generally consistent with expectations and should not be controversial.
Due to the brevity of the summary, Storebø and colleagues1 were not able to provide details but they did mention additional findings from these reviews based on assessments of risk-of- bias and quality-of-evidence. For the review of randomized controlled trials (Storebø et al., 2015), they state “… all included trials were assessed as being at high risk of bias [and] evidence certainly was rated as being ‘very low’”, and for the review of non-randomized trials (Storebø et al., 2018), they state “… All outcomes had low quality of evidence”. The perceived uncertainty about the estimates of benefits and harms led Storebø and colleagues1 to suggest (in the title) that “… the jury is still out on the benefits and harms of methylphenidate for children and adolescents with ADHD”. This conclusion is likely to be controversial.
The brief and concise summary provided by Storebø and colleagues1 did not discuss the controversy and debate that ensued after the publication of the first review2. This commentary is intended to provide context for interpreting the brief summary and appreciating the controversy and the debate that is likely to continue.
Swanson, J. M. (2018). Risk of Bias and Quality of Evidence for Treatment of ADHD With Stimulant Medication. Clinical Pharmacology and Therapeutics, 104(4), 638–643. https://doi.org/10.1002/cpt.1186
Excerpt from [Swanson J.M., 2018]
Storebø and colleagues addressed risk-of-bias in their brief summary, but they could not provide many details (which of course they provided elsewhere). Instead, they just mentioned the conclusions from the review of randomized trials (“… all included trials were assessed as being at high risk of bias”) and from the review of non-randomized trials (“… we deemed all studies without valid or eligible control groups to be at critical risk of bias”).
The peer-reviewed Cochrane protocols published in 20125 and 20166 specified most of the procedures implemented for the reviews for assessing risk-of-bias.
In the 2012 protocol, Storebø et al5 stated (a priori) concern about “… biases and other risks [in previous] studies and reviews” and intention to “… carefully assess these risks”. The protocol specified the use of the Cochrane Risk of Bias (RoB) Tool for randomized trials,7 which directs the systematic assessment of each trial on specific domains of risk.
There are 7 standard domains (selection bias-random assignment, selection bias-allocation concealment, performance bias-blinding of participants, detection bias-blinding of assessors, attrition bias-incomplete reporting, reporting bias-selective reporting, and other bias), but the protocol5 specified an additional domain (vested interest bias, which became a reason for controversy after the publication of the review).
The Cochrane RoB tool has 3 risk-of-bias categories (low, uncertain, or high), and the systematic assessment assigned one of these to each domain for each trial. To integrate the multi-domain information across trials, a table is constructed with one row for each trial and columns with colored cells (low = green, uncertain = yellow, high = red) showing the assigned categories for each domain across the trials. A rule is applied to classify each trial: as stated a priori in the protocol “… trials with one or more unclear or inadequate component [will be considered] as trials with high risk of bias”.
Following these procedures, Storebø et al2 reported that for all 185 randomized trials, only 6 met the stringent criterion of low risk-of-bias on all domains.
Even for these, unblinding by common non-serious adverse effects was considered likely, so the classifications for the two domains of blinding were changed to high, leading to all 185 trials being judged to have high risk-of-bias on these domains.
Figure 1 in the brief summary (Storebø et al., 2015) shows the risk-of-bias table for the subset of 19 trials that qualified (i.e., were randomized controlled trials with parallel groups) for the main review of the primary outcome (teacher ratings). All trials had at least one domain with high or uncertain risk-of-bias, so by the rule specified in the protocol, all of these trials were judged to have high risk-of-bias.
Swanson, J. M. (2018). Risk of Bias and Quality of Evidence for Treatment of ADHD With Stimulant Medication. Clinical Pharmacology and Therapeutics, 104(4), 638–643. https://doi.org/10.1002/cpt.1186
Table 1: The Critiques and Replies in the Discussion and Debate
| Critiques | Titles Showing Tenor of the Debate and Discussion |
| Storebø et al | Cochrane systematic review with meta-analyses Dissemination in BMJ |
| Banaschewski et al | The errors and misinterpretations in the Cochrane analysis Discussion Forum in ZKJP |
| Storebø et al | Response to Banaschewski et al. Letter to Editor ZKJP |
| Romanos et al | The Cochrane review indeed flawed Response to Letter ZKJP |
| Hoekstra et al | Is the evidence base flawed? Editorial ECAP |
| Storebø et al | The evidence base is in fact flawed Letter to Editor ECAP |
| Hoekstra et al | Response to: The evidence base is in fact flawed Letter to Editor ECAP |
| Banaschewski et al | Throwing the baby out with the bathwater Perspective EBMH |
| Storebø et al | Evidence is in fact of ‘very low quality’ Perspective EBMH |
| Storebø et al | MPH for ADHD (recommendations should be reevaluated) Clinical Synopsis JAMA |
| Shaw | Quantifying the Benefits and Risks of MPH for ADHD Editorial Opinion JAMA |
| Romanos et al | MPH for ADHD (flawed in both methods and interpretation) Letter to the Editor JAMA |
| Storebø et al | In Reply Letter to the Editor JAMA |
| Gerlach et al | What are benefits? (dogmatically biased devaluation) Editorial ADHD |
BMJ: British Medical Journal
ZKJP: Zellschrift fur Kinder- und Jugenosychiatrie und Psychortherapie
ECAP: European Child and Adolescent Psychiatry
EBMH: Evidence Based Mental Health
JAMA: Journal of the American Medical Association
ADHD: Attention Deficit Hyperactivity Disorder
Swanson, J. M. (2018). Risk of Bias and Quality of Evidence for Treatment of ADHD With Stimulant Medication. Clinical Pharmacology and Therapeutics, 104(4), 638–643. https://doi.org/10.1002/cpt.1186
Excerpt from [Storebø et al., 2015]
[…] We assessed the risk of bias of each included trial using the Cochrane ’Risk of bias’ tool. As shown, we assessed six cross-over trials (3%) as having low risk of bias in all domains. We assessed the remaining 179 trials (97%) as trials with high risk of bias. However, even the six cross-over trials had likely breaks in their blinding due to prevalent adverse events to methylphenidate (see below). Accordingly, we judged all 185 trials to be trials with high risk of bias. As assessed by the GRADE approach, the overall quality of evidence in this review is ’very low’ because of high risk of bias (including loss of blinding (explained below) and outcome reporting bias), inconsistency, and indirectness; this led to uncertainty in the robustness of our estimates. We rated the risk of outcome reporting bias for adverse events to be high, as we only managed to obtain data on total serious adverse events from 9 of the 185 included trials, and on total non-serious adverse events from 21 of the 185 included trials. We found no evidence for publication bias. It is likely that the trials initially judged to be at low risk of bias may, in fact, be trials at high risk of bias because methylphenidate gives rise to various prevalent and easily recognisable adverse events, which can lead to loss of blinding and hence can bias the ratings of symptoms, resulting in an overestimation of benefits and an underestimation of harms. […]
Q Based on what evidence, are you concluding that 97% of all trials conducted over the past 15 years, are of ‘a high risk-of-bias’, because you state (but do not provide concrete evidence for), the teachers risk deblinding, due to the ‘easily recognisable adverse events’?
[…] To ensure adequate blinding, it is therefore important for researchers to use a nocebo (’active placebo’) in the control group. As we found no trials employing nocebo tablets in the control group, the extent of this bias cannot be assessed. The fact that the intervention effect of methylphenidate on ADHD symptoms did not differ significantly between trials at low risk of bias compared with trials at high risk of bias may be taken as an indication that deblinding has occurred among former trials. […]
Q Although you state here that ‘the extent of the bias cannot be assessed’, elsewhere you claim that […] we identified no such trials, and as the use of nocebo tablets in all diseases is ethically questionable, any decision to conduct nocebo tablet controlled trials in children should be deferred pending the results of such trials in adults […], how, and on what evidence, can you then conclude, that the current evidence is biased?
[…] We chose to use teacher-rated outcomes as the primary measure for both ADHD symptoms and general behaviour, although a number of trials used or relied on parent reports. Importantly, we do not really know what a lower score on an ADHD symptom scale (like that reported in this review) means for a child’s quality of life and ability to live, learn and function with other people. […]
Q Based on what evidence, can you assume that parent-reports not as reliable, as teacher-reports?
[…] ADHD can exert a significant, negative impact on children’s quality of life, broadly defined. In each case the assessments were made by parents, teachers or independent assessors, rather than by children themselves. […]
Q Based on what evidence, makes it so, that you prefer the child’s own assessment on quality of life? And how, and by using which scientific evidence, would you test this?
[… ] These external assessors observed small beneficial effects of methylphenidate on quality of life. Children might well have had different views on their own quality of life, and the failure to include child-reported ratings of quality of life is a significant limitation on the completeness of the evidence. […]
Q Based on what evidence, do you consider it ‘ a failure ‘ and ‘ a significant ‘ limitation on the completeness of the evidence, that the child’s own assessment of their quality of life, is not included?
[…] Furthermore, observations of quality of life reported by parents, teachers and independent assessors may be subject to both systematic and random errors. […]
Q Based on what evidence, do you suspect, that these observations ‘ may be subject to both systematic and random errors?
[…] The majority of trials in this review compared methylphenidate with placebo, and we earlier highlighted the problems of threats to blinding in these trials, due to the prevalence of non-serious adverse events of methylphenidate. […]
Q Based on what evidence, do you assume that teacher-reports are subject to deblinding, due to the notability of the non-serious adverse effects (decreased appetite, sleep problems)? And again, based on what evidence, do you assume that a teacher assesses a pupil’s appetite, let alone sleep problems, by observing them for a few hours a day in school?
[…] Trials that assess methylphenidate using an ‘active placebo’ (or ’nocebo tablets’ – tablets with a placebo-like substance that causes similar adverse events as in the experimental drug arm), can strengthen double blinding and are recommended. We identified no such trials, and as the use of nocebo tablets in all diseases is ethically questionable, any decision to conduct nocebo tablet controlled trials in children should be deferred pending the results of such trials in adults. If these show that methylphenidate is superior to nocebo in treating ADHD symptoms, a rationale would exist for conducting such trials in children. (If another rationale could be mounted, nocebo controlled trials in children would not be ethically questionable, but currently no such case is evident to us). […]
Q Based on what evidence, can you claim that the placebo based trials, are more like to not report adverse events, when you cannot compare to the suggested nocebo based trials today?
We are currently preparing a second systematic review for the purpose of assessing harms of methylphenidate in observational trials with a duration of up to 36 months (Storebø in press). Given the possible association between methylphenidate and the adverse events identified, it may be important to identify people who are most susceptible to adverse events. To do this we must undertake large‐scale, high‐quality RCTs, along with studies aimed at identifying responders and non‐responders. Preliminary results from this review show a small proportion of serious adverse events reported after treatment with methylphenidate, but one case control trial highlights the risk of sudden death for adolescents. Just over a quarter of children appear to experience non-serious adverse events after methylphenidate treatment. Many claims have been made about significant increases in global rates of methylphenidate prescribing; this drug usually is prescribed for long-term use and seldom with medication-free periods. However, a recent paper reports that children in primary care in the UK did not continue methylphenidate treatment for longer than six months. Furthermore, the incidence of ADHD diagnoses in the UK fell between 1998 and 2010 (Holden 2013). In the USA, however, almost 70% of children with ADHD, estimated at 6.4 million children, take medication. […]
[…] This might mean that clinicians in the UK are more cautious about using methylphenidate, while US clinicians assume that evidence for the safe use of methylphenidate is sound. […]
Q Based on what evidence, can you make such an assumption, that UK clinicians are more cautious, and that US clinicians are not?
[…] Our assessment of the evidence does not preclude that individual patients may benefit from intervention with methylphenidate. However, despite more than 50 years of research in this field, we do not yet know how to identify those patients that may obtain more benefits than harms. Individual patient data meta-analyses are needed to try to identify such patient characteristics. […]
Methylphenidate may improve ADHD symptoms, general behaviour and quality of life in children and adolescents aged 18 years and younger with ADHD. We rated the evidence to be of very low quality and, as a result, we cannot be certain about the magnitude of the effects from the meta-analyses. The evidence is limited by serious risk of bias in the included trials, under-reporting of relevant outcome data, and a high level of statistical variation between the results of the trials. There is also very low quality evidence that methylphenidate causes numerous adverse events. The risk of serious adverse events seems low, but data were available from only 9 of the 185 included trials. It is also problematic that only 93 of the 185 included trials reported on specific and overall non-serious adverse events. […]
Accordingly, we cannot rule out the possibility that non-serious harms are more prevalent than reported in our review.
Q Based on what evidence, do you base your assumption of ‘the possibility of non-serious harms are more prevalent’?
[…] If methylphenidate treatment is considered, clinicians might need to use it for short periods, with careful monitoring of both benefits and harms, and cease its use if no evidence of clear improvement of symptoms is noted, or if harmful effects appear. […]
Q Based on what evidence, are you recommending that clinicians should only use methylphenidate, for a short period, when the wash-out of methylphenidate is very short, and that the adverse events then naturally stops as well? And how would you, and based on what evidence, assess ‘benefits and harms’ when all the current scientific evidence, is evaluated as ‘high risk-of-bias’ and ‘low quality of evidence’?
To assess whether any benefits are provided by methylphenidate, large, randomised, nocebo tablet (’active placebo’) controlled trials are needed to compare the drug with a nocebo tablet, if deblinding can be excluded. A problem with many pharmacological trials is that participants in the experimental drug group experience adverse events that compromise the blinding, as fewer adverse events occur in participants in the placebo group. However, given the ethical dilemmas regarding the use of nocebo tablets in children, such trials should be conducted first in adults with ADHD, unless one can provide a rationale to justify nocebo controlled trials in children (Moncrieff 2004; Po gain 2014). Nocebo controlled trials in adults demonstrating that methylphenidate is superior to nocebo in treating ADHD symptoms would provide a rationale for conducting such trials in children. As with RCTs, systematic reviews of RCTs assess average effects in groups of individuals. Such average effects may comprise strong benefits for a single participant or a few participants and no effect or negative effects for others. Despite more than 50 years of research in this field, we have no knowledge on how to identify patients that may obtain more benefits than harms. Individual patient data meta-analyses are needed in order to identify such patient characteristics. Therefore, it would be extremely helpful for review authors to gain full access to anonymised individual participant data for inclusion in meta-analyses examining these data (Gluud 2015b). Particular patient subgroups may benefit from an intervention if those with reduced rates of adverse events can be identified.
Q Based on what evidence, would you assess whether a patient subgroup may benefit from methylphenidate, when all the current scientific evidence, is evaluated as ‘high risk-of-bias’ and ‘low quality of evidence’?
Storebø, O. J., Ramstad, E., Krogh, H. B., Nilausen, T. D., Skoog, M., Holmskov, M., … Gluud, C. (2015). Methylphenidate for children and adolescents with attention deficit hyperactivity disorder (ADHD). Cochrane Database of Systematic Reviews, 2015(11). https://doi.org/10.1002/14651858.CD009885.pub2
Excerpt from (Storebø et al., 2018):
[…] Our findings suggest that methylphenidate may be associated with a number of serious adverse events as well as a large number of non‐serious adverse events in children and adolescents, which often lead to withdrawal of methylphenidate. Our certainty in the evidence is very low, and accordingly, it is not possible to accurately estimate the actual risk of adverse events. It might be higher than reported here. […]
[…] In the comparative studies, methylphenidate increased the risk ratio (RR) of serious adverse events (RR 1.36, 95% confidence interval (CI) 1.17 to 1.57; 2 studies, 72,005 participants); any psychotic disorder (RR 1.36, 95% CI 1.17 to 1.57; 1 study, 71,771 participants); and arrhythmia (RR 1.61, 95% CI 1.48 to 1.74; 1 study, 1224 participants) compared to no intervention. […]
Q Based on what evidence, can you assure the readers, that non of the psychotic episodes, was due to unidentified Schizophrenia or Bipolar Disorder? And, likewise, based on what evidence, can you assure, that those that experienced cardiovascular adverse events, did not suffer from a unidentified cardiovascular problem, before the methylphenidate was administered?
Q Based on what evidence, do you describe with scientific accuracy that ‘a large number of non‐serious adverse events’?
Q Based on what evidence, do you describe, with scientific accuracy that non‐serious adverse events ‘often lead to withdrawal of methylphenidate’?
Storebø, O. J., Pedersen, N., Ramstad, E., Kielsholm, M. L., Nielsen, S. S., Krogh, H. B., … Gluud, C. (2018). Methylphenidate for attention deficit hyperactivity disorder (ADHD) in children and adolescents – assessment of adverse events in non-randomised studies. Cochrane Database of Systematic Reviews, 2018(5), CD012069. https://doi.org/10.1002/14651858.CD012069.pub2
Based on my prior attempts to get a dialogue with Storebø et al., I look forward to hearing if this approach may induce them to answer my questions, which I originally placed to them back in 2015, but still have not received any answer on …
Time will tell …
/ADDspeaker
This website uses cookies.
Tilføj din kommentar her - Feedback er altid velkomment!