Mindstretching meta-analyses

Three recent papers on meta-analysis merit careful examination.

Predictive ability of meta-analysis

The first concerns the predictive ability of meta-analysis, namely the ability of a meta-analysis to predict the results of trials that may be done in the future [1]. The research workers calculated relative risks for 30 meta-analyses of different interventions in perinatal medicine and compared the results with the results of the largest trial done in each intervention. Twenty-four of the 30 meta-analyses correctly predicted the direction of effect in the largest trial.

A meta-analysis demonstrating a protective effects of more than 40% from an intervention had a 60% probability of correctly predicting results of the same magnitude of the largest trial.

The authors confirm the finding that "accumulative meta-analysis can help determine when additional studies are no longer needed and approve the predictability of previous small trials", referring to the classic paper of Lau et al [2]. But they emphasised that the results of meta-analysis are influenced by the readers of the technique, "especially the way the trials are selected".

This same theme was dealt with in two leading articles in the BMJ of the same week. March 25 was a big week for meta-analysis and the press.

An effective intervention that wasn't

The reasons for the leading articles in the BMJ was that in 1993 it was argued that magnesium treatment for myocardial infarction was, on the basis of a meta-analysis, "effective, safe, simple and inexpensive". However, the negative findings of ISIS 4, the Fourth International Study of Infarct Survival, contradicted the findings of meta-analysis.

ISIS 4 was a huge trial and offers the opportunity of comparing very large trials with meta-analysis. The authors of the leading article on misleading meta-analysis emphasised a number of points about meta-analysis:-

That more research is needed into the process of meta-analysis.
That registers of clinical trials are essential to reduce the risk of negative trials disappearing from view. The NHS R&D Programme's project register system is designed to overcome this problem, at least for trials in the UK.
That results of meta-analysis exclusively based on small trials should be distrusted because "several medium-sized trials of high quality seem necessary to render results trustworthy".
The results of meta-analysis should be subjected to careful analysis to test the robustness of the findings [3].

Too good to be true

The other leading article [4] was written by one of the authors of the meta-analysis in question. He and a colleague addressed the lessons to be learned from this changing conclusion, emphasising that there were two important lessons. The first was that a meta-analysis of small trials should not be a replacement for large, carefully conducted trials. Second was the need to be cautious of results that seemed too good to be true, and a more focused use of the lower confidence interval of risk reduction as a representation of what may be actually the clinical case - and is it useful?

Mindstretching megablast

Bandolier usually offers only one paper as a mind stretcher for busy people. These papers are so important that on this occasion we recommend several. And if you think that's it - look at reference 5.

References:

J Villar, G Carroli, JM Belizan. Predictive ability of meta-analyses of randomized controlled trials. Lancet 1995 345: 772-6.
J Lau et al. Cumulative meta-analysis of therapeutic risk for myocardial infarction. New England Journal of Medicine 1992 327: 248-54.
M Egger, GD Smith. Misleading meta-analysis: lessons from "an effective safe, simple" intervention that wasn't. British Medical Journal 1995 310: 752-4.
S Yusuf, M Flather. Magnesium in acute myocardial infarction. British Medical Journal 1995 310: 751-2.
KL Woods, DB Barnett. Magnesium in acute myocardial infarction. British Medical Journal 1995 310: 1669-70.

previous or next story in this issue