Skip navigation
Link to Back issues listing | Back Issue Listing with content Index | Subject Index

How good are systematic reviews of acupuncture?

Systematic review
Results
Comment

For over 150 issues now, Bandolier has been banging on about the need to examine evidence with a cold and fishy eye. If the concept of evidence-based medicine has done nothing else, it has focussed our attention on the propensity of papers in the medical sciences to give a partial view of the truth.

Bandolier looks for three things: quality, to avoid bias; validity, to make sure that what we get is what we want; size, to make sure that we aren't being pushed around by the random play of chance. Miss out on any one of these and the answer you are looking at may well be wrong.

Systematic reviews of acupuncture have tended to support the use of acupuncture, but also tended to use studies with known sources of bias, that were small, or had limited validity. What happens when you re-examine these systematic reviews using criteria of quality, validity, and size, to check out how robust their conclusions were? A systematic review of systematic reviews [1] tells us.

Systematic review


Systematic reviews published in English examining the efficacy of traditional Chinese acupuncture or mechanical or other forms of acupuncture were sought if they were published between the beginning of 1996 and the end of 2005. If there was more than one review on a particular topic, the most recent was used.

The reviews were examined for the conclusions of the original authors, and were also examined against prior criteria of quality, validity, and size. For instance, for quality, only trials that were both randomised and adequately blinded were acceptable, using sham acupuncture controls. For validity, patients had to have recognised clinical conditions, with groups comparable at baseline, and use relevant outcomes over relevant periods. For size, at least four trials and/or 200 patients were the minimum requirements.

Results


After removal of duplicate reviews, 35 systematic reviews on acupuncture remained. They examined the use of acupuncture in various painful conditions (n=18), stroke (n=2), nausea and vomiting (n=2), depression (n=2), and other conditions including insomnia, smoking cessation, weight loss, and asthma (n=11).

Most (22/35) claimed to use only randomised studies, and most were patient and assessor blind. Most (24/35) had results on fewer than 1,000 patients in total, though the number contributing to efficacy analysis was often much lower.

Of the 35 reviews, 17 concluded that there was either no evidence of benefit, or evidence of no benefit. Twelve had a qualified conclusion of some benefit for acupuncture, and six indicated that the evidence favouring acupuncture was strong (Table 1). The balance was somewhat less in favour of acupuncture for Cochrane reviews, and for those with no author affiliation with complementary medicine, but was somewhat stronger when there was some affiliation.


Table 1: Support by original authors by type of review and affiliation to department of complementary therapy



Authors' support for acupuncture
None
Qualified
Strong
All studies (n=35)
17
12
6
Cochrane (n=12)
8
3
1
Affiliated (n=18)
8
5
5
Not-affiliated (n=17)
9
7
1



When criteria of quality, validity, and size were applied, none of the systematic reviews demonstrated robust evidence of effectiveness for acupuncture. After removal of poor quality studies, most reviews had only trivial amounts of good quality evidence. Only six had more than 200 patients, and in these there was no evidence of benefit.

Comment


The point here is not really about acupuncture, but about systematic reviews. Just because something is labelled as a systematic review does not mean it is any good. We have to be just as vigilant now as ever. Even a review with a Cochrane label does not make its true. Four out of 12 Cochrane reviews on acupuncture were wrong. Caveat lector rules, OK?

Of course, that does not stop us thinking about the efficacy of acupuncture. Large, high-quality randomised trials of acupuncture have been published since the reviews. In fibromyalgia, chemotherapy-induced nausea and vomiting, breech presentation, tension headache, and migraine, all were negative compared with sham acupuncture. One in osteoarthritis of the knee, had statistical improvement over sham acupuncture at three months, but not later. Both large trials and this review of reviews come to the same general conclusion; that over a whole range of conditions and outcomes acupuncture cannot yet be shown to be effective.

Reference:

  1. CJ Derry et al. Systematic review of systematic reviews of acupuncture published 1996-2005. Clinical Medicine 2006 6:381-386.

previous or next story