How good are systematic reviews of acupuncture?
- Systematic review
- Results
- Comment
For
over 150 issues now, Bandolier has been banging on about the need to examine
evidence with a cold and fishy eye. If the concept of evidence-based medicine
has done nothing else, it has focussed our attention on the propensity of
papers in the medical sciences to give a partial view of the truth.
Bandolier
looks for three things: quality, to avoid bias; validity, to make sure that
what we get is what we want; size, to make sure that we aren't being
pushed around by the random play of chance. Miss out on any one of these and
the answer you are looking at may well be wrong.
Systematic
reviews of acupuncture have tended to support the use of acupuncture, but also
tended to use studies with known sources of bias, that were small, or had
limited validity. What happens when you re-examine these systematic reviews
using criteria of quality, validity, and size, to check out how robust their
conclusions were? A systematic review of systematic reviews [1] tells us.
Systematic
review
Systematic
reviews published in English examining the efficacy of traditional Chinese
acupuncture or mechanical or other forms of acupuncture were sought if they
were published between the beginning of 1996 and the end of 2005. If there was
more than one review on a particular topic, the most recent was used.
The
reviews were examined for the conclusions of the original authors, and were
also examined against prior criteria of quality, validity, and size. For
instance, for quality, only trials that were both randomised and adequately
blinded were acceptable, using sham acupuncture controls. For validity,
patients had to have recognised clinical conditions, with groups comparable at
baseline, and use relevant outcomes over relevant periods. For size, at least
four trials and/or 200 patients were the minimum requirements.
Results
After
removal of duplicate reviews, 35 systematic reviews on acupuncture remained.
They examined the use of acupuncture in various painful conditions (n=18),
stroke (n=2), nausea and vomiting (n=2), depression (n=2), and other conditions
including insomnia, smoking cessation, weight loss, and asthma (n=11).
Most
(22/35) claimed to use only randomised studies, and most were patient and
assessor blind. Most (24/35) had results on fewer than 1,000 patients in total,
though the number contributing to efficacy analysis was often much lower.
Of
the 35 reviews, 17 concluded that there was either no evidence of benefit, or
evidence of no benefit. Twelve had a qualified conclusion of some benefit for
acupuncture, and six indicated that the evidence favouring acupuncture was
strong (Table 1). The balance was somewhat less in favour of acupuncture for
Cochrane reviews, and for those with no author affiliation with complementary
medicine, but was somewhat stronger when there was some affiliation.
Table 1: Support by original authors by type of review and affiliation to department of complementary therapy
|
Authors' support for acupuncture
|
|
None
|
Qualified
|
Strong
|
| All studies (n=35) |
17
|
12
|
6
|
| Cochrane (n=12) |
8
|
3
|
1
|
| Affiliated (n=18) |
8
|
5
|
5
|
| Not-affiliated (n=17) |
9
|
7
|
1
|
When
criteria of quality, validity, and size were applied, none of the systematic
reviews demonstrated robust evidence of effectiveness for acupuncture. After
removal of poor quality studies, most reviews had only trivial amounts of good
quality evidence. Only six had more than 200 patients, and in these there was
no evidence of benefit.
Comment
The
point here is not really about acupuncture, but about systematic reviews. Just
because something is labelled as a systematic review does not mean it is any
good. We have to be just as vigilant now as ever. Even a review with a Cochrane
label does not make its true. Four out of 12 Cochrane reviews on acupuncture
were wrong.
Caveat
lector rules, OK?
Of
course, that does not stop us thinking about the efficacy of acupuncture.
Large, high-quality randomised trials of acupuncture have been published since
the reviews. In fibromyalgia, chemotherapy-induced nausea and vomiting, breech
presentation, tension headache, and migraine, all were negative compared with
sham acupuncture. One in osteoarthritis of the knee, had statistical
improvement over sham acupuncture at three months, but not later. Both large
trials and this review of reviews come to the same general conclusion; that
over a whole range of conditions and outcomes acupuncture cannot yet be shown
to be effective.
Reference:
- CJ Derry et al. Systematic review of systematic reviews of acupuncture published 1996-2005. Clinical Medicine 2006 6:381-386.
