Dear Bandolier, Preventing teenage pregnancy


Thanks for highlighting this important systematic review in your June 2000 issue (vol 7, no 6). Two major points about your commentary:

1. The question is not whether observational studies gave ‘significant’ results and RCTs didn’t (which could for example just be due to sample sizes) but whether they gave different results. For example, the question is whether for males starting intercourse the RCTs give a different odds ratio (0.81) to the observational studies (0.71). You have in effect quite wrongly interpreted non-significant as meaning ‘not different’.
2. If, as you report, the systematic review included observational studies which allocated non-compliers with the intervention to the control group for the purpose of analysis, this is analogous to analysing cross-overs in RCTs by treatment group rather than by randomised (or intended) treatment group. This is obviously biased and well known to be so, and no RCT would be likely to get published (or accepted as a doctoral thesis or as a final report) which attempted to do this.

• However, there are few if any quality controls in observational studies. Quite obvious design and analysis issues, particularly relating to selection biases (many of which are avoidable) are regularly flouted, and of course these badly designed and analysed studies lead to the wrong results. However, exactly the same would be true of badly designed and analysed RCTs. (There are thousands of published examples. You might for example consider your Cisapride example in the same issue). So your first hard lesson ‘that randomisation is everything’ is just not true. The first hard lesson is that quality of design and analysis is everything. The fact that we are very slowly getting to the point where a valid blue print is available for RCTs, and that only RCTs which match the blueprint will usually be accepted (as theses, for publication, by funding bodies) leads to the corollary that if we are going to accept published evidence without any critical appraisal then we should rely only on randomised studies. However, only the blind follow anything blindly.

As I’m sure you know, recent critical reviews have concluded that well designed and conducted observational studies can lead to the right answers (as far as we can know them), and that badly designed randomised trials can lead to the wrong answers. Thus the corollary of the real ‘hard lesson’ is that systematic reviews must take the quality of the studies into account, and that this applies equally or even more strongly to reviews of observational studies.


Yours sincerely,
Jon Nicholl.