Reporting bias in breast cancer clinical trials

The systems that regulate how drugs come to market are failing the very people that they are supposed to be protecting.  Behaviour that has been challenged for decades without resolution has now been publicly exposed through the work of people like Ben Goldacre and the AllTrials campaign.  One important facet discussed by Goldacre in his book, Bad Pharma, is the reality of reporting bias and results spin perpetrating clinical trials.  Reporting bias, such as selectively revealing or supressing drug-patient interaction, can make a drug look more effective than it really is or disguises potentially harmful side-effects.  Essentially, report bias is under-reporting the undesirable results from a clinical trial so that the benefits are maximised and the desirable hypothesis is proven.  There is a fundamental problem with this in that it violates the standards set by the scientific method and thus, renders any information obtained from such trials as tainted.  A hypothesis should not be desirable but should be part of an idea to be proved or disproved.  Accepting that there would be a desirable outcome from testing the hypothesis automatically makes the experiment bias.

There’s so much to write about when it comes to skewed clinical trials and evidence reporting and I encourage you to read further.  However, what I really want to discuss is a new commentary released by NICE (National Institute for Health and Care Excellence), the guidance body that makes recommendations to the NHS on which drugs they should buy.  It’s important to note that NICE can only make recommendations on a particular drug based on the evidence available to them.  Sometimes this can be very little and what is available is often bias.  The commentary was released through NICE’s ‘Eyes on Evidence’ bulletins which aims to provide access to significant new evidence as it emerges.  One section of the November bulletin caught my eye as it was titled ‘Bias in reporting of randomised controlled clinical trials in breast cancer’.  If you would like to know more about randomised controlled clinical trials (RCT’s) please click here.

I think the bulletin does a good job of reporting the evidence from the source publication (£) but lacks some of the important conclusive remarks made by the authors.  The study in question is a literature review, meaning that the authors went to the effort of finding the result of every RCT published between 1995 and 2011, and analysing them for bias reporting.  It sounds like a large job, and it is.  One way to tackle the review is to take all the RCT publications and compare the findings against what the trial aimed to achieve when it was publically registered.  When a trial is being designed the researchers will select a primary end-point – the final measure that determines the success of a trial.  For a cancer drug this may be something like ‘overall survival’ (OS), ‘disease free survival’ (DFS) or ‘progression free survival’ (PFS).  What’s important here is that the primary end-point is registered so that when the study data is released it will be clear whether or not the conclusions have been reached according to the registered end-point.  All clinical trials are supposed to be publically registered before they start in databases such as but practice is not enforced and many trials remain unregistered.  This is clear in the current review where it was noted that from the 164 trials selected only 30 (18%) were registered at prior to their start.  However, we should be very careful when interpreting this finding because only started in 2000 and at conception only included trials registered in the US, with EU trials being asked to register later on.  This means that any RCT in breast cancer from 1995-2000 is unlikely to be in the database.

What I think is more shocking is that from these 30 trials only 7 reported the same primary end-point than was registered.  This represents a serious failure to be transparent and makes post-trial evaluation by independent reviewers nearly impossible.  How does one review the effectiveness of a drug if the primary aim of the treatment is unclear?  In my opinion, changing the primary end-point looks like admission of spin and bias in the trial.

Reporting bias as a result of unclear end-points was evident amongst the 164 trials reviewed.  A total of 54 trials reported positive results based on non-primary end-points despite not finding a statistically significant difference in the primary end-point.  Essentially, if the aim of a new drug was to reduce deaths by breast cancer and the trial failed to show that, a positive conclusion can still be reached based on surrogate outcomes.  Interestingly, if you look at only the trials reporting no statistical significance for the primary end-point, you find that the percentage reporting positive results on surrogate outcomes increases dramatically.  This is a very sneaky way to spin the data to fit your preferred conclusion.  Currently, there is nothing legally wrong with this but it is irresponsible, misleading and means that the real-life risks and benefits of the drug are disguised from doctors and patients.

Another issue raised by the authors was the tendency to use DFS or PFS rather than OS as a primary end-point.  OS is a much better way to assess the effectiveness of a life-saving drug because it reports deaths by any cause and not just the condition being treated.  On merit, this takes into account deaths caused by drug side-effects, as well as deaths resulting from disease relapse.  On the other hand DFS and PFS only take into account the time before relapse or when the disease gets worse.  The review discussed here found that only 27 of the 164 trials (16.5%) reported OS as the primary end-point.  For women with breast cancer, neither DFS nor PFS have been shown to be adequate alternatives for OS, yet over 80% of the trials reviewed used them as end-points.

It was also noted that doctors will often only read the abstract of concluding remarks of a new clinical trial due to time constraints.  It is therefore essential that primary end-points are clearly described in these sections of a publication so that doctors can review the evidence accurately.  Primary end-points were rarely reported in the abstract or conclusions of the trials reporting positive results despite a non-significant primary end-point.

It would be nice to believe that this reporting bias is abnormal, but in fact the statistics presented in the article are consistent with reviews conducted elsewhere and on separate medical conditions.  This review simply adds more evidence to the fact that this fraudulent and misleading behaviour is rife amongst clinical trial reporting.  If there was ever a need for all trials to be published transparently, made fully accessible and regulated with enforcement, then it is now.

Please join the All Trials campaign and help put healthcare back in the hands of doctors and patients.

‘What Doctor’s Don’t Tell You’ don’t tell you – Ask for Evidence

Alternative medicine – two words that are non-scientifically proven to induce high blood pressure and a nauseating feeling of anger in any self-respecting person.  I know this because it’s what I felt when a colleague at work soiled my vision by thrusting my face into an issue of ‘What Doctor’s Don’t Tell You’.  Having previously worked as a breast cancer researcher and now working inside an organisation helping support breast cancer research, I think this particular issue was cast my way for validation of its absurdity.

If you are unaware of this magazine or its editors, Bryan Hubbard and Lynne Mctaggart, you can read more here.

This scene occurred in the wake of Angelina Jolie’s announcement of a double mastectomy and the front page of WDDTY carried the line “what they didn’t tell Angelina – when ‘bad’ genes don’t lead to breast cancer”.  Here we go, I thought, and dived straight in to read the main feature (N.B.  I am completely ignoring the 100 or so other pages in the magazine that are full of wonderfully presented rubbish due to personal health reasons and no one wants to read a blog post that’s 10,000 words).  As I began to read the article I became increasingly agitated as I was faced with more and more claims and statements that at face value looked like complete horse-shit.  But they had referenced many of their statements to what seemed like respectable sources, and I am fair, so I thought I’ll take a look at their references and then form an opinion.

To put my analysis of the article in context, it is worth me explaining something about the Angelina Jolie feature.  The theme here was to explain why Angelina Jolie’s breast cancer risk was not as much as she was told by her doctors (obviously they would be the last people to have a clue) because genetic risk factors are not as important as they are made out to be.  Also, apparently Jolie has done no favours by having a double mastectomy because there is no real benefit for women like her and she has now exposed herself to other risks.

My analysis below is in no way extensive but does show the different ways that WDDTY has misused evidence, provided confusing messages and overall completely misunderstood the subject matter they claim to know so much about.

Confusing ‘family history’ with ‘genetic history’

WDDTY lead in with the statement that ‘most women with a family history of breast cancer will never get the disease’.  This is correct, as having a mother or sister diagnosed with breast cancer approximately doubles the risk, but at the same time more than 80% of these women will never develop breast cancer.  However, this is followed up with ‘for women with one close relative with breast cancer, the lifetime risk is 8 per cent, which increases to just 13.3 per cent for those like Angelina Jolie, with two close relatives who had the disease’.  They cite a nice paper from 2001 that looks at familial breast cancer risk and does indeed quote those statistics.  However, none of these stats are relevant to Jolie’s risk because she also carries a genetic mutation to the BRCA gene, a factor that the paper does not include in their study.  It would seem on reflection that the articles author does not understand the difference between family and genetic history.

Cherry picking

Selecting bits of text from a reference that suit the story is something of a running theme in WDDTY.  Even when the reference is evidence for the opposite argument, it is possible to put a spin on it by just cherry picking what you want, without putting it in context.  This is just one example from the Jolie feature:

WDDTY state that there is ‘no solid evidence that just-in case double mastectomy increases survival’ along with a reference that does indeed show this.  But, and there is a very big but here, the same article also concludes that even taking into account the limitations of the studies in the review, double mastectomy should be considered for very high-risk groups i.e. those with BRCA mutations – such as Angelina Jolie.  So again we are left with a comment on a reference that has no relevance to Jolie’s breast cancer risk or her decision to have a mastectomy.

Misunderstanding references

 At one point in the article the authors say ‘new evidence shows that even a faulty BCRA1 gene, as Jolie has, may require epigenetic modification, or ‘silencing’ and deletion, before it can progress.  I took a closer look at the cited paper and what it is actually investigating is the epigenetic control of ‘normal’ BRCA genes and how these changes could drive tumour progression by producing a genetic state similar to a BRCA mutation.  The author’s even state: ‘this lends support to the idea that epigenetic silencing of the BRCA1 gene might channel tumour progression, akin to an underlying BRCA1 germline mutation resulting in a BRCA-like phenotype’.

So the reference does not back up the claim that Angelina Jolie’s risk is not as high as she was told.  More importantly, it does not justify printing information that claims BRCA mutations are not an important risk factor – or not as important as epigenetic changes.  There is no evidence for this claim.

Irrelevant information

The final part of the article attempts to tie everything together by saying that because Jolie had a double mastectomy she opens herself up to other risks that she may not know about.  This section is particularly bad and contains an incoherent suggestion that breast implants increase risk of anaplastic large-cell lymphoma.  This by definition is a lymphoma – a blood cancer – yet WDDTY are insistent that it is actually a rare form of breast cancer.  If this wasn’t bad enough they then reference a study that concluded there is an association between breast implants and ALCL but that the overall risk is ’exceedingly low’.  I haven’t even mentioned yet that the study was extremely small and limited to a distinct set of patients in Holland.

People will always ask, so what? Why waste time on it? It never hurt anyone.  The fact is that this information is poison and carries the potential to infect and harm members of the public.  Imagine the women who finds out she is a carrier of the faulty BRCA gene and is convinced not to consider preventative surgery.  People like her are vulnerable to information supplied by WDDTY, unless they are made aware, and shown how to pick the peanuts of truth out of the information turd.

There are many organisations that supply balanced health advice based on evidence that aims to help people make informed decisions.  I think that WDDTY, by representing the evidence in this way, are putting dangerous health advice in the public domain that could be read by vulnerable people.  This is a very serious issue as it could potentially put someone off having preventative surgery when the benefits far outweigh the risks for their particular circumstances.  For those in the high-risk group, such as Angelina Jolie, there is plenty of evidence for the benefits of double mastectomy and I can just cite their references as proof.

The Jolie article that I dissected is by no means the worst offender or is it the only dangerous piece of public health information published amongst their pages.  Alan Henness was kind enough to send me copies of every WDDTY and there is ill-informed advice on vaccination, heart disease, arthritis, dementia, all cancers, colds, flus, HIV….the list goes on.

I wanted to take some sort of action about this so I wrote a letter (copy here) with the help of ‘Sense about Science’, asking for evidence about their claims.  I want to encourage others to do the same.  Even just pick one article and have a go at dissecting it, exposing the lies and then asking Bryan and Lynn for evidence.  Individuals will struggle to challenge WDDTY but as a collective we can raise awareness and hopefully stop people being exposed to nonsense.

I am still waiting for a response from WDDTY.