In the shopping centre ...

As Jenny guided her trolley along the aisle, she heard a familiar voice coming from the other side of the cereals. It was her cousin, Elise, chatting to someone.

‘... Celbequine, Jack. Wonderful for arthritis too. It’s really worked for my tennis elbow.... And it’s completely natural so it can’t do any harm. Just herbs and vitamins. It’s been scientifically proven to cure people.’

‘Is that so eh? Maybe I should try it. My leg’s really been playing up lately. So where do I get it? How much does it cost?’

‘To be honest, Jack, it’s not cheap. But I look at this way:

‘Three months’ supply costs the same as a visit to my physio. Look, I’ve got a brochure here, tells you exactly how it works. It says: “Stimulates the body’s own immune system with a combination of herbs and vitamins and helps to relieve pain. Contains extracts of celery and barley, calcium, beta-carotene, vitamin A, vitamin D .... Research shows that this blend of powerful ingredients prevents cartilage loss and slows down the progression of arthritis.” It also says: “May cause nausea, stomach pain or diarrhoea.”

Anyway, it works for me.’

Jack thanked Elise and promised to think about the tablets.

Jenny continued her shopping, thinking about what she had just overheard. It was something she often thought about: how easy it is to convince people that something works by describing how it is supposed to work. In her 8 years as a practitioner she had come across this phenomenon all too often. And not just with her patients – she had seen many pharmaceutical company representatives use similar arguments when trying to convince her to prescribe their products.

Now Jenny happened to know Jack, and she was worried that he might buy the tablets, which she knew he could not afford, having recently been laid off work. It occurred to her that the pain in his leg might not even be caused by arthritis; it could be sciatic pain arising from an old back injury. In addition, he had previously undergone surgery for a stomach ulcer, which meant that he may be at high risk of suffering the tablets’ side effect of stomach irritation. In short, Jenny knew that, even if these tablets did relieve Jack’s leg pain, the harms might outweigh the benefit.

Jenny bumped into Elise in the next aisle, and invited her for a coffee after they’d finished shopping.

How believable are the claims? ...

Later, over coffee, Jenny admitted overhearing some of the conversation with Jack, and asked about the brochure’s claim that a trial reported 130 people were helped by the tablets. ‘But do they tell you how many people were treated altogether?’

‘Not that I recall,’ said Elise. ‘But why do you ask?’

‘Well, if 150 people were treated and 130 improved, that’s pretty good. But if thousands were treated and only 130 improved, it’s a different matter. That means a lot of people took tablets for no benefit, yet risking the known side effects, not to mention those that are not yet known. It’s important to know, not just how many got better, but how many were treated.’

Elise was intrigued: ‘I get what you’re saying, Jen, but the fact of the matter is that I felt better on those tablets, so they obviously work.’

‘There are many reasons why you could be feeling better. You might just have improved anyway, which happens more often with health problems than many people realise. And people tend to feel better when they have taken some positive action, even if the action itself has no effect. The point is, when health claims are made about any product, the research supporting these claims must be valid and the potential benefits and harms should be clearly described so consumers can make an informed choice.’

About the trial ...

After Jenny had seen her last patient later that day, she found Elise in her waiting room. Elise had rung the manufacturer to find out more about the trial, and was excited by the results. The 130 helped by the treatment were out of a group of 250, and the manufacturer said this meant about 50 per cent of people could expect to improve on the treatment. This is how the research was described to Elise:

During the first week of the opening of a new mall, free samples of Celbequine were handed out to shoppers who said they had joint pain or arthritis. Each sample of 25 tablets was enough for one week’s treatment, after which the shoppers were invited to re-order free supplies each month over a 6-month period. At the end of 5 months, there were still 525 people in the trial. They were asked to fill in a questionnaire to receive their final quota of free tablets. A competition offering a prize of gym membership was included with the questionnaire.

What transpired was that only 250 of the 525 questionnaires were completed and returned. Out of these, 130 indicated ‘considerable improvement’ within 6 months of starting the treatment. Elise calculated that 130 out of 250 meant about a 50 per cent follow-up rate, as stated by the manufacturer.

But Jenny was not so sure. ‘These figures could be important, but I’m still sceptical. First, you have to understand that pain from arthritis – or from many conditions for that matter – is not constant. It often fluctuates. Many of my patients with arthritis have long periods relatively pain free. They just improve spontaneously without any treatment.’

Jenny scribbled some figures as she continued: ‘They told you 130 people got better out of 250 who filled out the questionnaires. What about the other 275 people? Maybe they couldn’t be bothered to fill out the forms because they thought the treatment useless? You can see what will happen if we include them in the calculations.’

‘Yes,’ said Elise. ‘It might mean that only 130 out of 525 got better. That’s ... less than 25 per cent who felt any benefit. But, wait a minute, what if it worked the other way? What if the people who didn’t respond did feel some improvement but were then less likely to fill in the forms because they weren’t so conscious of their aches? That could mean that three-quarters of people felt some benefit. But how do we know which is correct?’

‘That’s the point exactly,’ said Jenny. ‘We don’t know. It’s very difficult to draw any firm conclusions from this sort of study. Suppose the prize of gym membership influenced who was likely to send in the form. What if it meant that people who were feeling fit and well were more likely to send it back? Or you could argue the other way: that the healthier, fitter people are less likely to be interested in such a prize because they already go to the gym. Either way, you could argue that the study was more likely to include certain types of people. This is what researchers call selection bias, and makes it difficult to know whether the treatment really works.’

Elise interrupted: ‘So what you’re really saying is that we don’t know whether the tablets made 25 per cent of people feel better or 75 per cent of people feel better?’

‘And we don’t even know that,’ continued Jenny. ‘Remember that many hundreds more people, maybe even thousands, were given samples. Why did all those other people drop out of the survey? Maybe they suffered stomach pains or other side effects. Again, this is selection bias. As we don’t know what happened to everyone in the trial, we have no way of knowing whether the tablets worked.’

Some things get better on their own ...

Jenny could see that much of this was new to Elise. She wanted to finish her train of thought, so she went on: ‘There is a very important phenomenon that is often overlooked. Researchers call it spontaneous remission, but all it means is that, in many instances, time heals. Our bodies have a marvellous capacity for recovery. For a lot of conditions, people get better without any treatment.

‘So getting back to the question of whether there is good evidence that Celbequine is effective: how do we know if the people who reported improvement were responding to these tablets or perhaps to something else that they were taking, or whether the pain just got better on its own, which might have happened even without Celbequine? Don’t forget that these are people who were well enough to be walking around a shopping centre when the trial began, which means their symptoms may not have been too bad to start off with. If that was the case, there’s a good chance they may have recovered on their own, with no treatment at all.’

The placebo effect ...

Reaching for a book from the shelf, Jenny continued: ‘Now there’s something else that often gets in the way when you’re trying to judge the effects of a treatment. Let me read you something from this book by Norman Cousins called Anatomy of an Illness as Perceived by the Patient. It’s a marvellous account of how he dealt with a very serious disease in a most unconventional way.

‘This is the part where he’s describing a placebo:

“A striking example of the doctor’s role in making a placebo work can be seen in an experiment in which patients with bleeding ulcers were divided into two groups. Members of the first group were informed by the doctor that a new drug had just been developed that would undoubtedly produce relief. The second group was told by the nurses that a new experimental drug would be administered, but that very little was known about its effects. Seventy per cent of the people in the first group received sufficient relief from their ulcers. Only 25 per cent of the patients in the second group experienced similar benefit. Both groups had been given the identical ‘drug’ – a placebo.”’¹

Elise thought for a moment. ‘So the implication is that the people in the trial could have been improving just because they were told they would – because they believed the tablets worked.’

‘Yes, there’s certainly the strong possibility that some were responding to the placebo effect,’ said Jenny. ‘The mind has mysterious powers. Sometimes believing is seeing! The placebo effect is very helpful but we want to know whether an intervention that has some risks and costs has an effect over and above its placebo effect.’

And other study flaws ...

‘And there’s another problem with the way that this study was done. Having accepted free samples of the product, I bet not many people could have said it did absolutely nothing to make them feel better, let alone that it made them feel sick. I’m not saying people deliberately lie, but there’s often a temptation to be more positive than one might genuinely be feeling. This is sometimes called “acquiescence bias” and is an example of measurement bias. Imagine you’re in the supermarket and you’re offered a sample slice of a lemon meringue pie that you accept. The salesperson asks how you like it, with one of those smiles that says “isn’t it just too delicious for words”. Many people would find it difficult to say otherwise.

‘So getting back to the questionnaires, we’ve seen there could be several biases: one in the way people interpret and report their health outcomes, called measurement bias, and another in the exclusion of who knows how many hundreds of people who originally entered the trial – selection bias. The bottom line is that there is still no sound evidence that Celbequine does more good than harm.’

Elise was a little hesitant: ‘I follow what you’ve said, Jen, but are you telling me there is absolutely no value in the testimonies of those 130 people who thought the treatment helped them?’

‘What I’m saying is that when you’re making an important decision about whether some treatment is effective or not, your judgement should be based on stronger evidence than the personal testimonies of just a few people who took the intervention. This is true whether you’re a researcher doing a study on the effect of an intervention, or a consumer, or a practitioner advising a patient.’

Personal experiences can be important ...

‘But your question is valid. Are anecdotes based on personal experience ever valuable? The answer is most certainly yes. If you experience a dramatic, immediate change in your symptoms after some treatment for a condition that usually lasts a long time without any treatment, and you experience the same, strong, rapid effect on several subsequent occasions, then your experience provides evidence of the treatment working for you. Let’s say, for example, you have regular migraine attacks that usually last several hours and a new tablet stops the pain within half an hour every time you take it. There’s little doubt the tablet is working, for you anyway. But for most medications, the effects are not that dramatic. For most medications, the improvements we are looking for are more subtle and occur over a longer period. This is where individuals’ reports are of little value.

‘Getting back to Celbequine; the information said it would take up to 6 months to work and also said something about retarding the development of arthritis long term. So in this case, individual reports are not a reliable guide to its efficacy. Remember that individual reports of improvement are no more than that – reports of how people feel after treatment. They do not tell us anything about what may have caused the improvement – whether it was the treatment, something else happening in the person’s life or just the passage of time.

‘Individual reports are of little use when we want to know what some medium-term or long-term change is caused by. That’s another story entirely. To make this kind of deduction, we need probabilistic data, that is, information about the percentage of people who improved. Moreover, we need to compare this percentage with the percentage who would have improved without the treatment.’

Let’s get sceptical ...

Jenny continued slowly and deliberately: ‘So the marketers of Celbequine might have done themselves a favour – if their product truly is as good as they claim – by doing their homework before embarking on a costly bit of research that was clearly full of weaknesses. On the other hand, if their claims are unwarranted, they might not want people to know. At the end of the day, if a product’s claims are genuine, well-designed research can only strengthen the claims, whereas, if there is no valid evidence to back their claims, we should remain sceptical.

‘To make well-informed choices about important decisions that may affect our health – our bodies and our minds – we need much more than opinions. We need evidence.’

They were both silent for a moment, then Jenny said: ‘But if you want my opinion, I think it’s time to call it a day!’

From detergents to treatment for acne ...

When they met again a few days later, Elise was excited as she described her investigations of a new detergent that had been advertised as a breakthrough in ‘enzymatic power’ that ‘gets whites whiter than white’.

As it was more expensive than her regular powder, Elise wanted to test the claim. She divided her whites into two, and used her regular detergent for one load and the new product on the other.

‘And then I thought that this is the way that Celbequine should have been tested,’ she said. ‘Get two groups of people with the same health problem; give one group the treatment and give the others nothing or the old treatment.’

‘Bravo,’ said Jenny. ‘You’ve hit the nail on the head. But to take it a step further, imagine that you were setting up a study for a new acne treatment. What would you do?’

‘You need a bunch of people with acne,’ replied Elise. ‘Teenagers. You could approach high schools or advertise in teenage magazines asking for volunteers. And then divide them into two groups. One group is given the treatment and the other ... what do they get? I mean what’s in it for the volunteers if they’re not getting the new treatment?’

‘Good point,’ said Jenny. ‘What say we tell the teenagers that we’re testing a new acne lotion. We don’t know whether it works or not but you could help us find out by taking part in a trial. The trial will work like this: we spin a coin – heads, you get the treatment, tails you get some other lotion that looks, feels, smells like the treatment but is inert, a placebo. If the new treatment is shown to work, we will offer a free course of treatment at the end of the trial to everyone who was given the placebo.

‘Doing it this way, you see, excludes other variables that might affect the outcome. For instance, the massaging action of applying the cream might do some good – or some bad for that matter.’

Elise was hooked. ‘And remember what you read me about the placebo effect? If the volunteers aren’t told whether they’re getting the treatment or placebo, this will stop their expectations influencing the results. No measurement bias, right?’

Randomised and blinded ...

‘Right! And there’s another thing. In a comparative study, it’s essential that the groups are similar if you want the results to be valid. A good way to ensure this is to allocate the treatment randomly. You’ve heard of randomised controlled trials, haven’t you? Well, randomisation addresses the possibility that those who did not get the treatment were sicker than those who did – or vice versa. Randomisation, or random allocation as it is sometimes called, can be done by the toss of a coin, or by other techniques – computers can be used to allocate patients randomly to a treatment or placebo.

‘Each volunteer is randomly allocated to the new treatment or to the placebo; in addition, as you’ve already said, people shouldn’t know what group they’re in. In other words, they should be masked or “blinded” to whether they are getting the treatment or placebo. This is the way drug licensing authorities assess new claims about the effects of drugs.’

The two women sat in thought for a while. Then Jenny went on:

‘Listen Elise, obviously you and I can’t rush off and do a randomised controlled trial on every new pill or powder on the market, but the healthcare system is continuously involved in studies of all types. There are researchers out there doing randomised controlled trials all the time ... or RCTs as they are sometimes called. What consumers should be doing, however, is asking their practitioners for evidence supporting their recommendations about treatments or any tests – especially for important decisions. The onus is on the person or organisation that recommends the product or the treatment or the service to supply sound evidence that it improves or prolongs life. This includes practitioners from all areas and doctrines of healthcare, and the pharmaceutical companies who recommend their products.

‘Thousands of studies are published every year – though not all of them are randomised controlled trials – and are accessible to practitioners either in journals or in summarised form on electronic databases. So when evidence is available, it should be used.’

Do the benefits outweigh the harms? ...

‘If I were approached by representatives of this company to recommend Celbequine to my patients, I would expect them to provide me with sound evidence that my patients are going to be better off. Failing that, I could do a computerised search to see whether there is any evidence in the medical literature. I could find Medline on the internet, for example, and look at the abstracts of studies published in the most important journals in recent years. Or I could look up the Cochrane Library, which is a regularly updated electronic library of summaries of all the randomised controlled trials (see page 138). The point is that, if I have no valid evidence that the benefits of a treatment outweigh its harms, I should be careful about recommending it. Most practitioners now have access to online computer systems in their offices to search for valid evidence. Mind you, being available on the internet means it can be accessed by anyone.’

Elise thought for a bit. ‘Why didn’t the manufacturer do the study right in the first place?’

‘Now that,’ said Jenny, ‘is a very interesting question. Maybe they don’t know about randomised controlled trials or maybe they think consumers won’t know the difference. It’s just possible, of course, that the stuff doesn’t work and they prefer not to make that knowledge public. Or maybe, good evidence is simply not available. After all, randomised trials can be complex and expensive to conduct.

So does it really work? ...

‘But let’s get back to the question at hand: does Celbequine really work? The only information that we have is not very convincing because it is based on individual reports. This can be misleading for many reasons: first, people often improve spontaneously with time; second, they might be responding to the placebo effect. Of course, it could be that the treatment really does work. But we can’t judge this from the information available. And yet we have this situation where expensive, potentially dangerous interventions are recommended without valid evidence that they work. What we need is good research to supply valid evidence so that all of us can make informed judgements about our health.’

Elise thanked her cousin and left with a mixture of new-found confidence as well as surprise that she had been so poorly informed before. Why, only a few days ago she would have thought herself as well informed as the next person. Then she realised, with some surprise, that she probably had been.

Note: the name ‘Celbequine’ is purely fictitious.

Reference

1.

Cousins N. Anatomy of an Illness as Perceived by a Patient. UK: WW Norton & Co Ltd.; 2005.