Testimonials and Research

If you visit the website of any pseudo-scientific practitioner one thing almost always displayed is a list of testimonials – effusive endorsements from previous clients. For example, here’s an extract from the website of Cynde Van Vleet, a Tellington T Touch practitioner from California:

My experience with Cynde and her TTouch work has been nothing but wonderful, rewarding, and informative.  Cynde is a most gentle  and intuitive animal lover who conducts herself professionally at all times, yet exudes warmth and kindness as well.  I have great respect and admiration for Cynde and the work she does with animals. Her knowledge of TTouch and the world of dogs is impressive, and I have learned a lot from her in our brief history together…

Similarly, here’s an example from alleged psychic Philena Bruce’s website:

“I came to Philena when I was in a very dark place. Through her patience, guidance and gentleness, I genuinely left feeling hopeful. She held me together emotionally and spiritually throughout this time. She lit the candle in my mind and let my spirit guide me to light. She has a wonderful personality and in my heart I know our paths were meant to cross.”
K.K., London, Project Manager

In most cases testimonials are intended to establish the credibility of the practitioner. In Philena’s case, the long list she provides also pushes the “entertainment only” disclaimer – which all psychics in the UK are now legally obliged to put on their literature – far out of the view of the casual visitor.

But are testimonials useful? Can we use them to work out how effective a treatment of practitioner is? Should we as consumers pay any attention to testimonials?

The most obvious problem with most testimonials, especially those you might find on websites, is with authenticity. It doesn’t take a huge amount of imagination to make up your own testimonials and in most cases it would be difficult to prove that they were fake. However, I doubt many of the testimonials I have seen have been faked. It seems more likely that they came from customers who were genuinely pleased with the service. I’ve no doubt that they have plenty of satisfied customers – I’ve met some of them.

However, we should keep in mind that testimonials are highly selective. They give us no indication of the percentage of customers who were satisfied with the service. Those who were not pleased with the service may not have given any report. They may have felt foolish for having tried it, so rather than write a negative testimonial, they may forget about it and get on with their lives. Even if they did write to complain, surely no one attempting to establish their credibility is going to publish their views. I’ve visited the personal websites of quite a few spiritual healers, fortune tellers and other sellers of pseudo-science, but I have yet to see a single negative testimonial. I think it unlikely that no customer has ever been dissatisfied with the services any of them provided.

How about if we independently collected testimonials from every customer who visited a particular therapist or fortune teller? Even then, we still wouldn’t be able to say with any certainty whether on not they had genuine abilities. We might be able to tell that they were friendly, professional, helpful and so on, but judging whether the treatment worked is something else. It has been well documented that people are more likely to judge something to have been a success once they have invested substantially in it – whether emotionally or financially. I suspect this is why payment or gift-giving always has to be part of the process of Reiki. Perhaps on some subconscious level people say to themselves:

“I’ve paid more than my weekly grocery bill for this treatment. Only a fool would do that for something which doesn’t work. I know I’m not a fool, so it must have worked!”

This effect is brilliantly explained in Carol Tarvis and Elliot Aronson’s book, Mistakes Were Made.

It seems testimonials are a pretty unreliable guide to whether a medical treatment is effective or whether a psychic prediction is accurate. Certainly they are a very poor substitute for research. They are a selective form of anecdotal evidence.

So why do they get used at all? I think there are several reasons.

Firstly, there are times when use of a testimonial is valid. It’s not just treatments with dubious efficacy that print praise from former clients. Testimonials are commonplace in all sorts of marketing – child carers, home builders, restaurants and so on.

In any situation where the customer is in a position to judge whether the service or treatment was effective a testimonial can be useful. For example, if the testimonial relates to a novel, film or restaurant, then the aim is entertainment and the customer is best qualified to say whether they have been entertained.

Secondly, testimonials are easy to understand. A clinical trial is not something that most people are willing to wade through. Doing so takes considerable effort and even an intelligent reader might not be able to judge the strength of the research.

Thirdly, testimonials are easy. All you need is a few satisfied customers to say you’re warm and friendly. A full research project is beyond the means of most independent therapists.

Lastly, testimonials are human. What other people tell us is how we gain most of our information, so it feels quite natural to read what other people say about a service or therapy.

None of these are especially good reasons to trust testimonials.

Mircoscope and clipboardPharmaceutical companies have to put all their products through clinical trials before they can be marketed. I don’t think that other treatments should be subjected to any lesser scrutiny.

In an ideal world each therapists’ techniques would be independently evaluated with something akin to clinical trials, before an official regulator’s endorsement could be given. The next best thing would be for their methods to be independently reviewed and researched to discover whether they can work. However, the money to do this research has to come from somewhere, perhaps a jointly-funded regulatory body whose research and methods are made public.

Alternative therapists have told me that clinical trials are biased and corrupt due to the companies pursuit of profit above all else. There have certainly been some cases where this is true and there’s a case for improving accountability and publication of medical research in general. However, I imagine the situation would be much worse if the pharmaceutical industry could validate new medicines simply by gathering testimonials.

I’ve also been told that using testimonials instead of research is acceptable for alternative treatments because laying your hands on someone or reading their palm doesn’t have harmful side-effects that ingested medicines can. However, the purpose of clinical trials is twofold; to ensure there are no dangerous side-effects and to ensure the treatment is effective. Giving someone an ineffective treatment – even for free – is irresponsible and dangerous in itself.

[BPSDB]

9 thoughts on “Testimonials and Research

  1. Testimonials are most effective if they come from known friends and acquaintances. I’ll take the word of a trusted family member, friend or co-worker over that of a televised talking head any day. Testimonials from people I don’t know are mostly useless, unless they are backed up by other kinds of evidence, such as documented studies, etc.

  2. I’d go one step further, Chappy – why are testimonials from close friends and family more valid than any other? Just because they may have your best interests at heart doesn’t make them right. My grandfather loves me and wants the best for me, no doubt, but when he testifies about how I should head on back to Jebus I smile, nod and think “bullshit”. One of my closest friends is a kinesiologist and naturopath – she’s convinced that kinesiology has cured her of all kinds of ills, and offers it to me every time we meet – but that doesn’t mean I’m going to discount the clinical trials which show it to be a fraud.

    Testimonals and personal recommendations both suffer from the same criticism – the sample size is way to small for any results to be statistically significant. Just as I wouldn’t trust a clinical trial with only one participant, so I’m not going to trust my mate’s recommendation of homeopathy, unless he’s a qulified medical researcher with a stack of RCT results on hand.

  3. yunshui,

    For the examples you give, I completely agree that it makes no difference to how trustworthy the information is. However, I’d trust a book recommendation from a friend (or fellow blogger I knew) more than one on Amazon. Perhaps this is the sort of thing the chaplain means?

  4. If a friend recommends me a book, a foodstuff, a new way of doing the laundry or a board game they think I’d like, I’ll almost always check it out. None of these things (apart maybe from the laundry idea) are assessed on anything other than subjective grounds. Testimonials – even from one’s mates – on issues that can be assessed objectively, using the scientific method, are less valid to me than the real-world objective valuation.

    To put it another way, if I tell you a book is “good”, you’ll assume that I mean I like the plot, or the language, or the cover… You may have a different opinion to me on these issues, but you can accept that by my standards it’s a “good” book. If I say a therapy is “good”, however, you take that to mean it works, it made me better, it reduced my symptoms. Those are testable, scientific claims, and they aren’t decided by opinion.

    I’m sticking with my position. My best mate may credit his continued existence to an invisible man in the sky, but his testimony is meaningless next to our observations of reality.

  5. You stole my thunder, Eshu – I was planning a very similar post for later this month. But now I have a good source to use as a reference. :)

  6. Well that’s one to me… and about 110 to you! I guess you had a head start.

    I’ll look forward to your version.

  7. There’s been quite a bit of psychology research how the psychology of the consumer affects the effectiveness of testimonials. Testimonials are particularly effective when used on people with a high SNI (susceptibility to normative influence) i.e. people with a strong willingness to conform to others expectations and desire to fit in.

  8. Thanks LadyFlasheart. There’s plenty more homework I could do on the psychology of this, which could be an interesting project for another day.

    This makes me think that there are two slightly different properties of testimonials here:

    1. Effectiveness – How likely it is to influence people’s decisions. As Chappie pointed out, testimonials from friends and family are likely to be more “effective” in this respect.
    2. Genuine usefulness – How likely the testimonial is to contain accurate information pointing towards the truth. As Yunshui said, testimonials (whether from friends and family or not) are less likely to point towards the truth than independent evidence.

    I’m sure there should be better terms for those two.

  9. Pingback: Daylight Atheism > How to Think Critically: Testimonials