There are many things that quite a lot of people now know are complete nonsense … astrology, homoeopathy psychics, etc… Why? Well because when you test them in a truly double-blind scientific manner you eliminate the normal human bias, and soon discover in a consistent manner that the claim asserted is not true. You will of course still have true believers for whom no quantity of evidence will change their minds, most probably because they are too emotionally and financially invested in the concept, and so excuses will be offered up … moon in wrong position, you had the wrong vibrations and blocked the energy, or some other similar bit of meaningless babble that enables them to reject such testing.
Ah but what happens when presented with something that is well-established and also believed by many – to be specific wine-tasting. There is an article in the Observer that blows the entire concept out of the water and because of it I’ve changed my mind. I’ve personally always bought into the idea that there were experts out there and assumed that my lack of wine-tasting skill was just that, and so assumed that when it came to the detection of what constituted a good wine from an average, my lack of ability was simply me being a bit of an untrained wine Luddite.
Apparently not, it is not just me, even the top ranking wine experts can be completely bamboozled.
David Derbyshire writes in the Observer about Robert Hodgson, owner of a Californian winery who puts his wines into various state competitions and observed rather inconsistent results …
some whites rated as gold medallists in one contest do badly in another. Reds adored by some panels are dismissed by others. Over the decades Hodgson, a softly spoken retired oceanographer, became curious. Judging wines is by its nature subjective, but the awards appeared to be handed out at random.
“This is all a bit suspicious”, he thinks to himself, and since he was now retired and had a good background in statistics, he decided to put it all to the test …
Hodgson approached the organisers of the California State Fair wine competition, the oldest contest of its kind in North America, and proposed an experiment for their annual June tasting sessions.
Each panel of four judges would be presented with their usual “flight” of samples to sniff, sip and slurp. But some wines would be presented to the panel three times, poured from the same bottle each time. The results would be compiled and analysed to see whether wine testing really is scientific.
The first experiment took place in 2005. The last was in Sacramento earlier this month. Hodgson’s findings have stunned the wine industry. Over the years he has shown again and again that even trained, professional palates are terrible at judging wine.
“The results are disturbing,” says Hodgson from the Fieldbrook Wineryin Humboldt County, described by its owner as a rural paradise. “Only about 10% of judges are consistent and those judges who were consistent one year were ordinary the next year.
“Chance has a great deal to do with the awards that wines win.”
You can see what comes next. A lot of people have a lot invested in the idea and so there are the usual justifications that start flying about.
“Oh but those judges were not true experts”
Wrong, those tested read like a who’s who of the American wine industry from winemakers, sommeliers, critics and buyers to wine consultants and academics.
In fact their ability to detect the same wine was quite laughable, when he presented the judges with the exact same wine from out of the same bottle minutes apart, but told them it was different, they rated it differently. A wine deemed to be a good 90 would be rated as an acceptable 86 by the same judge minutes later and then an excellent 94.
“Ah but they were more or less right, it was only a few points”
True, but even a few points matters and can greatly influence the price and the market.
As you might imagine, many label this to all be bullshit (we are back to that emotional/financial investment once again), so it is interesting to see if Mr Hodgson is the lone skeptic here. Is he? No, not at all …
French academic Frédéric Brochet tested the effect of labels in 2001. He presented the same Bordeaux superior wine to 57 volunteers a week apart and in two different bottles – one for a table wine, the other for a grand cru.
The tasters were fooled.
…In 2008 a study of 6,000 blind tastings by Robin Goldstein in the Journal of Wine Economics found a positive link between the price of wine and the amount people enjoyed it. But the link only existed for people trained to detect the elements of wine that make them expensive.
In 2011 Professor Richard Wiseman, a psychologist (and former professional magician) at Hertfordshire University invited 578 people to comment on a range of red and white wines, varying from £3.49 for a claret to £30 for champagne, and tasted blind.
People could tell the difference between wines under £5 and those above £10 only 53% of the time for whites and only 47% of the time for reds. Overall they would have been just as a successful flipping a coin to guess.
The article then goes on to ask the question as to why so many are fooled like this and suggestes that it may have something to do with the complexity of wine … 400 aroma compounds, 23 different types of alcohol, etc…
In the end the truth appears to be that what constitutes a “good” wine is very subjective and not in any way objective at all. It means that you can pay £100 and discover it is something you hate, or pay £5 and be delighted to find something you love, and so perhaps the best way forward is to just try, and if you like it, then great and if you don’t, well no matter who pronounces it to be “great”, just ignore them.
Meanwhile, is there any hope of finding a reliable judge of wine? Apparently not …
Robert Hodgson is determined to improve the quality of judging. He has developed a test that will determine whether a judge’s assessment of a blind-tasted glass in a medal competition is better than chance. The research will be presented at a conference in Cape Town this year. But the early findings are not promising.
“So far I’ve yet to find someone who passes,” he says.
Links
- The Observer Article
- A paper on an automatic wine review generator that spews out random wine review words which are apparently as meaningful as the real thing : For example “Fine minerality, dried apricots and cedar characterise this sage-laden wine bursting with black fruit and toasty oak.“