Why wine ratings are a joke (or...the joke is on us)

I admit it, I am a ratings whore. It seems like I am continually searching out the 94 rated Robert Parker Wine Advocate or the 96 Wine Spectator rated wine that "only" cost $95. Sometimes I get so mad at myself for chasing those whore wines with the knowledge that the three tier distribution system doubled the price of the wine the week before I bought it because they got an advance copy of the magazine with the "high" rating and knew they could get away with taking an obscene price increase because idiots like me will pay for the rating.

Intrinsically I knew there was something wrong with this but I could not put my finger on why. I knew it was not just that I was trusting another person's palate to determine what I thought was "good". It was deeper than that, soul deep. It finally hit me today and it all has to do with the science of the ratings. Screw the fact that people taste differently. The science of rating wines is no science at all.

As a researcher working in the food science area, I utilized statistical experimental design, demanding testing protocol that would yield statistically significant results. Without the proper test design, no statistically valid conclusions can be drawn about your test product. I knew that in developing a new product, I had to be sure to the 95% confidence level that the conclusions I was drawing were right. If I was trying to show that one product was better than the other or if I was trying to draw conclusions regarding the characteristics of my product (subjective (taste) or objective (pH)), I needed to have a statistically valid sampling program so that I tested enough samples to be truly confident of my conclusions.

Now back to wine ratings. There are absolutely NO STATISTICALLY VALID CONCLUSIONS that can be drawn from wine ratings. Parker, Wine Spectator, Tanzer, whomever, they all work off of one bottle, maybe two, provide from thousands, maybe tens of thousands, maybe even millions of bottles of the "same" wine available to us ratings whores. It is pure folly to believe that that one bottle in any way, shape or form represents the batch average of that wine. The chances of it representing the average are about the same as you hitting the PowerBall when the jackpot hits $300 million. Not very good odds, indeed.

What is the cause of the variation and why do you need more than one bottle to draw conclusions. First, the biggest tank you are likely to see in most wine production facilities is maybe a few thousand gallons. Most blending tanks are considerably smaller than this. Even a wine with as few as 100 cases (250 gallons) would like be blended in more than one batch since it was probably stored in 4 or 5 50-gallon oak barrels for aging. This means that most wines are blended in batches, tanks being refilled due to size limitations. At a minimum, to truly have a statistically significant sampling program, you would need to have multiple bottles from each of the blendings, certainly sampled throughout the tank. Add small barrel aging, which is inherently variable, and more bottles would need to be sampled to generate statistically valid results.

But it goes deeper than that. Winemakers know that they have variation, bottle to bottle, caused by the vagaries of blending and mechanically filling bottles as well as the closures used. Mechanical movement of the wine, oxygenation during filling, incomplete or inconsistent flow patterns from tanks and in lines are among the filling issues that contribute variation from bottle to bottle or at best, case to case. Add that to the batch issues described above and closure variation (cork problems and variability) and the number of bottles that need to be tested to have a statistically significant sample size increases dramatically, perhaps exponentially.

This does not even begin to address the statistical issues surrounding a human based measurment system. I admit that it is impossible to develop an objective machine-based system to measure human taste. Maybe in 30 or 50 years, but today, no. I am sure nerds somewhere are workingon it, but I have to believe it is years away. Unlike a machine, human tasters are not all the same and the same taster is not the same from tasting to tasting or even bottle to bottle within the same tasting session. Maybe Jim Laube has a cold or had garlic for dinner last night or has an owie on his tongue. His subjective taste would be changed due to these factors. Palate fatigue is also a common problem, even among highly trained tasters who have been in the business for years. I believe this is why big, huge, overoaked, overextracted wines have become the darlings of the tasting world. Sorry Jimmy Laube and Bobby Parker, but when you deign to taste 30, 40, 50 or more wines at a seating, you know that by the time you hit the 10th, your taste buds are more or less shot. The only wines that could possibly get your attention and score well after that are the ones that hit you over the head with fruit or oak. The germination of the "international style", I feel, was the vintners' response to palate fatigue and their commercial and financial need to have a highly scoring wine to get us ratings whores to buy their wine. The tasters (Parker, Tanzer, Wine Speculator, etc.) are at fault for tasting so many wines at once in a situation that is, at best, untenable, at worst, impossible. And for not demanding statistically significant sample sizes they can be wholly blamed. In their defense (slightly) I understand that huge wine corporation and small winemakers rely on those ratings to get us wine whores like me know what to buy. ARRGGGH How do you win as the consumer?

I have a BS in Chemical Engineering from UC Davis, probably the greatest wine training University in the world (sorry Bordeaux). I wonder what Maynard Amerine or the current faculty have said about the science (or lack thereof) surrounding wine ratings. I hope they have written about it. It is one thing to test a bottle and say that bottle is good. Or maybe tasting a few bottles out of a reasonably small sample, the number of bottles determined by the math, and concluding that based on the scores, that subset of the wine is good. But the more I consider what we accept as "proof" that a wine is good, the more my head hurts and I kick myself in the butt for chasing scores.

So what is a wine whore to do? Good question because it is not feasible to test every wine when you want to buy it unless you are independently wealthy (and few of us wine whores are). The independently wealthy among us buy the ratings anyway and could care less if they make any statistical sense. When you have unlimited resources what do you care what the statistics say? If you don't like it, f..k it, pour it down the drain and get another bottle of something else. So us wine whores have to stop buying ratings. Find producers or regions of the world or wine styles or whatever you like and stick with them.

A perfect example for me is Ridge and Turley. It used to be that any wine Ridge or Turley produced got a big score from the Wine Speculator. Recently that has not been the case. In fact, I think Ridge refused to submit their recent zins to the Speculator for rating because they knew they would get hammered by Jimmy Laube. It seemed to me (and famous vintner Randy Lewis even wrote a letter to the Spectator agreeing with my conclusion) that the Speculator, or Jimmy Laube, had decided that any extracted, high alcohol zin is bad and refused to give them decent ratings, even if they were perfectly balanced, zingy and tasty and deserved the ratings. I had been mad that the wines I was paying good money for were being rated for shit. Now I understand that the ratings mean next to nothing. My palate keeps saying that, by and large, these wines are still excellent and deserve a better fate, no matter what the Speculator says. So I keep buying them because I trust the producers and my taste buds say they are good. Now I have been critical of California Zin, even in this blog, in the recent vintages and I do believe it has been off the pace and variable at times in the 2000's. But there are still plenty of good zins that I have purchased and my taste buds are saying that the wines are improving with age in many cases.

We have to quit buying wines solely based on ratings. I cannot begin to count the hundreds of wines that never lived up to the ratings I purchased. In fact, I would say it is the rare wine that I rate as highly as the experts. This is especially important when you realize that often, the greedy distributors automatically raise the price of any wine that scores a 90, completely ripping off us wine buyers. Now I am not against capitalism, but these suckers are doing it based on a subjective score that we all know is flawed and they were happy to sell the wine at the lower price pre-score. Can you say price gouging? Sound like gasoline to you? Me too.

The truth is that wine ratings can give you a general impression of a wine. But they should never be used for any more than that. We have to realize that wine is a highly variable, highly changeable, highly subjective art form and the best you can hope is that when you get around to drinking it, it meets the taste profile you had hope for when you purchased it. Wine whores of the world, wake up and smell the coffee. Quit chasing that high score because it is likely that the score is flawed or that you won't agree with it anyway. Be your own man (or woman). Your enjoyment of the wine will probably go up and no doubt your pocketbook will thank you.