Predicting wine scores from chemistry, and from price and region
I tried to predict wine ratings two ways, from lab chemistry and from price and region. The models beat always guessing the most common score, but not by enough to pick out the rare great bottle.
Given what you can measure about a bottle of wine, can you predict how good someone will say it is? I tried it two ways, with two very different datasets, in spring 2022. The code is on GitHub at despresj/cse_project.
The plan I dropped
I started with a Bayesian approach. I wanted to fit ordinal and hierarchical regression models in Stan and compare them against ordinary GLMs, and partway through I had an ordinal regression with weakly informative priors at about 58% accuracy on an 80/20 split.
The trouble was the priors. A Bayesian model earns its keep when you can say something real before seeing the data, and I had no idea what a reasonable level of volatile acidity is. With vague priors, the models performed about the same as standard machine learning methods and ran much longer. So I switched to scikit-learn.
Two datasets
The first is the UCI wine quality data from a study of Portuguese vinho verde: 6,497 wines (1,599 red, 4,898 white), 11 lab measurements each, and a quality score from 0 to 10 given by professional judges. The paper that introduced the data (Cortez et al., 2009, doi:10.1016/j.dss.2009.05.016) reported roughly 64% accuracy with a support vector machine.
Two things about it matter more than the features. The classes are very unbalanced: scores run from 3 to 9, 43.7% of wines are a 6, and only 5 of the 6,497 got a 9. Any model that says “6” every time is right 43.7% of the time, so that’s the bar to beat, not 1 in 7. And only 5,320 of the 6,497 rows are unique. The duplicates look like several judges giving the same wine the same score.
The second dataset is a Kaggle scrape of Vivino ratings: 13,834 wines with a consumer rating, number of ratings, price, year, wine type (4), country (33), region (861), and winery (3,505). There’s no chemistry, and the raters are a mix of casual and serious drinkers, not trained judges. Winery sits inside region inside country, which calls for a hierarchical model. To fit it into scikit-learn I one-hot encoded the categories into a very sparse matrix instead, and I knew that was a compromise.
Scaling mattered more than I expected
My first models on the chemistry data were slow and bad. Before tuning anything, I held the model fixed (a multinomial logistic regression) and changed only how the features were scaled. Then I refit it on 1,000 random train/validation splits for each method.
With no scaling, mean AUC was 0.590. With a Yeo-Johnson power transform it was 0.733, and the spread across splits shrank from a standard deviation of 0.024 to 0.015. Yeo-Johnson is the Box-Cox transform extended to handle zero and negative values. It only narrowly beat plain standardization (0.726), though. The solver I used (SAGA) is gradient-based and converges poorly when features sit on wildly different scales, and here they do: total sulfur dioxide averages about 116 and density about 0.995. I used Yeo-Johnson for everything after this.
Three models and an ensemble
On the chemistry data I tuned three classifiers with grid-searched cross-validation:
- A regularized multinomial logistic regression, tuning the penalty (L1 or L2) and its strength.
- A support vector machine with an RBF kernel, tuning only the regularization strength.
- XGBoost, tuning tree depth, column subsampling, and learning rate.
Then I combined them in a soft-voting ensemble, with XGBoost weighted three times as heavily as the others because it was clearly the strongest. Here are the results from the saved run on the 1,300 held-out wines (20% of the data):
| Model | Accuracy | AUC | RMSE |
|---|---|---|---|
| XGBoost | 63.8% | 0.836 | 0.719 |
| Logistic regression | 51.5% | 0.760 | 0.805 |
| SVM | n/a | 0.758 | n/a |
| Ensemble (1:3:1) | 60.3% | 0.838 | 0.746 |
Only AUC is reported for the SVM. The final report lists the ensemble at 67.3% accuracy and 0.889 AUC, from a run whose outputs I don’t have; the numbers here come from the saved run.
In the saved run, XGBoost on its own beat the ensemble on accuracy; even at a 3:1 weight, the two weaker models pulled it down. Every accuracy in the table clears the 43.6% you’d get on this test set by always guessing 6, and nothing clearly beats the original paper.
The table also hides where the errors fall:
The ensemble predicts almost nothing outside 5, 6, and 7. It got none of the 39 wines rated 8 and none of the 43 rated 4. It got 45 of the 216 sevens right and called most of the rest a 6. It was within one point 95.1% of the time, which sounds better than it is when nearly every prediction lands in the middle. The wines a buyer would care about most, the unusually good and bad ones, are exactly the ones it can’t pick out.
Price, year, and region
For the Vivino data I used the same Yeo-Johnson preprocessing with regression models instead of classifiers. RMSE was about 0.91 for the Vivino models and about 0.75 for the chemistry ensemble.
Those two numbers aren’t directly comparable. The targets are on different scales (a judge’s 0 to 10 score versus a consumer average on a 5-point scale, and the regressor’s target went through its own transform), and the rows are different wines rated by different people. The comparison I actually wanted, chemistry and price and region for the same wines, wasn’t possible with the data I had.
Where that leaves it
Wine quality can be predicted somewhat better than guessing, but not well enough to find the rare great bottle. The chemistry models clear the roughly 44% you’d get by always predicting 6, and they lean heavily toward the middle scores.