Beer experiment philosophy – is your Control my Control?

[lupulus]

For those who do not know me, I have a doctorate degree in physiology, and I am currently working in clinical pharmaceutical research. (This is not bragging, just to tell the reader I have been running / reading / reviewing experiments for a while).
I have been reading many beer/brewing papers and beer experiments in the recent past. Most (homebrew) beer experiments that we have seen in the recent past compare the standard process of that brewer (aka control) for brewing a given style with the same process plus the addition of an experimental variable. In fairness, in research papers many times they compare more than one experimental treatment to a control. Similarly to the mentioned homebrew experiments, when one want to test something at home, one will do the same, brew one beer with the standard process and another beer with the standard process plus the experimental variable and compare them.
Assuming taste is used as the metric to compare the beers, sometimes the comparison will yield a significant difference that the brewer (friends/ colleagues) can taste blindly, sometimes this difference is detected by such a majority that would yield a statistically significant difference, sometimes no significant difference can be detected. When discussing why differences were detected or not, it is important to critically think what can affect the odds of detecting a significant difference. These are some key factors:
1. Brew a beer that you can brew with little or no flaws every time. Flaws can confound the tasters; if flaws are more noticeable than your treatment, the treatment effect will be difficult to detect for most tasters. Also, if two beers that you brew to be identical are different, how can you expect to detect whether the difference is due to the treatment, or to your brewing variability. This is not the time to brew something you have not brewed before.
2. Brew a beer style that maximizes the odds of detecting a difference, use styles with subtle flavors, and especially think about the postulated effect if it the effect is taste-related. For example, do not use IPAs for taste effects, for an astringency effect, weissbier is a good style, for a malty effect, Munich helles may be better than a subtle beer with little malt flavor (light American lager)
3. Use ingredients you have used before for the same style. Use a newly opened bag of fresh hops.
Can your process affect the results?
I think it does. Your process is your brewhouse signature. Two brewers can do everything perfect as stated above, one finds differences between the control and the experimental treatment and others does not. What happened? Many times, brewers get to a great beer through different processes. Some use little carbonates in stouts, some use high-bicarbonate water; some get maltiness through decoction, some get maltiness using specialty malts in an infusion mass; some add dark grains at the beginning, some at the end of the mash; some boil for 120 min, some only for 60 min; some do thorough cold break separations; others not so much; some overpitch with no problem, some underpitch with no problem. An experimental treatment that can make a significant difference in some great brewer’s process, may not make a difference in other great brewer’s process.
Would you add other factors to this list? Would you amend this list. Your comments, thoughts, and corrections will be genuinely appreciated.
Cheers,

[Owenbräu]

I definitely agree with this, and agree there are more reasons stated than above.

For example, I have a 5 year old who loves mac n cheese. However, she likes my mac n cheese better than my wife's. It comes from a box. It's boiled noodles, cheese, butter and milk. How could they possibly be different? Well...

My wife makes the noodles a little al dente. Plus, when it's time to add the butter, milk and cheese mix, she turns the stove off, mixes and stirs. The noodles taste very much like raw butter and the cheese is still a bit gritty. I leave the burner on when I add the ingredients. I continue to stir over heat until the butter is completely melted and the cheese, butter and milk become thick and creamy. My noodles are cooked a bit longer, and it tastes more cheesy and less buttery. Same recipe. Same box of ingredients. Same milk and butter from the fridge. Yet, two very different presentations of mac n cheese.

The same thing happens in brewing. Even on dual, identical systems, your time and attention are split, and you may unintentionally alter variables. At the same time, I never understood why someone would brew two unique batches when a split batch is more appropriate and eliminates variation, e.g., yeast comparisons. I also have a Ph.D. in Biology, and I am published internationally in peer reviewed journals under extreme scrutiny. There are a lot of variables people test under questionable methodology that just make me scratch my head and ask, "why did they do that?" Are you looking for generalizations, or examples from experiments we have seen online?

[Techbrau]

I'll add two more thoughts:

1) Complex manufacturing processes (such as brewing) are not the sum of their parts. Therefore, the results of changing a single process variable in isolation do not generalize to the brewing process as a whole. Case in point, let's say you wanted to test whether an oxygen permeable fermenter like a plastic bucket produced a different beer than a non-oxygen permeable fermenter like a stainless steel keg. After fermenting both beers to final gravity, you open racked them both to poorly purged kegs and let them sit on gas for a couple weeks to carbonate before taste testing them. Guess what - the amount of oxygen introduced during kegging in all likelihood completely trumped whatever differences may have existed between the exposure in the fermenters. You've introduced a confounding variable whose effect size is likely orders of magnitude higher than the effect you're trying to measure. Therefore, if you saw little to no difference between the finished beers, you can't conclude that in general the difference in oxygen exposure between a bucket and a keg is negligible. But you may end up concluding just that if you were blind to your confounding variable.

Fundamentally, this is why I like experiments that take into account a holistic view of the underlying theory behind a hypothesized phenomenon to really get at the roots of things. I'm not as big of a fan of experiments that tweak the process one variable at a time in a superficial way, because the space of process parameters is combinatorially large and exploring it that way is so inefficient that you are unlikely to really get anywhere. Furthermore, if you make the folly in assuming that your results generalize when they shouldn't due to the impact of confounding variables, then you may end up convincing yourself (and lots of other people) that e.g. HSA Is a myth.

2) Human tasters as a measurement device. People argue this topic to death and there are valid points on both sides of the issue. My personal belief is that this xkcd is relatively accurate: https://xkcd.com/915/

I think that when it comes to most things that involve a subjective sense of "taste", a priori the majority of people are relatively insensitive to subtleties, and to the layman only the most obvious differences are apparent. I do think that if one spends enough time actually paying attention and actively trying to become attuned to the subject matter at hand, over time they can develop sensitivity to subtleties that they would otherwise be unable to perceive.

Similar phenomenon are well-documented e.g. in linguistics. As infant humans when we begin to learn spoken language, over time our brains become attuned to recognize the phonemes that exist in our native language, and often we actually lose the ability to distinguish between phones that do not exist in our native language. A good example of this are native Japanese speakers who learn English as adults. The English /l/ and /r/ sounds do not exist in Japanese, and it has been demonstrated that many of these native Japanese speakers cannot even perceive a difference between an /l/ and and /r/ spoken by a native English speaker. You can read more about this phenomenon here https://en.wikipedia.org/wiki/Perceptio ... e_speakers

To make the connection back to beer triangle tests, if you formed a panel of Japanese speakers and asked them to distinguish between sound recordings of /l/s and /r/s, you would probably fail to find a statistically significant result. Does that result prove that English speakers also would be unable to distinguish between an /l/ and an /r/?

Some people argue that developing experience with tasting beer is nothing more than learning an associative mapping between flavor descriptors and your sensory perceptions, and that the raw sensory "image" perceived by an experienced taster and a novice taster is underlyingly the same. The /l/ and /r/ example above demonstrates that this notion is false, at least in the case of perception in a different sensory modality. The /l/ and /r/ phenomenon can only be explained by different neural pathways forming over time in the brains of people who speak different languages. It is reasonable to assume that the same phenomenon can happen with all forms of perception, since the same neural circuitry is involved. So while a novice and a connoisseur may have the same raw signal representation being fed into their brain (i.e. equally sensitive tongues/tastebuds and olfactory receptors), their neural circuits responsible for perception may differ substantially.

[Crunk]

I love this, I am in the company of some highly educated and intellectual folks. This helps with my learning curve immensely. I don't have a college degree, I graduated from high school, then went into the army.

Sorry for high jacking the topic.

[Brody]

1) Different Process = Different Results: Process wise you often look at an experimenters proceeders and think that you would have done something differently.

2) Shitty Evaluation Setting: Presentation in tiny plastic cups isn't ideal in my mind for evaluating since I find the color, clarity, and head important. Also, the environment may also not be ideal. I've gone to enough "Thank you for your $45, here's your wrist band & tiny cup, go get drunk as hell" beer fests to know they suck for evaluating. Would rather be at home alone with a well poured pint for that.

I think the only way to test a variable and know if it's meaningful to you is to do it on your own system. If one had the time & capacity to brew the same beer with a few tweaks we could learn a lot. I've done a couple but it can be a pain trying to manage. So filling in the gaps with literature and opinions of knowledgable folk (like everyone here) is key for me.

[Bilsch]

This great information needs to get posted over at the AHA in the lodo exbeeriment thread. If not there then HBT.

[lupulus]

[caedus]

So I want to start out by saying that everyone here are absolutely right in their detailed posts about experiments. I just want to provide a counter point so this doesn't become a circlejerk, just some food for thought.

Taking Tech's example of the oxygen permeability test: It would absolutely throw off the experiment if we were testing *only* the O2 pickup/time. However, I always felt these experiments were not testing variables in a vacuum. The Brulosophy experiments (let's be honest, while he didn't invent them, that's what we are talking about here) are designed to be a *homebrewer's* experiment. So the variable in question goes from "What is the impact of plastic v. stainless" to "What is the impact of Plastic v. stainless on my homebrew system".

I think the importance of the experiments are shifted once one realizes this. Marshall isn't trying to push out academic papers, and I don't think anyone is expecting blog posts to be considered Science (with the capital S).

Again. I am not disagreeing with any of you. In most Quality Control books I have read there are entire chapters if not sections (and entire degrees worth) of education on how to evaluate data accurately.

[Natebriscoe]

Completely agree with everyone here. I will say that for the most part "homebrew experiments" are meaningless, due to the amount of repeatability most homebrewers/homebrew systems have. Then the tasters, some people can't tell the difference between a pils malt and a pale ale malt.

Sent from my SM-G930V using Tapatalk

[Techbrau]