German Brewing

+1 on Tech's comments
I have done a very thorough review of the Scientific (capital S) literature on HSA, and my observations are similar to what Tech describes, the methods part of their papers is not very good, and most of these papers would have been rejected in other scientific areas on methods alone.

@ Caedus - nice meeting you
I do not think one can make a distinction between capital S Science vs lowcaps science. There is no such a thing (to me). We cannot make antimatter at home, but it is not beyond us to brew very good beer, which is the media in which we test our experiments; and I do think that blog-posts can be considered science, if they follow science rules.
I was not referring to Brulosophy itself (or the IGOR program), but I wish for homebrewers to realize that if they wand to test a hypothesis, they have much better odds if they follow some simple rules (see signal to noise ratio comments) ; and many Brulosophy experiments do not help in this regard, inasmuch as many times they do not critically choose the best style for their experiment (minimize noise) nor they do not assess whether the control beer is a good representation of the style (minimize noise); among other things. Also, if you have done research, you know that sometimes you mess up; if you mess up, just do it again, do not proceed to tasting, and do not publish it. On the other hand, some Brulosophy material is very good; the first Brulosophy podcast on kettle trub has an excellent literature review by Malcolm Frazer.

Well said techbrau.

I routinely see three issues with brewing experiments. I won't be shy to admit that all of these are routinely present in Brulosophy articles. Being the King is not easy.

1) Being unable to prove correlation does not equal disproving correlation. This bleeds into the "sum of the parts" arguments. If my process or beer is able to hide a flaw from an introduced variable (and I am unaware of this philosophy), I could be quick to say "X variable does not change my beer"

2) Triangle tests only prove so much and the result is often heavily interpreted. For instance, trying to decide if people can tell the difference between two hops. It's totally possible that drinking these two beers back to back would make it harder, not easier, to tell the difference between the two. If I woke up one morning and was asked, is this beer singly hopped with Citra or Mosaic? I would probably perform better than if I was given three glasses of heavily hopped beer and asked to tell them apart. The human tongue (and brain) is complicated.

3) If you are searching for "statistical significance" it is certainly possible that you will not find it (even if it truly is), especially when using a single metric. Statistics is more than just a one or two or even three trick pony. To go back to the hop example, if I fail to reach statistical significance on a group of blind triangle testers, I may be inclined to say there is no significance. However, what happens when you re-test those who got the difference? Do they repeat the feat? If so, this is very valuable data. Perhaps a majority of people cannot taste the difference, but some indeed can. A single triangle test does not cover this.

I leave this to the academics in the respective departments who have the necessary intellectual foundations and equipment.

I think one other thing to consider is a repeated measures design. One snapshot in time is just that. It doesn't capture the evolution of flavors or help reveal at what point in the process flavors change.

Well, what else can an amateur do? Triangle tests are reasonably easy to arrange (people love free beer), but when it comes to accurate and OBJECTIVE data (ie correct measurements, real and reliable data), then that becomes astronomically expensive really fast.

Not that I am against triangle tests or homebrewers doing experiments for as long that the inherent limitations are understood and accepted.

I'm not advocating any particular test, be it analytical or organoleptic. Whichever you select, then repeatedly measure using that method over time with a batch to get multiple data points along the process. If you select triangle tests, for example, then perform them in the mash, post-boil, post-ferment, after spunding (or force carbonating), after lagering for 1 week, 2 weeks, 4 weeks and 8 weeks. Sometimes, you may lose quality long before the variable you were originally testing. Say for example, you want to test Narziss cold vs Narziss warm fermentations, but you've already worked over, oxidized and/or heat stressed the wort. If you tested repeatedly along the process, then you would know the results were affected long before you even got to the fermentation. If your test is analytical, say measuring DO or sulfite consumption, then the same applies. You would see problems with the system long before you got to the intended control vs treatment experiment. This harkens back to the holistic approach to experimental design and testing.