The fallacy of the frequentist games here is assuming that SIDS is triggered by a large number of small independent factors which can sometimes add up to a threshold. Having two babies of the same mother die of SIDS strongly implies (but doesn't prove) some common cause which isn't part of that normal distribution of small factors.
An abusive mother is one possibility, but it is not the only one by a long shot. A genetic defect is another possibility. So is a toxic mold or chemical in the home.
The problem with tossing out frequentism entirely is that it borders on throwing out inductive reasoning entirely. Briggs seems to border on Hume. But his arguments are strong enough to have me hitting the books to look carefully at the math that leads to those low p values which turn out not to be all that predictive. I'm guessing that it's overly assuming a normal distribution of noise.
But more generally, I agree with what you're saying. We don't know cause and p-values don't tell us cause in any way. The data is "mildly" inductive, but the problem is that explanation isn't proof. And frequentists forget that there are an infinite number of ways to be wrong in misinterpreting what data "means." You can find an infinite number of hypotheses "consistent with the data," and some will be quite compelling and "feel" right, but that's the same as what the road to hell is paved with. That's why science makes predictions within a given datum and then we judge the strength of the hypothesis by its predictive value. P-values (most times) offer no prediction and claim what they're not supposed to: that the p = .0095 assuming the null is true, blah blah blah, more logical fallacies I'll write about in the next one, but current statistics as practiced and taught now are a mess. An absolute mess, and that is not an accident. It's exactly what one would expect to come out of frequentists statistics that allows inferential leaps at the magical "wee pee" (as Briggs would say) of p = .0095.
Of course nothing is replicable. There's nothing to replicate.
Low p-values are not predictive. P-value purists ike Fisher would also tell you that probability is ontological - it inheres in objects and events - and will speak of "random chance" as if it is some kind of "force" of the universe. Frequentists also tend to be Humean, like Popper, in that they're deductivists and/or falsificationists. Thayt's why they create things like "special rules" (as Popper called them) - i.e. "significance testing" - or turn science in to "consensus."
Jaynes preferred that frequentists not use the word probability at all because they're not dealing with probability, as understood by Henri Laplace, Harold Jeffries, Claude Shannon, R.T. Cox, Polya, Jaynes, and many other less famous minds who worked in radar, for example, at places like Hughes Aircraft.
Briggs is absolutely not a Hume disciple at all; quite the contrary. Hume denied induction while Briggs knows all "common" (objective) knowledge is the fruit of induction, only later to be "proved" deductively.
I'm still getting caught up in such things. I've been mucking with probability distributions, expectations values and whatnot since attending Bob's School of Quantum Mechanics. And I've done a fair amount of estimation theory from my satellite tracking days. But I've never taken an undergraduate statistics class or seen the derivation of p values and whatnot.
Currently reading M.G. Bulmer's very readable "Principles of Statistics" to get up to speed in this debate. But I need to look at the math with a bit of care before I can have a real opinion.
Heisenberg's Uncertainty Principle coupled with classical chaos teach us that nature provides a bounty of random number generators.
But the existence of nonlinearity and unmodeled deterministic features of systems teach that we should not expect all noise to be neat normal distributions. When dealing with molecules on the order of Avogadro's Number, the law of large numbers is our friend. When dealing with a medical condition where the dominant factors number in the dozens, it's an entirely different game. I want to see how standard tests account for this.
Half a lifetime ago my first job after graduating from Bob's School of Quantum Mechanics was doing QA work on the GPS system. The measurement residuals (measured ranges minus predicted ranges) were neither pure noise nor did they for nice Gaussian histograms. There were outliers. There were sequences of clean unmodeled signal.
And this is for a system which was modeled down to General Relativity corrections for the clocks and light pressure on the satellites. Experiments on less well known systems like the human body...
I still have homework to do. My intuition tells me that there is a less deeply philosophical objection to p values out there, but I don't yet have the chops to back up that intuition.
Depending upon your appetite for math and history, Aubrey Clayton's book "Bernoulli's Fallacy" is a very thorough tour through the mathematical history that leads to (what to the modern eye looks like) the Bayesian vs. non-Bayesian/frequentists camps. In reality, it's much deeper than that.
Another place to drink deeply is from Jaynes himself; "Probability Theory: the Logic of Science" is the work of an extraordinary mind, one of Oppenheimer's two graduate students who left the west coast and went to NJ to set up things there. Jaynes was the youngest invitee to Einstein's 70th birthday party. He might be one of the few people with the chops and the resume to openly mock the vote of the Fifth (1927) Solvay Conference.
And if you really want to read a takedown of Popper and what came to be the predominant academic view of science than David Stove's "Scientific Irrationalism: Origins of a PostModern Cult" is a requisite; not long by page count, but it takes some time to work through and fully understand.
It got late and I was too tired to follow the math in Bulmer, so I switched over to Bernoulli's Fallacy the other night and found the writing clear and entertaining in the preface. Started reading the introduction today and OOPS! 10 yard penalty against the home team (page 9).
The urn problem as stated is no different than computing the odds of a full house on a five card deal in poker. It is the same category of problem. Exactly. Starting with the hypothesis of only two colors of stones and that the sampling is random (analogous to the cards being fairly shuffled in the poker problem), you can most definitely compute the odds of "drawing a hand" that is within a desired error tolerance of the ratio of pebbles in the urn. And since the odds of exceeding the tolerance can be mathematically proven to decrease as the number of samples increases, we have as close to a scientific theory as one can ever get.*
It all spills out from the binomial distribution. Chapter 1 of my Stat Phys class at Bob's School of Quantum Mechanics.
The real question is: is the urn problem generalizable, or is it a case of approximating the bull as an infinitely conductive sphere?
_________
*ALL science is subject to the possibility that gremlins are messing with us in the laboratory, or that the laws of physics might fundamentally change tomorrow. See Jack Vance's most excellent short story "The Men Return."
I got a copy of Bernoulli's Fallacy when doing my order of affordable stat books. But I wanted to start my reading with something more mainstream.
The selection on Amazon was dominated by Statistics for People Who Hate Math types of books and over large statistics textbooks that start at $150. So I scooped up what Dover had to offer.
Bulmer's book is turning out to be a gem. A breezy educated writing style, at about the level of C. S. Lewis doing theology. But it also includes real math. I'm following it surprisingly well without having to work any problems. Very unusual for a Dover book. And the book presents statistical vs. inductive probability right in the first chapter. A good foundation before going deeper.
I probably won't go full on deep philosophically. As an unfrozen caveman physicist, I just want enough background to code up some interesting use cases. I want to create algorithms which create hypothetical experiments that are likely to produce bad p values. Should be more persuasive to people who don't have a Jesuit education than the sorts of arguments I'm reading on Substack. The percentage of educated people who have read Aristotle is shrinking.
"The problem is that equations seem to exert a kind of magical thrall over the human mind..."
Boy, howdy, is that true.
I find myself drawn in by Bayes' theorem, too.
The fallacy of the frequentist games here is assuming that SIDS is triggered by a large number of small independent factors which can sometimes add up to a threshold. Having two babies of the same mother die of SIDS strongly implies (but doesn't prove) some common cause which isn't part of that normal distribution of small factors.
An abusive mother is one possibility, but it is not the only one by a long shot. A genetic defect is another possibility. So is a toxic mold or chemical in the home.
The problem with tossing out frequentism entirely is that it borders on throwing out inductive reasoning entirely. Briggs seems to border on Hume. But his arguments are strong enough to have me hitting the books to look carefully at the math that leads to those low p values which turn out not to be all that predictive. I'm guessing that it's overly assuming a normal distribution of noise.
But more generally, I agree with what you're saying. We don't know cause and p-values don't tell us cause in any way. The data is "mildly" inductive, but the problem is that explanation isn't proof. And frequentists forget that there are an infinite number of ways to be wrong in misinterpreting what data "means." You can find an infinite number of hypotheses "consistent with the data," and some will be quite compelling and "feel" right, but that's the same as what the road to hell is paved with. That's why science makes predictions within a given datum and then we judge the strength of the hypothesis by its predictive value. P-values (most times) offer no prediction and claim what they're not supposed to: that the p = .0095 assuming the null is true, blah blah blah, more logical fallacies I'll write about in the next one, but current statistics as practiced and taught now are a mess. An absolute mess, and that is not an accident. It's exactly what one would expect to come out of frequentists statistics that allows inferential leaps at the magical "wee pee" (as Briggs would say) of p = .0095.
Of course nothing is replicable. There's nothing to replicate.
Low p-values are not predictive. P-value purists ike Fisher would also tell you that probability is ontological - it inheres in objects and events - and will speak of "random chance" as if it is some kind of "force" of the universe. Frequentists also tend to be Humean, like Popper, in that they're deductivists and/or falsificationists. Thayt's why they create things like "special rules" (as Popper called them) - i.e. "significance testing" - or turn science in to "consensus."
Jaynes preferred that frequentists not use the word probability at all because they're not dealing with probability, as understood by Henri Laplace, Harold Jeffries, Claude Shannon, R.T. Cox, Polya, Jaynes, and many other less famous minds who worked in radar, for example, at places like Hughes Aircraft.
Briggs is absolutely not a Hume disciple at all; quite the contrary. Hume denied induction while Briggs knows all "common" (objective) knowledge is the fruit of induction, only later to be "proved" deductively.
I'm still getting caught up in such things. I've been mucking with probability distributions, expectations values and whatnot since attending Bob's School of Quantum Mechanics. And I've done a fair amount of estimation theory from my satellite tracking days. But I've never taken an undergraduate statistics class or seen the derivation of p values and whatnot.
Currently reading M.G. Bulmer's very readable "Principles of Statistics" to get up to speed in this debate. But I need to look at the math with a bit of care before I can have a real opinion.
Heisenberg's Uncertainty Principle coupled with classical chaos teach us that nature provides a bounty of random number generators.
But the existence of nonlinearity and unmodeled deterministic features of systems teach that we should not expect all noise to be neat normal distributions. When dealing with molecules on the order of Avogadro's Number, the law of large numbers is our friend. When dealing with a medical condition where the dominant factors number in the dozens, it's an entirely different game. I want to see how standard tests account for this.
Half a lifetime ago my first job after graduating from Bob's School of Quantum Mechanics was doing QA work on the GPS system. The measurement residuals (measured ranges minus predicted ranges) were neither pure noise nor did they for nice Gaussian histograms. There were outliers. There were sequences of clean unmodeled signal.
And this is for a system which was modeled down to General Relativity corrections for the clocks and light pressure on the satellites. Experiments on less well known systems like the human body...
I still have homework to do. My intuition tells me that there is a less deeply philosophical objection to p values out there, but I don't yet have the chops to back up that intuition.
As you well know, the water gets deep in a hurry.
Depending upon your appetite for math and history, Aubrey Clayton's book "Bernoulli's Fallacy" is a very thorough tour through the mathematical history that leads to (what to the modern eye looks like) the Bayesian vs. non-Bayesian/frequentists camps. In reality, it's much deeper than that.
Another place to drink deeply is from Jaynes himself; "Probability Theory: the Logic of Science" is the work of an extraordinary mind, one of Oppenheimer's two graduate students who left the west coast and went to NJ to set up things there. Jaynes was the youngest invitee to Einstein's 70th birthday party. He might be one of the few people with the chops and the resume to openly mock the vote of the Fifth (1927) Solvay Conference.
And if you really want to read a takedown of Popper and what came to be the predominant academic view of science than David Stove's "Scientific Irrationalism: Origins of a PostModern Cult" is a requisite; not long by page count, but it takes some time to work through and fully understand.
It got late and I was too tired to follow the math in Bulmer, so I switched over to Bernoulli's Fallacy the other night and found the writing clear and entertaining in the preface. Started reading the introduction today and OOPS! 10 yard penalty against the home team (page 9).
The urn problem as stated is no different than computing the odds of a full house on a five card deal in poker. It is the same category of problem. Exactly. Starting with the hypothesis of only two colors of stones and that the sampling is random (analogous to the cards being fairly shuffled in the poker problem), you can most definitely compute the odds of "drawing a hand" that is within a desired error tolerance of the ratio of pebbles in the urn. And since the odds of exceeding the tolerance can be mathematically proven to decrease as the number of samples increases, we have as close to a scientific theory as one can ever get.*
It all spills out from the binomial distribution. Chapter 1 of my Stat Phys class at Bob's School of Quantum Mechanics.
The real question is: is the urn problem generalizable, or is it a case of approximating the bull as an infinitely conductive sphere?
_________
*ALL science is subject to the possibility that gremlins are messing with us in the laboratory, or that the laws of physics might fundamentally change tomorrow. See Jack Vance's most excellent short story "The Men Return."
I got a copy of Bernoulli's Fallacy when doing my order of affordable stat books. But I wanted to start my reading with something more mainstream.
The selection on Amazon was dominated by Statistics for People Who Hate Math types of books and over large statistics textbooks that start at $150. So I scooped up what Dover had to offer.
Bulmer's book is turning out to be a gem. A breezy educated writing style, at about the level of C. S. Lewis doing theology. But it also includes real math. I'm following it surprisingly well without having to work any problems. Very unusual for a Dover book. And the book presents statistical vs. inductive probability right in the first chapter. A good foundation before going deeper.
I probably won't go full on deep philosophically. As an unfrozen caveman physicist, I just want enough background to code up some interesting use cases. I want to create algorithms which create hypothetical experiments that are likely to produce bad p values. Should be more persuasive to people who don't have a Jesuit education than the sorts of arguments I'm reading on Substack. The percentage of educated people who have read Aristotle is shrinking.