Bayesian methods conform to the Likelihood Principle, while frequentist methods do not. Thus, proofs of the Likelihood Principle such as Birnbaum’s (1962) appear to be threats to frequentist positions. Deborah Mayo has recently argued Birnbaum’s proof is in fact no threat to frequentist positions because it is not valid for a sampling theorist. I agree that Birnbaum’s proof is not valid for a sampling theorist, in the sense that replacing his premises with corresponding claims about what sampling distribution ought to be used for inference blocks the proof. However, I do not accept either that Birnbaum’s premises are false or that his proof as originally formulated is invalid. That being said, I do not think that Birnbaum’s proof shows that frequentist methods should not be used.
There are actually at least two different Likelihood Principles: one, which I call the Evidential Likelihood Principle, says that the evidential meaning of an experimental outcome with respect to a set of hypotheses depends only its likelihood function for those hypothesis (i.e., the function that maps each of those hypotheses to the probability it assigns to that outcome, defined up to a constant of proportionality); the other, which I call the Methodological Likelihood Principle, says that a statistical method should not be used if it can generate different conclusions from outcomes that have the same likelihood function, without a relevant difference in utilities or prior probabilities.
Birnbaum’s proof is a proof of the Evidential Likelihood Principle. It is often taken to show that frequentist methods should not be used, but the Evidential Likelihood Principle does not imply that claim. The Methodological Likelihood Principle does imply that claim, but Birnbaum’s proof—at least as originally presented—is not a proof of the Methodological Likelihood Principle.
There are two ways one might respond on behalf of the claim that Birnbaum’s proof does show that frequentist methods should not be used. The first is to argue for an additional premise that would allow one to derive the Methodological Likelihood Principle from the Evidential Likelihood Principle. The second is to argue that my point is pedantic because Birnbaum’s proof might as well be recast as a proof of the Methodological Likelihood Principle. I will consider each of these responses in turn.
One could derive the Methodological Likelihood Principle from the Evidential Likelihood Principle by invoking what I call the Evidential Equivalence Norm, which says that a statistical method should not be used if it can generate different conclusions from outcomes that are evidentially equivalent, without a relevant difference in utilities or prior probabilities.
I admit that the Evidential Equivalence Norm has substantial intuitive appeal. It seems to say less than Hume’s dictum, “A wise man proportions his belief to the evidence,” which is often regarded as a truism. But is this truism actually true? I don’t know how to justify a norm except by showing that it is an effective means to some desired end. I am sure that there are situations in which proportioning one’s belief to the evidence in some sense is the best means to an important epistemic end, but it’s not obvious to me that all situations have that character. Because I accept the Evidential Likelihood Principle, proportioning one’s belief to the evidence means for me using a method of belief updating that conforms to the Likelihood Principle. Bayesian conditioning is not the only such method, but it is the only one I know of that seems worth taking seriously. And there are situations in which I am not at all inclined to think that using Bayesian conditioning is a sensible thing to do. Take, for instance, the search for the Higgs boson. One could try to use the data generated at CERN to update one’s subjective prior probability that the Higgs exists in the reported energy range, but it’s hard to see why we should think that doing so will serve the goal of arriving at a true (or approximately truthlike) theory given that we seem to have no reasonable basis at all for choosing our priors in this case. I am inclined to agree with Larry Wasserman that frequentist hypothesis testing is exactly the right tool for this case, despite its shortcomings.
I am of course running roughshod over a number of subtle and important issues. My point is only that the Evidential Equivalence Norm cannot be taken for granted. If norms are justified only insofar as they help us achieve our ends, then pointing out that the Evidential Equivalence Norm appeals to our intuitions is not enough to show that it is justified.
A second way to respond to my claim that Birnbaum’s proof is no threat to frequentist methods because it only establishes the Evidential Likelihood Principle is to claim that it would be unproblematic to recast Birnbaum’s proof as a proof of the Methodological Likelihood Principle. One would simply have to reformulate Birnbaum’s premises (what he calls the Sufficiency and Conditionality Principles) in a methodological vein. Thus, the Sufficiency Principle would become the following:
A statistical method should not be used if it can generate different conclusions from outcomes that give the same value for a sufficient statistic, without a relevant difference in utilities or prior probabilities.
And the Conditionality Principle would become the following:
A statistical method should not be used if it can generate different conclusions from the outcome of a mixture experiment and the corresponding outcome of the component of that mixture experiment that was actually performed.
It is easy to show that these premises do imply the Likelihood Principle. But the same argument I just gave for resisting the Evidential Relevance Norm can be given for resisting these claims: if a norm can be justified only by showing that it helps us achieve desired ends, then the fact that these principles gratify our intuitions is not enough to justify them.
Accepting Birnbaum’s proof of the Likelihood Principle does require frequentists to give up the claim that their methods respect evidential equivalence. It does not require them to give up the claim that their methods are, at least in some cases, the best ones we have.
Want to keep up with new posts without having to check for them manually? Use the sidebar on the left to sign up for updates via email or RSS feed!