Neyman and Pearson (e.g. 1933) treat the problem of choosing the best rejection region for a simple-vs.-simple hypothesis test as what computer scientists call a 0/1 knapsack problem. Standard examples of 0/1 knapsack problems are easier to grasp than hypothesis testing problems, so thinking about Neyman-Pearson test construction on analogy with those examples is helpful for developing intuitions. It is also illuminating to think about points of disanalogy between those scenarios and hypothesis testing scenarios, which give rise to possible objections to the Neyman-Pearson approach.
The Standard Bayesian-Frequentist Debate About Stopping Rules
Bayesians generally reject the frequentist view that inference and decision procedures should be sensitive to differences among “stopping rules”—that is, the (possibly implicit) processes by which experimenters decide when to stop collecting the data that will be fed into those procedures—outside of unusual cases in which the stopping rule is “informative” in a technical sense.
Frequentists often argue for their position by claiming that ignoring differences among noninformative stopping rules would allow experimenters to produce systematically misleading results. For instance, Mayo and Kruse (2001) consider the case of a subject who claims to be able to predict draws from a deck of ESP cards. On a frequentist approach, if a 5% significance level is used and the data are treated as if the sample size had been fixed in advance, then the probability of rejecting the “null hypothesis” that the subject has no extrasensory abilities within the first 1000 observations if that hypothesis is true is 53%, and the probability of rejecting it within some finite number of observations is one. Accordingly, Mayo and Kruse claim that whether the experimenter had planned to stop after 1000 trials all along or had planned to stop as soon as a statistically significant outcome had occurred must be reported and must be taken into account in inference and decision. [Read more…]
How My Work Provides an Argument for Subjective Bayesianism
Subjective Bayesianism seems to be the only serious alternative to methodological likelihoodism that conforms to the Likelihood Principle. (Objective Bayesian methods violate the Likelihood Principle in their rules for selecting priors.) My arguments for the Likelihood Principle and against methodological likelihoodism can thus be used as an argument for subjective Bayesianism.
Common Criticisms of Subjective Bayesianism
Subjective Bayesianism is often criticized for being too subjective and too permissive. It’s about what you believe rather than, say, what your evidence supports. On standard formulations, it only requires that one’s beliefs be synchronically and diachronically coherent–that is, to conform to the axioms of probability at a time and to update by Bayesian conditioning over time. It doesn’t tell you what you ought to believe about a given hypothesis on the basis of your evidence except as a function of what you believed about it before and what you believe about other related hypotheses.
I don’t find these objections persuasive. Subjective opinions are important. What am I to use in making decisions if not my own values and opinions (appropriately influenced by my evidence and my interactions with others, of course)? I’d like to be able to give an account of “what one’s evidence supports” in an objective sense that provides good norms of belief or action, but I’m not aware of any such account (likelihoodism does not provide it), and skeptical arguments such as Hume’s problem of induction lead me to doubt that such an account is possible.
My Main Concern About Subjective Bayesianism
The main in-principle problem with subjective Bayesianism, it seems to me, is not that it uses subjective beliefs when the agent in question has them, but that it doesn’t give much guidance to the agent who doesn’t have them. (It faces many additional problems in practice, but I will leave those for another day.) [Read more…]
Where We’ve Been
I have argued for the Likelihood Principle, which says that the evidential meaning of a datum with respect to a partition depends on the probabilities that the elements of that partition ascribe to that hypothesis, up to a constant of proportionality. (Here)
From the Likelihood Principle, I have argued for the Law of Likelihood, which says that the degree to which a datum favors one element of a partition over another is given by the ratio of the probabilities that those hypotheses ascribe to that datum. (Here and here)
I have argued against methodological likelihoodism, which says that characterizing data as evidence in accordance with the Law of Likelihood is an adequate self-contained methodology for science (at least as a fallback option in cases in which prior probabilities are “not available”). (Here)
Where We’re Going: The Methodological Likelihood Principle
The next claim I want to consider is the Methodological Likelihood Principle, which says that an adequate methodology for science respects evidential equivalence as characterized by the Likelihood Principle. [Read more…]
The Problem for Methodological Likelihoodism
I have argued that methodological likelihoodism is false by arguing that an adequate self-contained methodology for science provides good norms of commitment vis-à-vis hypotheses, articulating minimal requirements for a norm of this kind, and proving that no purely likelihood-based norm satisfies those requirements.
The Solution? Appeal to Long-Run Operating Characteristics
One might attempt to rescue methodological likelihoodism by lowering one’s standards. Perhaps no purely likelihood-based norms of commitment are among the canons of rationality, but such norms are nevertheless useful in practice when deployed judiciously. [Read more…]
Likelihoodists admit that their methods are not useful for guiding belief and action directly. One could maintain that characterizing data as evidence is valuable in itself, apart from any possible use in guiding belief or action, but this view is remarkably indifferent to practical considerations. I do not know how else to argue against it, but I do know how to respond to various likelihoodist attempts to make it seem plausible (as I have done in posts here, here, and here).
Royall (2000) seems to provide some reason to believe that likelihoodist methods can in fact reasonably be used to guide belief and action when prior probabilities are not available. He claims that “a paradigm [for statistics] based on [the Law of Likelihood] can generate a frequentist methodology that avoids the logical inconsistencies pervading current methods while maintaining the essential properties that have made those methods into important scientific tools” (31). In other words, likelihoodist methods are warranted by their long-run operating characteristics in the same way that frequentists take their methods to be, without being subject to the many objections that frequentist methods face (such as that they violate the Likelihood Principle).
In fact, likelihoodist methods are not warranted by their operating characteristics in the same way that frequentist methods purport to be. [Read more…]
I have been arguing against the view that a likelihoodist methodology is a viable alternative to Bayesian and frequentist methodologies. A plausible response to this claim is that likelihoodist methods are useful because characterizing data as evidence is valuable in itself. To justify this claim a likelihoodist needs to explain what evidential favoring as measured by likelihood ratios is, if not the degree to which the datum in question warrants a Bayesian shift in one’s beliefs. If there is nothing more to the likelihoodist’s notion of evidential favoring than this, then likelihoodism is merely a footnote to Bayesianism: all of its proper uses have an underlying Bayesian rationale and justification.
Edwards (1972) and Royall (1997) each argue in effect that there is no need to articulate the notion of evidential favoring in the sense in which likelihood ratios measure it. I criticized Royall’s argument in a previous post. I criticize Edwards’s argument here. [Read more…]
Evelyn Marie Adams won the New Jersey state lottery twice in the span of five months. The New York Times described this event as a “1 in 17 trillion” long shot, referring to the probability that one would win twice if one bought one ticket each for exactly two New Jersey state lotteries. Others pointed out that because Adams actually bought many tickets to many different New Jersey state lotteries, the probability that she would be a double winner is somewhat higher than this number. It is, nonetheless, very small. A likelihoodist would be forced to say that her double win favors to an enormously high degree the hypothesis that the lottery was somehow rigged in her favor over the hypothesis that the lottery was fair.
In his (2012), Elliott Sober writes that the Law of Likelihood “seems to endorse the naïve impulse to see conspiracies everywhere, to always think that a hypothesis of Causal Connection is better than the hypothesis of Mere Coincidence.” After all, an apparent coincidence typically involves a co-occurrence of two events that would be much more probable if the events were causally connected than if they occurred independently. Thus, the likelihood ratio of a suitable hypothesis of Causal Connection to a hypothesis of Mere Coincidence is typically very large in cases of coincidence such as the Adams case.
A common reply to the claim that a marvelous coincidence such as Adams’s double win indicates some kind of Causal Connection is that one must put that coincidence into proper perspective. In the case of Adams’s double win, one should consider not the probability that Adams would win the New Jersey lottery twice, but that somebody, somewhere would win some state lottery twice. The probability of the latter is in fact very high (Diaconis and Mosteller 1989), so when applied correctly the Law of Likelihood implies, as it should, that Adams’s double win does not indicate any kind of conspiracy.
This reply has great intuitive appeal, but it seems to violate one of the precious few claims on which philosophers nearly unanimously agree: the Principle of Total Evidence. [Read more…]
In addition to the problems discussed in previous posts, the claim that likelihoodist methods for characterizing data as evidence are valuable in themselves faces the problem that the evidential significance of the fact that one learned that Z=z can be very different from the evidential significance of Z=z itself. The Principle of Total Evidence requires taking into account facts about the process by which one learned that Z=z, but doing so is generally difficult at best and requires thinking about stopping rules, multiple testing, and other issues from which a likelihoodist approach is supposed to be gloriously free. [Read more…]