Neyman and Pearson (e.g. 1933) treat the problem of choosing the best rejection region for a simple-vs.-simple hypothesis test as what computer scientists call a 0/1 knapsack problem. Standard examples of 0/1 knapsack problems are easier to grasp than hypothesis testing problems, so thinking about Neyman-Pearson test construction on analogy with those examples is helpful for developing intuitions. It is also illuminating to think about points of disanalogy between those scenarios and hypothesis testing scenarios, which give rise to possible objections to the Neyman-Pearson approach.
The Standard Bayesian-Frequentist Debate About Stopping Rules
Bayesians generally reject the frequentist view that inference and decision procedures should be sensitive to differences among “stopping rules”—that is, the (possibly implicit) processes by which experimenters decide when to stop collecting the data that will be fed into those procedures—outside of unusual cases in which the stopping rule is “informative” in a technical sense.
Frequentists often argue for their position by claiming that ignoring differences among noninformative stopping rules would allow experimenters to produce systematically misleading results. For instance, Mayo and Kruse (2001) consider the case of a subject who claims to be able to predict draws from a deck of ESP cards. On a frequentist approach, if a 5% significance level is used and the data are treated as if the sample size had been fixed in advance, then the probability of rejecting the “null hypothesis” that the subject has no extrasensory abilities within the first 1000 observations if that hypothesis is true is 53%, and the probability of rejecting it within some finite number of observations is one. Accordingly, Mayo and Kruse claim that whether the experimenter had planned to stop after 1000 trials all along or had planned to stop as soon as a statistically significant outcome had occurred must be reported and must be taken into account in inference and decision. [Read more…]
How My Work Provides an Argument for Subjective Bayesianism
Subjective Bayesianism seems to be the only serious alternative to methodological likelihoodism that conforms to the Likelihood Principle. (Objective Bayesian methods violate the Likelihood Principle in their rules for selecting priors.) My arguments for the Likelihood Principle and against methodological likelihoodism can thus be used as an argument for subjective Bayesianism.
Common Criticisms of Subjective Bayesianism
Subjective Bayesianism is often criticized for being too subjective and too permissive. It’s about what you believe rather than, say, what your evidence supports. On standard formulations, it only requires that one’s beliefs be synchronically and diachronically coherent–that is, to conform to the axioms of probability at a time and to update by Bayesian conditioning over time. It doesn’t tell you what you ought to believe about a given hypothesis on the basis of your evidence except as a function of what you believed about it before and what you believe about other related hypotheses.
I don’t find these objections persuasive. Subjective opinions are important. What am I to use in making decisions if not my own values and opinions (appropriately influenced by my evidence and my interactions with others, of course)? I’d like to be able to give an account of “what one’s evidence supports” in an objective sense that provides good norms of belief or action, but I’m not aware of any such account (likelihoodism does not provide it), and skeptical arguments such as Hume’s problem of induction lead me to doubt that such an account is possible.
My Main Concern About Subjective Bayesianism
The main in-principle problem with subjective Bayesianism, it seems to me, is not that it uses subjective beliefs when the agent in question has them, but that it doesn’t give much guidance to the agent who doesn’t have them. (It faces many additional problems in practice, but I will leave those for another day.) [Read more…]
Where We’ve Been
I have argued for the Likelihood Principle, which says that the evidential meaning of a datum with respect to a partition depends on the probabilities that the elements of that partition ascribe to that hypothesis, up to a constant of proportionality. (Here)
From the Likelihood Principle, I have argued for the Law of Likelihood, which says that the degree to which a datum favors one element of a partition over another is given by the ratio of the probabilities that those hypotheses ascribe to that datum. (Here and here)
I have argued against methodological likelihoodism, which says that characterizing data as evidence in accordance with the Law of Likelihood is an adequate self-contained methodology for science (at least as a fallback option in cases in which prior probabilities are “not available”). (Here)
Where We’re Going: The Methodological Likelihood Principle
The next claim I want to consider is the Methodological Likelihood Principle, which says that an adequate methodology for science respects evidential equivalence as characterized by the Likelihood Principle. [Read more…]
I have been arguing against the view that a likelihoodist methodology is a viable alternative to Bayesian and frequentist methodologies. A plausible response to this claim is that likelihoodist methods are useful because characterizing data as evidence is valuable in itself. To justify this claim a likelihoodist needs to explain what evidential favoring as measured by likelihood ratios is, if not the degree to which the datum in question warrants a Bayesian shift in one’s beliefs. If there is nothing more to the likelihoodist’s notion of evidential favoring than this, then likelihoodism is merely a footnote to Bayesianism: all of its proper uses have an underlying Bayesian rationale and justification.
Edwards (1972) and Royall (1997) each argue in effect that there is no need to articulate the notion of evidential favoring in the sense in which likelihood ratios measure it. I criticized Royall’s argument in a previous post. I criticize Edwards’s argument here. [Read more…]
Evelyn Marie Adams won the New Jersey state lottery twice in the span of five months. The New York Times described this event as a “1 in 17 trillion” long shot, referring to the probability that one would win twice if one bought one ticket each for exactly two New Jersey state lotteries. Others pointed out that because Adams actually bought many tickets to many different New Jersey state lotteries, the probability that she would be a double winner is somewhat higher than this number. It is, nonetheless, very small. A likelihoodist would be forced to say that her double win favors to an enormously high degree the hypothesis that the lottery was somehow rigged in her favor over the hypothesis that the lottery was fair.
In his (2012), Elliott Sober writes that the Law of Likelihood “seems to endorse the naïve impulse to see conspiracies everywhere, to always think that a hypothesis of Causal Connection is better than the hypothesis of Mere Coincidence.” After all, an apparent coincidence typically involves a co-occurrence of two events that would be much more probable if the events were causally connected than if they occurred independently. Thus, the likelihood ratio of a suitable hypothesis of Causal Connection to a hypothesis of Mere Coincidence is typically very large in cases of coincidence such as the Adams case.
A common reply to the claim that a marvelous coincidence such as Adams’s double win indicates some kind of Causal Connection is that one must put that coincidence into proper perspective. In the case of Adams’s double win, one should consider not the probability that Adams would win the New Jersey lottery twice, but that somebody, somewhere would win some state lottery twice. The probability of the latter is in fact very high (Diaconis and Mosteller 1989), so when applied correctly the Law of Likelihood implies, as it should, that Adams’s double win does not indicate any kind of conspiracy.
This reply has great intuitive appeal, but it seems to violate one of the precious few claims on which philosophers nearly unanimously agree: the Principle of Total Evidence. [Read more…]
In addition to the problems discussed in previous posts, the claim that likelihoodist methods for characterizing data as evidence are valuable in themselves faces the problem that the evidential significance of the fact that one learned that Z=z can be very different from the evidential significance of Z=z itself. The Principle of Total Evidence requires taking into account facts about the process by which one learned that Z=z, but doing so is generally difficult at best and requires thinking about stopping rules, multiple testing, and other issues from which a likelihoodist approach is supposed to be gloriously free. [Read more…]
This post is part of a series (introduced here) in which I present objections to possible responses to the claim that likelihoodism is not a viable alternative to Bayesian and frequentist methodologies because it does not address questions about what to believe or do. I am currently considering the response that likelihoodism is a viable alternative methodology because characterizing one’s data as evidence is valuable in itself. I addressed two arguments for this claim here and here.
A likelihood ratio is a ratio of posterior odds to prior odds under Bayesian updating. Likelihoodists agree with frequentists that prior probabilities that merely represent some individual’s degrees of belief are not appropriate for use in science. Unlike frequentists, they maintain that a likelihood ratio “means the same” as a measure of evidential favoring regardless of whether prior probabilities they regard as legitimate are available or not. But what is a measure of evidential favoring in the likelihoodist’s sense, if not a measure of the shift in one’s degrees of belief that the data warrant?
It seems that the likelihoodist claim that characterizing as evidence is valuable in itself requires an answer to this question. However, no likelihoodist to my knowledge has attempted to provide one. Edwards (1972) and Royall (1997) each address it by arguing in effect that no answer is needed. I responded to Royall’s argument in my previous post. Unfortunately, I do not have access to my copy of Edwards’s book at the moment, so instead of responding to his argument I will consider an objection to the claim that a likelihood ratio “means the same” across different contexts that is due to Ian Hacking.
Hacking coined the phrase “Law of Likelihood” in his (1965), but he used it to refer only to the qualitative claim that E favors H1 over H2 if Pr(E;H1)> Pr(E;H2). He did not include the quantitative claim that the likelihood ratio Pr(E;H1)/Pr(E;H2) is a measure of the degree to which E favors H1 over H2. Edwards (1972) seems to have been the first to combine the qualitative and quantitative claims into a single principle, which he called the Likelihood Axiom (31). Today it is standard to follow Royall in using Hacking’s phrase “Law of Likelihood” for the conjunction of the qualitative and quantitative claims.
In a review of (Edwards 1972), Hacking expresses doubts about both the quantitative and the qualitative parts of the Law of Likelihood and argues specifically against the assumption of the quantitative part that a likelihood ratio “means the same” in different contexts (1972, 136). [Read more…]
This post is part of a series (introduced here) in which I present objections to possible responses to the claim that likelihoodism is not a viable alternative to Bayesian and frequentist methodologies because it does not address questions about what to believe or do. I am currently considering the response that likelihoodism is a viable alternative methodology because characterizing one’s data as evidence is valuable in itself.
One argument for this claim is that the notion of rational belief is closely tied to the notion of evidential support. As Hume put it (with the sexism removed), “A wise [person] proportions his [or her] belief to the evidence.” The notion of rational belief is important for epistemology and methodology if anything is, so a methodology that correctly characterizes data as evidence is obviously valuable.
I addressed this argument here. In short, the problem with it is that likelihoodism explicates aspects of an incremental notion of evidence as good grounds for a shift in one’s beliefs, rather than an absolute notion of evidence as good grounds for belief itself. Thus, likelihoodism does not characterize data as evidence in a sense to which Hume’s dictum applies. The relevant dictum would instead be, “A wise person proportions shifts in his or her beliefs to the evidence.” But likelihoodist methods do not implement this dictum; Bayesian methods do. Likelihoodist methods characterize how beliefs should shift without attending to the beliefs themselves. They are for that reason potentially useful for the purpose of summarizing one’s data in a way that allows one’s audience members each to update his or her own subjective prior probability distribution, but using likelihoods in that way is an essentially Bayesian procedure with a Bayesian rationale that does not vindicate likelihoodism as a viable and genuinely distinctive methodological program.
A likelihood ratio is a ratio of posterior odds to prior odds under Bayesian updating. Likelihoodists agree with frequentists and disagree with Bayesians in that they claim that legitimate prior probabilities often do not exist. Unlike frequentists, they maintain that a likelihood ratio “means the same” regardless of whether prior probabilities are present or absent. Of course, they cannot mean that a likelihood ratio is a ratio of posterior odds to prior odds under Bayesian updating regardless of whether prior probabilities are present or not. What they say is that it retains its meaning as a measure of evidential favoring regardless of whether prior probabilities are present or not. But what is a measure of evidential favoring in the likelihoodist’s sense, if not a measure of the shift in one’s beliefs that the data warrant?
No likelihoodist to my knowledge has attempted to answer this question. [Read more…]