Neyman and Pearson (e.g. 1933) treat the problem of choosing the best rejection region for a simple-vs.-simple hypothesis test as what computer scientists call a 0/1 knapsack problem. Standard examples of 0/1 knapsack problems are easier to grasp than hypothesis testing problems, so thinking about Neyman-Pearson test construction on analogy with those examples is helpful for developing intuitions. It is also illuminating to think about points of disanalogy between those scenarios and hypothesis testing scenarios, which give rise to possible objections to the Neyman-Pearson approach.
The Standard Bayesian-Frequentist Debate About Stopping Rules
Bayesians generally reject the frequentist view that inference and decision procedures should be sensitive to differences among “stopping rules”—that is, the (possibly implicit) processes by which experimenters decide when to stop collecting the data that will be fed into those procedures—outside of unusual cases in which the stopping rule is “informative” in a technical sense.
Frequentists often argue for their position by claiming that ignoring differences among noninformative stopping rules would allow experimenters to produce systematically misleading results. For instance, Mayo and Kruse (2001) consider the case of a subject who claims to be able to predict draws from a deck of ESP cards. On a frequentist approach, if a 5% significance level is used and the data are treated as if the sample size had been fixed in advance, then the probability of rejecting the “null hypothesis” that the subject has no extrasensory abilities within the first 1000 observations if that hypothesis is true is 53%, and the probability of rejecting it within some finite number of observations is one. Accordingly, Mayo and Kruse claim that whether the experimenter had planned to stop after 1000 trials all along or had planned to stop as soon as a statistically significant outcome had occurred must be reported and must be taken into account in inference and decision. [Read more…]
My conversations at the Munich Center for Mathematical Philosophy keep coming back to stopping rules, so I’ve decided to write a paper on the topic. Here is the general line that I plan to develop.
The major virtues and vices of Bayesian, frequentist, and likelihoodist approaches to statistical inference.# Introduction
My goal in this post and the previous one is to provide a short, self-contained introduction to likelihoodist, Bayesian, and frequentist methods that is readily available online and accessible to someone with no special training who wants to know what all the fuss is about.
In the previous post, I gave a motivating example that illustrates the enormous costs of the failure of philosophers, statisticians, and scientists to reach consensus on a reasonable, workable approach to statistical inference. I then used a fictitious variant on that example to illustrate how likelihoodist, Bayesian, and frequentist methods work in a simple case. In this post, I discuss a stranger case that better illustrates how likelihoodist, Bayesian, and frequentist methods come apart. [Read more…]
I have been recommending the first chapter of Elliott Sober’s Evidence and Evolution to those who ask for a good introduction to debates about statistical inference. That chapter is excellent, but it would nice to be able to recommend something shorter that is readily available online. Here is my attempt to provide a suitable source. [Read more…]
The Sense in Which Frequentist Methods Violate the Likelihood Principle
It is widely accepted that frequentist methods violate the Likelihood Principle. After all, there can be two pieces of data A and B such that the Likelihood Principle implies that A and B are evidentially equivalent with respect to the set of hypotheses H, yet frequentist methods will yield different conclusions about H depending on whether A or B is fed into them.
But What About Using Frequentist Considerations to Choose Among Priors?
There is another sense in which many frequentist methods do not violate the Likelihood Principle. A frequentist method is often equivalent (in a sense) to a Bayesian method with a particular prior probability distribution. From a Bayesian perspective, such frequentist methods involve updating a prior probability distribution in a way that does conform to the Likelihood Principle. They violate the Likelihood Principle only by using “implied priors” that vary with the sampling distribution of the experiment to be performed. [Read more…]
The Neyman-Pearson decision theory is rarely strictly applied, but when it is, it does not permit anything but the rejection of theories…. Nor does the decision theory provide any judgments about which of several false theories is the better approximation. But in all of science…the principle job is not to reject false theories but to find the better approximations and reject the worse approximations.
–Clark Glymour et al., Discovering Causal Structure (1991), p. 92
Likelihoodists admit that their methods are not useful for guiding belief and action directly. One could maintain that characterizing data as evidence is valuable in itself, apart from any possible use in guiding belief or action, but this view is remarkably indifferent to practical considerations. I do not know how else to argue against it, but I do know how to respond to various likelihoodist attempts to make it seem plausible (as I have done in posts here, here, and here).
Royall (2000) seems to provide some reason to believe that likelihoodist methods can in fact reasonably be used to guide belief and action when prior probabilities are not available. He claims that “a paradigm [for statistics] based on [the Law of Likelihood] can generate a frequentist methodology that avoids the logical inconsistencies pervading current methods while maintaining the essential properties that have made those methods into important scientific tools” (31). In other words, likelihoodist methods are warranted by their long-run operating characteristics in the same way that frequentists take their methods to be, without being subject to the many objections that frequentist methods face (such as that they violate the Likelihood Principle).
In fact, likelihoodist methods are not warranted by their operating characteristics in the same way that frequentist methods purport to be. [Read more…]
Other advocates of the Likelihood Principle have argued that while Stein’s purported counterexample illustrates a conflict between the Likelihood Principle and frequentist reasoning, the problem lies with the frequentist reasoning rather than with the Likelihood Principle. I argue here that the example does not even illustrate a conflict between the Likelihood Principle and frequentist reasoning. The result Stein has establishes does not indicate a conflict between the Likelihood Principle and reasoning that is correct from any defensible frequentist perspective. It is superficially similar to a result that would indicate such a conflict, but that result does not follow from Stein’s argument.
Want to keep up with new posts without having to check for them manually? Use the sidebar on the left to sign up for updates via email or RSS feed!