Neyman and Pearson (e.g. 1933) treat the problem of choosing the best rejection region for a simple-vs.-simple hypothesis test as what computer scientists call a 0/1 knapsack problem. Standard examples of 0/1 knapsack problems are easier to grasp than hypothesis testing problems, so thinking about Neyman-Pearson test construction on analogy with those examples is helpful for developing intuitions. It is also illuminating to think about points of disanalogy between those scenarios and hypothesis testing scenarios, which give rise to possible objections to the Neyman-Pearson approach.
The Standard Bayesian-Frequentist Debate About Stopping Rules
Bayesians generally reject the frequentist view that inference and decision procedures should be sensitive to differences among “stopping rules”—that is, the (possibly implicit) processes by which experimenters decide when to stop collecting the data that will be fed into those procedures—outside of unusual cases in which the stopping rule is “informative” in a technical sense.
Frequentists often argue for their position by claiming that ignoring differences among noninformative stopping rules would allow experimenters to produce systematically misleading results. For instance, Mayo and Kruse (2001) consider the case of a subject who claims to be able to predict draws from a deck of ESP cards. On a frequentist approach, if a 5% significance level is used and the data are treated as if the sample size had been fixed in advance, then the probability of rejecting the “null hypothesis” that the subject has no extrasensory abilities within the first 1000 observations if that hypothesis is true is 53%, and the probability of rejecting it within some finite number of observations is one. Accordingly, Mayo and Kruse claim that whether the experimenter had planned to stop after 1000 trials all along or had planned to stop as soon as a statistically significant outcome had occurred must be reported and must be taken into account in inference and decision. [Read more…]
My conversations at the Munich Center for Mathematical Philosophy keep coming back to stopping rules, so I’ve decided to write a paper on the topic. Here is the general line that I plan to develop.
The major virtues and vices of Bayesian, frequentist, and likelihoodist approaches to statistical inference.# Introduction
My goal in this post and the previous one is to provide a short, self-contained introduction to likelihoodist, Bayesian, and frequentist methods that is readily available online and accessible to someone with no special training who wants to know what all the fuss is about.
In the previous post, I gave a motivating example that illustrates the enormous costs of the failure of philosophers, statisticians, and scientists to reach consensus on a reasonable, workable approach to statistical inference. I then used a fictitious variant on that example to illustrate how likelihoodist, Bayesian, and frequentist methods work in a simple case. In this post, I discuss a stranger case that better illustrates how likelihoodist, Bayesian, and frequentist methods come apart. [Read more…]
I have been recommending the first chapter of Elliott Sober’s Evidence and Evolution to those who ask for a good introduction to debates about statistical inference. That chapter is excellent, but it would nice to be able to recommend something shorter that is readily available online. Here is my attempt to provide a suitable source. [Read more…]
In my previous post I presented some reasons to resist a clever counterexample to the Law of Likelihood developed by Mike Titelbaum. In that post I chose to stay at the level of intuitions about the example and about what kinds of features we might want a measure of evidential favoring to have. In this post I go deeper by examining Mike’s example in light of the purpose of the Law of Likelihood. [Read more…]
Last week’s post in which I presented a purported counterexample to the Law of Likelihood (due to Mike Titelbaum) generated a lot of interest. In this post I comment on the counterexample. Next week I plan to “go meta” by commenting on the purpose of the Law of Likelihood and of counterexamples such as Mike’s.
A talk I recently gave prompted Mike Titelbaum to develop a purported counterexample to the Law of Likelihood. In this post I present the counterexample with minimal comments for the sake of eliciting reactions to it that are not influenced by my thoughts. I plan to follow up with my comments next week. [Read more…]