Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> You just need to make sure your math acknowledges the procedure.

I'd love to hear some clarification on how "[making] sure your math acknowledges" stopping randomized tests early works, because it flies in a face of how actual statistics works.



> it flies in a face of how actual statistics works.

The terms you're looking for are 'sequential testing', 'optimal stopping', and 'adaptive trials'.


consider fair coin flip sequences. stopping at arbitrary points of your choice can never really affect anything serious. but you need to make sure you're doing math right, meaning if you stop when something gets a lead you don't use incorrect math that says the coin is biased.

data is data. as long as the method being used to collect individual data points is fine, and they are collected independently, then the data you get as a result is gonna be OK, the rest (like arbitrary stop time) doesn't ruin it. you just have to avoid bad math.

what ruins data is stuff like throwing 10% of the heads results in the trash or using other approaches in which data can be selectively discarded or not discarded. so just stopping arbitrarily can be a problem if you might never stop and throw out the results if you don't like them. but if you do something like "stop after 1 million data points max, or when i feel like it earlier" then your data is still OK because it cannot get selectively ignored.

stopping earlier cannot make a fair coin look unfair or anything like that.

this is not some random unknown position that flies in the face of how actual statistics works. something like this is the standard bayesian position, and i think it's true. (i strongly object to bayesian epistemology, but i think bayesian statistics is correct).

not ALL stopping rules are OK but lots are. you don't HAVE to use simple ones like "gather X data points, stop".

see stuff like:

http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a...

http://lesswrong.com/lw/mt/beautiful_probability/

> And then there's the Bayesian reply: "Excuse you? The evidential impact of a fixed experimental method, producing the same data, depends on the researcher's private thoughts? And you have the nerve to accuse us of being 'too subjective'?"





Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: