Nice article. It reminds me of my year living in London, and taking the bus everyday to Imperial College from West End Lane in West Hampstead. There was a stop on both sides of the road - one for the outbound bus, and one for the inbound (the bus went from central London to a terminus and then returned mostly on the same route). Now we did not use schedules - way too inaccurate at rush hour, and the busses there were pretty frequent anyway. But we did expect an even chance of the inbound bus arriving before an outbound one did. My daughter and I became convinced after a while that this was not happening, so we invented a game (which we called "The Game of Life".) When our bus (inbound) arrived first, we added 1 to our score. We subtracted 1 for every outbound bus that passed before ours arrived (there were often more than 1). We realized that the result would be slightly skewed to the negative, but we expected the outcome to be close to 0 over time. Of course it was not. Anyway we extended the game to many statistical situations. For example, you go to the checkout line at the supermarket, and there are N people in front of you. When you get to the front of the line, you count the people behind you - call that M. If M is bigger than N, you scored life points. If it is smaller, you lost some. So you add M-N to your running score, and you get an idea of how lucky you are in life. However, I never followed up with any real analysis, so I enjoyed this article.
Imagine if (in the future) some item like a phone can detect this information around you, and automatically record it. Forming games ontop of this life data would be weird, neat, fun and sad all at the same time. Imagine seeing a real example of where someone else is just more lucky than you are in stupid but impactful (on your morale) ways.
If it didn't seem so tedious to track, I'd love to implement an app to record this info. Unfortunately no one I know would care, and I'm sure I'd get too lazy to keep it accurate. Neat nonetheless, thanks for the cool thoughts :)
Similar to the recruiters that throw away the top half of the application stack because they don't want unlucky people in their company I could see such data become valuable to some people.
Your use of "of course" seems to imply that there's some statistical reason that the probability of the next bus being inbound vs outbound wouldn't be equal. Is there? If so, it seems like it must be a different reason than the one in the article. What am I missing...?
Because we might score -1, -2 or worse if 2 or 3 busses went in the other direction before ours came, but if ours came first, we score 1. We get on the bus and thus don’t know if another one or more arrives first on our side.
This reminds me of a mathematical paradox that makes me doubt your conclusion: "In this country, every couple wants to have one daughter. They keep having children until they have a daughter, and then they stop. What gender balance should we expect?"
Couples can have any number of sons, and every couple has exactly one daughter. Still, the accepted mathematical solution is an equal gender ratio for the couples' children.
I think the "paradox" comes from how people implicitly assume "any number of sons" is somehow distributed or weighted in a way that favors towards numbers of 1 or above.
In contrast, "0 sons" is going to describe a full half of all marriages.
Not really. In the son/daughter case, the calculations are:
expected daughters: 1
expected sons: 1/20 + 1/41 + 1/82 + 1/163 + 1/324 + …
So number of expected daughters = 1, number of expected sons = 1. In practice since women can't have an infinite number of children, then this wouldn't be an infinite series, so the real number of expected boys would be lower than one, but there you go…
Now, for the bus case, you get +1 if your bus turns up first, and -1 for every other bus that turns up first. Assume that it is completely random, then:
expected + score is: 1/2 1
expected - score is: 1/2 * -1 + 1/4 * -2 + …
The expected number of sons is 1, and the expected number of daughters is 1 (by the framing of the problem, in every possible scenario, there is exactly one daughter), but the expected value of the ratio is not 1:1. E[X]/E[Y] = E[X/Y] is not a valid identity.
I read that and it seems wrong. The question asked "what fraction of the pop is female" but his argument is that 3 families of 4 girls and 1 family of 12 boys make the fraction of girls in the average family 75% (the average of 100% x3 and 0% x1) which is non-sensical to me.
>E[X]/E[Y] = E[X/Y] is not a valid identity.
is completely irrelevant here because it is being used to point out that a non-answer is wrong.
It is impossible to catch more than one inbound bus on any given occasion, whereas any number of outbound buses might pass.
BTW, on a slightly unrelated point, if there's no timetable, but the interval between buses is maintained reliably, the expected waiting time is uniformly distributed over that interval.
If you have to get a second bus, you need to convolve two of those two uniform distributions to find out the distribution of overall journey times. This is a trapezoidal distribution, which is just about analytically manageable.
But a journey with two transfers (3 buses in total) results in a likely overall time distributed according to a uniform distribution convolved with a trapezoidal distribution, which is a very weird non-smooth shape. You can see why people choose to model distributions with Gaussians, which are well-behaved (convolve two Gaussians, get another Gaussian). The Gaussian just lends itself ideally to recursive applications, hence recursive filtering (e.g. Kalman filters).
Also, gaussians are great approximations for large n, too, since the convolution of any distribution with itself n times (for n "large enough") is close to gaussian (by the CLT. More generally, there are very nice error estimates for many distributions).
I suspect this analysis can be carried out and yield quite good results in the gaussian case (a careful analysis might even yield error bounds on the result).
Yes. If you spend your whole life on one long multi-transfer bus journey, you'll end up with a gaussian.
It's a bit less clear that gaussians should be used when e.g. fitting a coordinate to an astronomical feature, which might not actually be symmetrical.
The other useful property that the gaussian has is its separability, in the 2D case. That is unique to the gaussian and counts for a lot.
Eh, I don’t think that many are required. Convergence to a Gaussian is pretty fast (you should check out page 299 of [0]), at four or five a Gaussian is already a quite good approximations.
It's not the next bus, it's the number of buses going the other way, which will on average be greater than 0.5 by the same reasoning given in the article.
It's also possible for the result to be biased because of scheduling. If inbound buses pass every 10 minutes at 16.00, 16.10, 16.20,... and outbound buses at 16.01, 16.11, 16.21, ... you'll usually see an inbound bus first. Though I expect this was not the case here.
This is pretty counterintuitive. In the game described you should expect to see one 'wrong way' bus per play on average, not half as you might expect. On the other hand, you have an exactly even chance of catching your own bus before seeing a wrong one, so if your scoring system had been +1 for your bus and -1 for one or more wrong ones, then you would indeed score 0 over time. But with your point per bus scoring system your expected score turns out to be -0.5 per play.
This reminds me the bet in the bitcoin community [1]. If on average bitcoin blocks are produced every 10 minutes, and you learn that 5 minutes ago someone found a block, what is the average time you will wait for the next block? It turns out it's 10 minutes, not 5 minutes as you would intuitively think. (it's a memoryless process, so average expected time till block is always the same - 10 minutes - no matter how many blocks were recently found).
In other words, when you're waiting for bitcoin transaction to be confirmed and go to check how long ago the most recent block was produced, in order to estimate how soon the next one will come - you're doing it wrong. Even if previous block was found 9 minutes ago, you're average waiting time for the next block is still 10 minutes.
And a related counterintuitive fact (again, assuming 10 minutes):
1. If you pick a block randomly (uniformly), its average length is 10 minutes.
2. If you pick a point t0 in time randomly (uniformly), the average length of the block you're in is 20 mins (and the average length from t0 to next block is 10 mins, and the average length from previous block to t0 is also 10 mins (and, needless to say, 10+10=20...)).
Here's an even simpler example: a flip of a fair coin.
Suppose this is your first flip, one would intuitively think that there is 50% chance of H and 50% T.
Suppose you flipped once and got H. For your second flip, one might intuitively think that since the number of times getting H over the long run is 50% of the total flips, and we already have a flip of H, to "balance it out" the next flip should have a smaller probability of getting H. Instead that is wrong. The next flip still has 50% chance of H.
Suppose further that one has performed N flips, all of them H. One might even think that because of the way the geometric distribution works, it is very unlikely for the next flip to be H again. Instead that is wrong. The next flip still has 50% chance of H.
This is actually wrong. The average expected time till next block is almost never 10 minutes because hashpower goes on and offline all the time. It gets adjusted every 2016 blocks based on historical block timing so that if no changes occur then future blocks would be 10 minutes on average - but changes always happen so this is never accurate. As such, you do learn something by looking at prior block times.
What you say is not a rebuttal of the parent comment. Parent explicitly said that average block times are 10mins in the assumption. The most recent block time doesn't change that.
Of course it doesn't speed up your own time, since you have perfect information about your own hashpower. But it does tell you information about the total hashpower that's online, statistically.
I'll give an extreme example to make this clearer. Suppose 10X hashpower just came online an hour ago. It's quite likely that ~60 blocks have been found in the last hour, assuming the difficulty adjustment hasn't happened since. Seeing this, one could deduce that hashpower went up by ~10 and that the expected time till next block is roughly 1 minute instead of 10.
Now, in most cases hashpower doesn't change that drastically but it remains true that recent block times give you more than 0 information about hashpower and therefore about the expectation for future block times.
Slightly related, my ghost town had few buses and sparses. I could never rely on printed hours. If I got there 10 min earlier to be sure, I'd still never be sure I'd wait 20 min for nothing because it was 11 min early. Of course half the time if I decide to walk to the next town where buses are many, I'd see all my town buses (both ways) pass me <yell-at-cloud.png>
I think it made me completely careless about time, I would just go between stops and take the first one, go with the flow. By experience I'd know the range it would take for me to reach big places around the area.
I had a friend who was completely foreign to this mode of thinking, she was very dilligent and fully trusting (although she mostly used trains so a lot less divergence).
It reminds me of kid studies about intelligence / wealth ratios. When you're environment is random, you think random. When it's predictable you planify.
> a Poisson process is a memoryless process that assumes the probability of an arrival is entirely independent of the time since the previous arrival. In reality, a well-run bus system will have schedules deliberately structured to avoid this kind of behavior: buses don't begin their routes at random times throughout the day, but rather begin their routes on a schedule chosen to best serve the transit-riding public.
I've never really understood any example involving a poisson process. They always seem to involve bus arrivals or light bulbs burning out, and I can't understand why the memory less property would ever make any sense for these.
Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting?
What is an actual phenomenon that is well modeled by a poisson process?
Phone calls. Say, an office worker gets 8*6 phone calls in an average shift of 8 hours, so one every 10 minutes on average. It doesn't matter how long ago the last call was, since the customers don't coordinate.
This real world example still doesn't perfectly match the theory. For example, if there was no call for a long time, it may indicate that it's some special day or the phone line is malfunctioning or whatever and it could mean that the next call is probably further in the future than the model would say.
> What is an actual phenomenon that is well modeled by a poisson process?
Radioactive decay. Collisions of fluid molecules. Unstimulated (i.e. not in a laser) photon emission due to electron transitions in an atom. Lots of pretty memoryless stuff going on at the microscopic level.
> Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting?
I don't think it's saying anything about how long you've been waiting, and you don't know when was the last arrival.
It's saying that if you pick a random point on the timeline, the expected wait time doesn't change. That's because by taking a random point you have more chances of landing in a larger stretch of wait time than in a smaller one.
> I don't think it's saying anything about how long you've been waiting, and you don't know when was the last arrival.
This is exactly what memorylessness says something about.
Your second paragraph isn't unique to Poisson processes, but the author right at the start says that the expected value of the waiting time is the same as the average interarrival time, which indicates Poisson.
>What is an actual phenomenon that is well modeled by a poisson process?
I used it in a cell tissue simulation where the user could define how frequently the cells divide. If you start with 100 cells and want them each to divide, on average, every N iterations, using a Poisson formula to decide if a cell splits or not based on a random number is ideal, very precise (in the aggregate), and avoids a lot of odd artifacts.
I think your question is deeper than it first appears, at least it was for me cause I forgot to distinguish gaussian from poisson. One phenomenon is retinal response (per card 8 at this nicely designed site https://quizlet.com/124228940/optometry-stevenson-lectures-f...)
Hah great analysis. One factor with bus' is the schedule is likely planned to minimize early arrivals at the risk of being late more often. Usually when a bus is early it has to sit and wait until its departure time. A late running bus can be more efficient, and if kept until departure time might not ever get a chance to average down the bursts of lateness.
I don't think this is always true. My city measures buses as being on time at inner stops as being between something like 5 minutes early and 1 minute late (yes, you read that right). Timing stops, where they have to wait, are pretty infrequent (mostly bus terminals).
I don't know how common it is but it does exist. And buses perpetually being early means that if you're on time you wait even longer for the next one.
OneBusAway is surprisingly accurate, at least in my experience. Google Maps has very good transit support too.
One reason buses are late is because a bus must travel a circuit. Cars provide linear transportation, so the delay can only happen in the direction of your travel. Since buses run a circuit, they are impacted by delays in the direction opposite of your travel as well.
Your bus might be late because the return route has traffic or other delays. Or maybe a drunk or drug user got in a fight with the driver and the police were needed. Or someone in a wheelchair had a problem getting onto the lift.
I was carless in DC for a year and Google Maps was ALWAYS wrong about when busses when arrive. My friend recommended an app called Transit which was right about 90% of the time, which was a godsend for me.
Seattle - Google Maps often suggests taking 2 buses for a 1 bus trip - to goto Tacoma on the 594, instead of going down the hill and taking the 594, it recommends going to the bus tunnel and taking another bus for about a mile, then transferring to the 594.
But it shows stop locations and plans routes quite effectively. Using the 2 apps in tandem creates a workable solution.
In my experience Google Maps is mostly useless - I know this because OneBusAway is so accurate and they always disagree.
Although Google Maps' transit planner is invaluable for finding possible combinations of buses to use, I rely on OneBusAway to tell me which one is actually going to be faster right now.
Why is modeling required? Can’t we just put a sensor on every bus, and just return the empirical expected time it takes for the next bus to drive to your station given the time of day and day of week?
That's how it works (at least the OneBusAway feed in the Seattle area). They started in the 90s with an RFID transponder on bus stops, read by the bus as it went past; more recently they use things like odometry or GPS to feed information into the system.
But:
> just return the empirical expected time it takes for the next bus
There is a world of complexity in "the empirical expected time", there... expected according to what models?
Anecdotally, I think it's especially hard to model because any given delay is probably attributable to one or a few specific incidents. This isn't a situation where everything averages out and we can use a nice tractable AWGN model; we're down in the muck and the shot-noise.
The memorylessness of the Poisson process makes the statistical aspect a bit trivial. But here's an interesting variant: how should you update your beliefs while waiting if there is a certain probability that the bus won't come at all? "The Ups and the Downs of the Hope Function in a Fruitless Search", Falk et al 1994: https://www.gwern.net/docs/statistics/bayes/1994-falk
I've encountered the inspection paradox in debates about factory farming and people talking past each other points.
If you take the average farm, chances are that it's doing humane farming. But if you take the average animal, it has an overwhelming chance of being in an industrial farm.
Reminds me of a similar article that measured a similar kind of question about the wait times for NYC subways conditional on how long you've been waiting (https://erikbern.com/2016/04/04/nyc-subway-math.html). I think it's a pretty safe bet that people who like this post will like this article as well.
It strikes me that even with a perfectly regular starting schedule, buses might clump together in time because the schedule is probably dynamically unstable. To explain, picking up passengers from a stop costs time and a long time between buses implies a high probability that passengers will be waiting at a given stop. This further adding to the delay and shortens the time to the next bus in the schedule.
I'm sure drivers try to actively manage this, but if they didn't I suspect the system would naturally evolve toward pairs of buses leapfrogging each other on long routes.
I think another confusing factor about that specific example is that bus shouldn't ever start before their schedule. Otherwise you run the risk of a bunch of people missing their bus even though they showed up on time. I think bus, trains and planes can only be late.
Can't believe no one has yet mentioned the PASTA theorem - Poisson Arrivals See Time Averages (https://en.wikipedia.org/wiki/Arrival_theorem#Theorem_for_ar...).
It is one of the theorems I remember the most from my Queuing Theory classes at the university!
Trams don’t have traffic in the way that buses do, and there are only minor differences in the drivers that could cause anomalies (I.e. you can’t steer a tram the wrong way) so they’re much easier to keep on time.
Is the inspection paradox what would happen if you surveyed everyone on how many siblings they had, and every sibling double-counted N-1 times (where N is the number of siblings in their family), inflating the resulting "average number of siblings", or is that something different?
This is my first exposure to it, but yes, I think so. My paraphrase is: The Inspection Paradox is when you ask someone "in the mix" about the mix. You're only going to get an accurate estimation of the mix by standing outside of the mix.
So yes, if you want an accurate count of siblings, you would consult some spreadsheet that just lists how many children each family had. If you go and start asking the families themselves (those "in the mix") then your results will be skewed.
Is there a "evil" distribution which maximises the waiting time? Or is the Poisson distribution already the theoretical "evil" maximum that a public transport provider can achieve?
Well but anything else would equally be bound to the extreme of the rule. "has to come within a 10 minute timeframe" = send one at the beginning, the next one at the end, so always 2 come together.
He'd have to ensure that he arrived at the stop at random times, though. If he aimed for a particular time in order to catch a particular bus, as people often do, that would bias the results beyond recognition.
That would probably make it seem, though, that the buses actually don't arrive (on average) every 10 minutes, since you'd oversample the buses that take longer than 10 minutes.
I'd argue that it's not oversampling at all, but, rather, that the measure of "average bus arrival time" is what's invalid or misleading.
After all, the point of the bus arrivals isn't in service of the bus (or driver) but of the passengers. Observed average wait time at each bus stop is a better measure. The even better measure would be average wait time weighted by number of passengers [1].
[1] which is tougher to measure empirically, or even model, than just average wait time for that one person, since it requires counting passengers boarding, not just bus arrival times.
@mmt to clarify, you seem to be treating bus stops independent of alternative means of transportation. Measuring the average wait time of people at the bus stop is not enough: there are people who chose to ride a bike today instead of waiting at the bus stop, because of what happened to them yesterday at the bus stop.
> you seem to be treating bus stops independent of alternative means of transportation
Perhaps you misunderstood my point, which was more about data and statistics, as is the article itself, rather than transportation.
A similar argument could apply to the article's example of "average class size", where that's a valid statistic when observed by a teacher (or facilities manager), but misleading to a potential student. Something like "average size of a freshman's classes" would be more meaningful to a prospective student, and "oversampling" would not be a valid complaint there, either.
> instead of waiting at the bus stop, because of what happened to them yesterday at the bus stop.
It sounds like you're suggesting that there's an even better measure than the two I proposed, rather than the original measure being better. If so, I don't dispute that there could be many more, as I never claimed "best".
In this instance, though, measuring people who never show up to the bus stop in the first place is impossible, and even measuring those who showed up but abandoned waiting (i.e. never boarded) is impossible without additional instruments (whereas, presumably, electronic fare collection equipment could closely enough approximate counting boardings).
Wow that is...quite the edit lol. I'm all for small corrections and addendums, but completely changing the meaning of your comment from an attempted callout is something else.
This is a straightforward consequence of modelling an arrival process as a Poisson distribution with a constant rate of arrival lambda...
Go from arrival to cumulative arrivals to time of arrival to recurrence of arrival (next arrival). All are Poisson processes, including the recurrence process, which has a fixed expected value.
It's just the Poisson process, e.g., with a nice chapter in E. Cinlar, Introduction to Stochastic Processes.
Buses come as arrivals. So bus arrivals are a stochastic arrival process where stochastic just means varying randomly over time where, really, the randomly doesn't mean anything, includes deterministic arrivals, that is, known exactly in advance, but also admits any case of unpredictability.
Well, in short, if have a stochastic arrival process with stationary, independent increments, then the arrival process is a Poisson process and there is a number, usually denoted by lambda, so that the times between arrivals are independent, identically distributed random variables with exponential distribution with arrival parameter, the arrival rate, lambda. The stationary means that the probability distribution of the times between arrival does not change over time. The independent increments means that the time from one arrival to the next is independent of all the past history of arrivals.
The exponential distribution has the property, easy to verify with simple calculus, that the conditional expectation of the arrival time given that the arrival time is already greater than some number is the same as the expected arrival time.
So, net, if bus arrivals form a Poisson process, then the time until the next bus arrives is the same after waiting five minutes as not having waited at all.
Cinlar's treatment is nice because it is qualitative, that is, has assumptions that can often be confirmed or believed just intuitively. And we might not believe that bus arrivals meed the assumptions.
This subject can continue with, say, hazard curves for equipment failures and a lot more about Poisson processes.
E.g., the sum of two independent Poisson processes, say, Red buses and Blue buses, assuming that they are Poisson processes, is also a Poisson process with arrival rate the sum of the Red and Blue arrival rates. If randomly throw away some arrivals, then what is left is also a Poisson process with arrival rate adjusted in the obvious way.
In Feller's volume II is the renewal theorem that the sum of independent arrival processes, Poisson or not, with mild assumptions, converges to a Poisson process as the number of processes summed grows. So, if the users of a sufficiently busy Web site act independently with mild assumptions, then the Web site will see arrivals accurately as a Poisson process.
The vanilla Poisson process is Geiger counter clicks.
There is much more to the pure and applied math and applications of Poisson processes.
> When waiting for a bus that comes on average every 10 minutes, your average waiting time will be 10 minutes.
This is very ambiguous. Unless he gives a time frame the numbers do not make sense. Average in a week? Average in a year? This is not how it works in real life.
And I cannot accept his premise. My experience tells me that, in New York, when I used to take a bus to work, sometimes the bus was coming as I was walking to the stop; sometimes I would wait a long time. Sometimes not very long. There was no observable bias.
In statistics, "average" often means "expected value". No time frame is specified (although you could consider it an infinite time frame). With a small sample size your actual average might not be 10 minutes, but as your sample size grows, it will tend toward 10 minutes.
If you are talking about spherical-cow style poisson buses, yeah (that's what the author means by "reasonable assumptions). But as the author concludes, bus arrival times are not well modeled by a poisson process.
