This is the 1 Bus in Boston.

It stops by several of my favorite coffee shops, two world-renowned universities, a great farmer’s market in Central Square, a wonderful Thai restaurant in Boston, and the beautiful Charles River.

In the spring of 2011, the MBTA added real-time tracking for the 1 Bus (and for all of its buses) via NextBus. To this day, real-time tracking continues to amaze me. On numerous occasions, this new fangled technology has saved me from standing in the freezing cold! But every so often, NextBus leads me astray. Perhaps a bus mysteriously disappears from the tracker. Or, the bus that was supposed to arrive two minutes ago is now ten minutes away. Over the last few years, I have observed that while NextBus’s predictions tend to be reasonably accurate, there are clearly patterns to when its predictions tend to be wrong and by how much time. Using NextBus’s public API and tracking data, I decided to explore these patterns by asking NextBus, once every other minute, to predict arrival times for every stop that the 1 Bus makes. Here’s what I’ve learned so far.

NextBus predictions in pictures

The series of histograms in this section, based on NextBus predictions collected for bus trips between April 4th, 2014 and May 7th, 2014, illustrate a few of the most prominent empirical trends between bus predictions and actual arrival times. Histograms are truncated to show the most common range of errors. In each histogram, “error” represents the difference between a bus’s predicted arrival time and its actual arrival time rounded down to the nearest integer. For example, consider a NextBus prediction of five minutes. An error of zero indicates that the bus arrived as expected, five to six minutes after the original prediction. A error of one minute, for contrast, indicates that the bus was one minute late relative to the original prediction and actually arrived some time between six and seven minutes.

The first set of histograms, shown above, tracks the performance of one-minute predictions for six key stops along the 1 Bus’s southbound route from Cambridge to Boston. For the route’s originating stop at Holyoke Gate, over 90% of one-minute predictions were correct. Given that the MBTA often stages its buses at originating stops a few minutes prior to departures, this high accuracy rate is not surprising. However, NextBus has more difficulty further down the route, and the error rates increase for all other stops. Prediction accuracy reaches a minimum of just over 50% at Newbury St. (perhaps due to greater uncertainty about traffic flows near the Mass. Ave. bridge and on- and off-ramps for the Mass Turnpike), and then improves slightly for the remainder of the 1 Bus’s trip toward Dudley.

The potential impact of less predictable traffic flows on NextBus’s predictions at the middle of the 1 Bus’s route suggest two ideas to explore. First, the magnitude of NextBus’s prediction errors may depend on actual prediction times as long predictions are subject to more unexpected changes in traffic than short predictions. Second, and related to the previous point, as a bus travels along its route, predictions for stops that are further in the route are subject to greater uncertainty due to longer time windows for a bus to accumulate or make up delay (compare the histogram above at the 1 Bus’s originating stop versus its Newbury St. histogram). Consequently, longer predictions and predictions for stops further along a bus route may exhibit larger errors than either shorter predictions or predictions for stops near the beginning of a route. The next set of charts explores these potential relationships between accuracy, prediction length, and distance from the originating stop. The charts shown below display the median error for each stop and prediction, grouped into two-minute bins. NextBus predictions ranging from 0 to 119 seconds were placed in the first bin; predictions ranging from 120 seconds to 239 seconds were placed in the second bin; and so forth. For the analysis month, each prediction bin contains between 2000 to 4000 predictions.

As the 1 Bus travels south from Harvard, NextBus’s prediction performance noticeably worsens with increasing prediction time and distance from the originating stop. At the originating stop, predictions and median errors show that 1 Bus tends to leave on schedule, and predictions of all lengths are quite accurate. However, for stops at Central Square, MIT, Newbury St., Mass. Ave. Station, and Dudley, longer prediction times from NextBus consistently exhibit larger errors. For the same stop, shorter prediction times are correct more often than longer prediction times. Also, errors tend to occur in only one direction. The positive median errors suggest that when predictions are wrong, buses are more likely to arrive later than predicted, not earlier.

In addition to the correlation between increasing prediction times and increasing prediction errors for each stop, the charts above also reveal a correlation exists between the distance of a stop from the originating stop and the magnitude of NextBus’s prediction errors. The differences between predictions for Central Square, which appears relatively early in the 1 Bus’s southbound route, and for Dudley, the southbound route’s terminating stop, illustrate this trend. For both stops, predictions of zero or one minute tend to be accurate. However, looking at a prediction of 20 minutes for Central and Dudley, the prediction for Central has a relatively small median error of approximately 1 minute, while the prediction for Dudley has a relatively large median error of approximately 4 minutes. The farther the bus stop along the route, the more likely NextBus’s predictions are to exhibit larger errors.

Practically speaking, regular bus riders are likely to be aware of these patterns. People that board at Harvard quickly learn that buses generally depart exactly as predicted, while riders across the river might expect to have a few more minutes to run and catch a bus. The rarity of early bus arrivals leads to a simple heuristic for “optimizing” bus wait times that finds its roots in common sense: determine how long it takes to walk to the nearest bus stop and then head out the door with just enough time to catch the next bus. Probabilistically, this “greedy” approach works wonderfully for maximizing sleep time in the morning for risk-neutral bus riders. Unfortunately, as the first set of histograms clearly show, every so often the bus arrives just ever-so-slightly earlier than expected. The devil is in the tail events.

Time-of-day considerations

The previous analysis aggregated all predictions for one month, lumping predictions from different hours, weekdays, and weekends together. However, given the regularity of traffic patterns, does NextBus tend to give more accurate predictions during some parts of the day and less accurate predictions during others? For example, do predictions tend to be worse during busy commute times in the morning and afternoon? To explore how NextBus performs throughout the day, the graphs below show NextBus’s errors for a range of predictions during weekdays, broken down by hour. As the MBTA does not operate the 1 Bus between 2AM and 5AM, those hours have no corresponding graphs. The predictions shown below are for the 1 Bus’s southbound stop at Newbury St; on average, most of the one-minute prediction bins used to create these graphs had between 30 to 100 prediction samples over the analysis month.

A few interesting patterns appear in the series of charts above that support the idea that NextBus’s prediction accuracy also depends on the time of day. As noted before, short predictions—predictions between zero and two minutes—remain reasonably accurate throughout the day. In the morning, from 5AM to 7AM, buses tend to arrive earlier than predicted. During the morning rush hour, from 8AM to 9AM, longer predictions are less likely to be accurate. This trend is more noticeable in the early afternoon, between 2PM and 6PM, when predictions are most unreliable and median errors increase to as much as 10 minutes. As the evening continues, NextBus’s prediction accuracy slowly improves. These hourly differences reveal quite a bit of useful detail about NextBus’s predictions for commuters: buses tend to arrive earlier than predicted in the early morning and later than predicted during traditionally busy traffic times. Not all predictions, even for the same stop and the same relative arrival time, are equally reliable.

Thus far, the charts shown have explored three independent ideas: first, that longer predictions are positively correlated with larger errors; second, that longer distances from the originating stop are positively correlated with larger errors; and third, that predictions exhibit larger errors during busy traffic times. The series of charts below explores these three factors in combination by taking the previous hour-by-hour breakdown of predictions for Newbury St. and superimposing predictions for the remaining five stops of interest. From this series, the three trends are striking. Predictions for the originating stop at Harvard are consistently better than other stops, while predictions for Dudley, the farthest stop, frequently exhibit the largest errors. Short predictions for all stops are better than long predictions, and prediction errors are largest during busy traffic periods in the afternoon and early evening. In aggregate, these trends suggest that NextBus may be able to correct at least some of these systematic offsets with modifications to their prediction algorithm.

On bus clumping: one day, while walking back from one of my aforementioned favorite coffeeshops...

I noticed three 1 buses driving together on the Mass. Ave. bridge. Three! Although buses do clump, the NextBus data shows that observer bias likely plays a large role in creating the “ARGH!” factor that most riders feel about clumping while waiting for a bus. In April 2014, for 1 Bus’s trips from Harvard to Dudley, only approximately two percent of arrivals at a stop occurred within three minutes of the previous bus’s arrival at that same stop. The graph below illustrates the total number of “clumped arrivals” by stop and time of day for an arbitrarily chosen three minute interval. In this graph, each counted incident reflects the arrival of a single bus; if multiple buses arrive within the same three-minute interval, each additional bus counts as a new incident. As the MBTA operated over 2700 bus trips from Harvard to Dudley in April 2014, the number of clumped arrivals is relatively small compared to the number of nonclumped arrivals. However, as most incidents of clumping tend to occur during the morning and evening commutes (from 8AM to 9AM and from 5PM to 8PM), riders of the 1 Bus during these times are more likely to observe the nearly simultaneous arrival of multiple buses at the same stop.

With regards to commuting via bus...

The mean total trip time for the 1 Bus varies in a mostly predictable fashion throughout the day: trip times are shortest in the early morning and late night and “peak” between 8AM and 6PM. The 1 Bus, on average, takes about 40 minutes to complete a trip from Harvard to Dudley. Morning commuters that depend on the 1 Bus can shorten their travel time by taking the bus at 6AM instead of 8AM (the disutility of taking a 6AM bus is enough to convince this night owl to bear the extra 10 minutes of travel time at 8AM), and riders that want to avoid the evening rush hour should catch a bus after 6PM. The chart below illustrates the 1 Bus’s mean trip times broken down by hour during the analysis month.

Practically speaking...

In general, the 1 Bus tends to arrive earlier than predicted during periods of low traffic and later than predicted during periods of high traffic. These patterns are most prevalent in the morning, between 5AM and 8AM, and in the evening, from 2PM to 7PM. The farther down the route a stop is, and the longer a prediction is, the more likely that buses will arrive later than predicted. Afternoon bus riders should expect greater prediction errors, although short predictions of one and two minutes remain quite accurate for all stops and times of day. The average time for the 1 Bus is approximately 40 minutes, and riders traveling after 7PM will generally experience shorter trip times. Bus clumping, though irksome, occurs rather infrequently.

Lastly, the empirical evidence that inspired me to conduct this exploration has taught me that if I walk briskly, and if all the walk signs turn at the right time, and if enough people board at the early stops to slow the bus down, then I-can-leave-my-apartment-an-extra-minute-late-and-miss-the-bus-at-Harvard-but-barely-catch-it-just-a-few-blocks-further-down-on-Mass.-Ave. Phew!

This is the first article in a series of four. Other posts in this series:

Part 2: “A peek inside the block box.”
Part 3: “Predictions from Predictions.”
Part 4: “The Case for Public Transportation.”

Posted May 28, 2013; last substantially updated May 30, 2014

Next: Don’t wait for statistical significance (and other A/B testing lessons learned)

I am Tommy Leung, an engineer and amateur chef. These are my curiosities. (RSS)