That's not really true. Using a beta distribution model (i.e. treating each mile driven as a single event), we can say that with 99% probability the true accident rate is at least 1/1B miles and no more than 7/100M miles.
Before the accident occurs, these numbers were 1/100B miles and 5.3/100M miles.
If a second fatality occurred, those numbers would increase to 3.4/1B miles and 9.3/100M miles.
However, I'm pretty sure that their claims of safety are based on more than just this calculation: "...a wealth of internal data demonstrating safer, more predictable vehicle control performance..."
100M miles of driving is easily enough time to observe things like fewer near misses, greater distance between the Tesla and other cars, etc. In math terms, what Tesla is most likely doing is working off some model relating driving behavior with accident rates and observing that 1 single accident does not invalidate this model.
"How many miles would autonomous vehicles have to be driven to demonstrate that their failure rate is statistically significantly lower than the human driver failure rate? ... Suppose a fully autonomous vehicle fleet had a true fatality rate that was A=20% lower than the human driver fatality rate of 1.09 per 100 million miles, or 0.872 per 100 million miles. We apply Equation 7 to determine the number of miles that must be driven to demonstrate with 95% confidence that this difference is statistically significant ... It would take approximately 5 billion miles to demonstrate that difference. With a fleet of 100 autonomous vehicles test-driven 24 hours a day, 365 days a year at an average speed of 25 miles per hour, this would take about 225 years."
Hmmn. If you assume that increased adoption still leaves us with a similar cohort of drivers traveling on a similar collection of roads with similar driving habits, then this is statistically legitimate.
But each of those assumptions is problematic at best. It's a variant of the "Soldiers in Vietnam" problem. In 1961, the U.S. had 3,200 soldiers in Vietnam, and only 16 of them died. If straight extrapolation worked, you could infer that in 1968, when troop levels peaked at 536,000, we'd be looking at 2,680 deaths.
In Tesla's case, it's likely that a lot of the test miles so far have been incurred on California freeways or their equivalent, where lanes are wide, markings are good, rain is rare and snow is unheard of. Start moving more of the traffic load to narrower roads where nastier weather is common, and you're adding stress factors. Demographics change, too. The high price tag of Tesla vehicles today means that most drivers are likely to be in their prime earning/wealth years (let's say 35 to 70.) If the technology becomes more ubiquitous, we'll need to see how it performs in the hands of people of teenagers and the elderly, too. Once again, the risks rise.
This could be true. If so, their secondary metrics (e.g. # of near misses, avg distance to other cars, etc) should rise as Tesla use expands out of CA.
Again, please note that all I'm saying is that the secondary metrics are the primary predictor here since they are statistically significsnf. The primary metric is just an order of magnitude check since we don't have enough data (note the width of my posterior) to do more than reject the secondary metrics if they are crazily off.
What are you basing that on? Looks like you've misplaced the decimals there someplace, because you're concluding that the likely rate is around 1/10 of the observed rate. Depending on your priors, you can get some difference of course, but that sounds absurd - can you more precisely explain the method you used to arrive at that conclusion?
Finding that the lower end of a credible interval close to zero is an order of magnitude below the top end is pretty common. It just means you need more data, that the beta hasn't approached a gaussian yet.
Please go read the tutorial I linked to. It explains exactly how this analysis works. Here's the code:
In [1]: from scipy.stats import beta
In [2]: from numpy import percentile
In [3]: data = beta(1, 100e6+1).rvs(1e7)
In [4]: percentile(data, 0.5), percentile(data, 50), percentile(data, 99.5)
Out[4]: (5.0460240722165642e-11, 6.9295652915902601e-09, 5.3005747758871231e-08)
You're definitely messing something up somewhere in that analysis. Where's that numpy/betadistribution snippet coming from?
Back to basics: If you assume that each mile is uncorrelated, (on the large not a crazy assumption), then the overall number of accidents over N miles can be modeled by a binomial distribution with N trials and some crash probability p. You're trying to use bayesian analysis to estimate that p.
The core concept there is that the probability of the observation given a certain p is directly proportional to the probability of the p given a certain observation.
Given p, it's trivial to compute the chance of the observation; just plug it into the pmf. For exactly one crash (and a small p), you can even quite accurately approximate it by Np/exp(Np) due to the trivial binomial coefficient for k=1 and the taylor expansion of the base-near-1 power.
So regardless of the numerical analysis that lead you to believe otherwise: you can just plug in numbers can get a numeric approximation of the (scaled) distribution of the probability mass for p. When I do that, both with an exponential prior (i.e. p in 0.1-0.01 is as likely as p in 0.01-0.001) or a uniform prior (i.e. p in 0.1-0.2 is as likely as p in 0.2-0.3) I get the unsurprising result that the maximum likelihood estimation for p is 1/N, and that the median p is a little lower; at around 1 accident in 1.5e8 miles
In practice that means that common sense is correct: if you observe 1 accident in N miles, a reasonable approximation of the best guess of the true accident rate per mile is 1/N.
Now, you can do this analytically, and the conjugate prior of the binomial is indeed a beta distribution. But that forces you to pick parameters for the prior from that beta distribution, and it's easy to do that wrongly.
A reasonable prior might be Beta[1,3] which assumes p is likely closer to 0 than 1. Analytically [then](http://www.johndcook.com/blog/conjugate_prior_diagram/), the posterior distribution is Beta[1+1,3+1e8-1]; with a mean p of 2e-8 (but note that the details of the chosen parameters definitely impact the outcome). This is quite close to the numerically derived p, though probably less accurate since it is constrained to a unreasonable prior.
So I'm not sure exactly what your python script is computing, but commonsense, numerical analysis, and Bayesian analysis all arrive at roughly 1/N in my case - you probably made some mistake in your reasoning somewhere (if you elaborate in detail how you arrive at this numpy snippet, maybe we'll find the error).
Whoops, you are right, my posterior should have been Beta[2,100e6+1] (I chose a uniform prior). With that I get a median probability of 1.7e-8, and a mean probability of 2e-8. Good catch!
Note that this is not particularly sensitive to whether you choose beta[1,1] or Beta[1,3] as the prior.
Yeah, that makes virtually no difference. I'm not familiar with picking these priors, so I could imagine picking (say) Beta[0.5,1] or Beta[5,30] as prior too, and that does make some difference (as in the prior alpha has some impact).
Is there some principled reason to pick one over the other, or is it just "whatever looks reasonable"?
I also can't explain why plain numerical simulation doesn't come up with almost identical p values - I get 1/1.5e8 to 1/1.7e8, but that's 6.25e-9. I mean, it's the same order of magnitude, but it's nevertheless quite different.
Oh well, unless you've got some insight there, I guess I'm going to give up on that analysis - that kind of nitty gritty stuff is a timesink ;-).
Before the accident occurs, these numbers were 1/100B miles and 5.3/100M miles.
If a second fatality occurred, those numbers would increase to 3.4/1B miles and 9.3/100M miles.
However, I'm pretty sure that their claims of safety are based on more than just this calculation: "...a wealth of internal data demonstrating safer, more predictable vehicle control performance..."
100M miles of driving is easily enough time to observe things like fewer near misses, greater distance between the Tesla and other cars, etc. In math terms, what Tesla is most likely doing is working off some model relating driving behavior with accident rates and observing that 1 single accident does not invalidate this model.