Genuinely curious: how would you even go about advancing autonomous driving without testing it in the streets?
I'm absolutely certain that they've attempted to simulate millions of hours of driving, yet this odd behavior happened in real life, which now can be studied and fixed.
If we never had the software on the wild, how would you ever really test it?
>Genuinely curious: how would you even go about advancing autonomous driving without testing it in the streets?
Genuine response:
1 don't let customers do the testing, especially if you don't train them (I mean real training about failures not PR videos and tweets and some small letter manual with disclaimers)
2 use employees, train drivers to test, have some cameras to check the driver to make sure he pays attention.
3 postpone testing until the hardware and software is good enough so you don't ignore static objects.
4 make sure you don't do monthly updates that invalidates all your previous tests.
IMO there is so much 'machine learning' in the tesla self driving system is there any way to know a bug is 'fixed' other than just running it through a probably totally boring set of tests that doesn't even approach covering all scenarios?
Yeah, I guess you could always be safer about it, but I'm really not sure it would be enough. If we substitute FSD for any software, you have code tests, QA, the developers test it, and bugs still go through. It's inevitable.
Unfortunately it's always about the incentives and on a capitalist society the only incentive is money. So even if they could be safer, they wouldn't do it unless it's more profitable, specially being a publicly traded company.
In a sense, a self-driving car might actually be easier to test for than complex software - at least parts of it.
After all, normal (complex) software tends to have lots of in depth details you need to test for; and a surface area that's pretty irregular in the sense that it's hard to do generalized testing. Some bits can be fuzz tested, but usually that's pretty hard. It's also quite hard for a generalized test to recognize failure, which is why generalized test systems need lots of clever stuff like property testing and approval testing, and even then you're likely having low coverage.
However, a self-driving car is amendable to testing in a sim. And the sim might be end-to-end, but it needn't be the only sim you use; the FSD system almost certainly has many separate components, and some of those might be easy to sim for too; e.g. if you have a perception layer you could sim just that; if you have a prediction system you might sim just that; etc.
And those sims needed be full-sim runs either; if you have actual data feeds, you might even be able to take existing runs, and the extend them with sims; just to test various scenarios while remaining fairly close to real world.
I'm sure there are tons of complexities involved; I don't mean to imply it's easy - but it's probably tractable enough that given the overall challenge, it's worth creating an absolutely excellent sim - and that's the kind of challenge we actually have tons of software experience for.
I'm absolutely certain that they've attempted to simulate millions of hours of driving, yet this odd behavior happened in real life, which now can be studied and fixed.
If we never had the software on the wild, how would you ever really test it?