Maximise the video and watch the left hand HUD pane from 0:10 to 0:11.
The dotted black line coming from the front of the car (which I am assuming is the intended route) quickly snaps from straight ahead, to a hair pin right, to a normal right turn.
Ignoring the fact that the right turn happened to be into a pedestrian crossing with people on it - what was the car even trying to do? The sat-nav shows it should have just continued forwards.
I am astounded that software capable of these outputs is allowed on the roads. When could crossing a junction then taking a hairpin right back across it, ever be the correct thing to do?
I've made this argument here a few times and am always shot down, but I think its important to highlight that the airline industry has an extremely robust history of automation and HCI in critical transportation scenarios and it seems to me that all the lessons that we have learned have been chucked out the window with self-driving cars. Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars. The issue of response time, too, is one we cant ignore, and it is certainly a far greater challenge in automobiles.
I don't have answers, but it does seem to me like we are not placing a premium enough on structuring this tech to optimize driver supervision over the driving behavior. Granted, the whole point is to one day NOT HAVE to supervise it all, but at this rate we're going to kill a lot of people until we get there.
Hi, aerospace software engineer and flight instructor here. I think you get shot down because the problems just aren't comparable. While I agree that there may be some philosophical transfer from aircraft automation, the environments are so radically different that it's difficult to imagine any substantial technological transfer.
Aircraft operate in an extremely controlled environment that is almost embarrassingly simple from an automation perspective. Almost everything is a straight line and the algorithms are intro control theory stuff. Lateral nav gets no more complicated than the difference between a great circle and a rhumb line.
The collision avoidance systems are cooperative and punt altogether on anything that isn't the ground or another transponder-equipped airplane. The software amounts to little more than "extract reported altitude from transponder reply, if abs(other altitude - my altitude) < threshold, warn pilot and/or set vertical speed." It's a very long way from a machine learning system that has to identify literally any object in a scene filled with potentially thousands of targets. There's very little to worry about running into in the sky, and minimum safe altitudes are already mapped out for pretty much the entire world.
Any remaining risk is managed by centralized control and certification, which just isn't going to happen for cars. We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty about what the vehicle will encounter when it gets there. Nor are we going to create a centralized traffic control system that provides guarantees you won't collide with other vehicles on a predetermined route.
So it's just a completely different world with completely different requirements. Are there things the aerospace world could teach other fields? Yeah, absolutely. Aerospace is pretty darn good at quality control. But the applications themselves are worlds apart.
I'm actually in complete agreement! What sticks out to me is your assessment that the flight environment is "embarrassingly simple from an automation perspective", which I agree as well (as compared to cars). And yet despite that simplicity and decades at it, we still run it with an incredible robust infrastructure to have a human oversee the tech. We have super robust procedures for checking and cross-checking the automation, defined minimus and tolerances for when the automation needs to cease to operate the aircraft, training solely focused on operating the automation etc. But with cars, we somehow are super comfortable with cars severely altering behavior in a split-second, super poor driver insight or feedback on the automation, no training at all, with a human behind the wheel who in every marketing material known to man has been encouraged to trust the system far more than the tech (or law), would ever have you prudently do.
I'm with you that they are super different, and that the auto case is likely much, much harder. But I see that and can't help but think that the path we should be following here is one with a much greater and healthy skepticism (and far greater human agency) in this automation journey than we are currently thinking is needed.
I agree completely. It's a very difficult problem from a technical perspective, and from a systems perspective, we've got untrained operators who can't even stay off their phones in a non-self-driving car. (Not high-horsing it here; I'm as guilty of this as anyone.) Frankly I'll be amazed if anyone can get this to actually work without significant changes to the total system. Right now self-driving car folks are working in isolation - they're only working on the car - and I just don't think it's going to happen until everyone else in the system gets involved.
> we still run it with an incredible robust infrastructure to have a human oversee the tech
Airplanes are responsible for 200-300+ lives at a time, so it’s quite incomparable to road vehicles. Of course it makes sense to have human oversight in case something goes wrong.
On the flip side, the average car driver is not very skilled nor equipped to deal with most surprises - hence the ever present danger of road traffic.
I’m not sure why AI drivers are held to such insanely high standards.
The claim that self-driving cars are being held-up to a higher standard than human drivers is simply false. Self-driving cars so far have a far worse record than the average of human drivers, which is remarkably good. Human accidents are measure in term of "per million miles driven". Self-driving cars have driven a tiny total distance compared to all the miles human drivers have driven.
I’m reminded of GWB’s “soft bigotry of low expectations”. (Even if you didn’t like him, it remains apt.)
Sorry, most drivers I’ve seen avoid the kid that follows the bouncing ball into the street and all the other random events. A few drivers are crap. But justifying automation based on that few is lazy and incorrect. Elon is just full of his own crap on Tesla’s autopilot.
> most drivers I’ve seen avoid the kid that follows the bouncing ball into the street and all the other random events
I've seen many poor drivers over the decades, but can't think of any human drivers who would reliably and repeatably crash into a stationary fire truck in their lane. "Full Self Driving" vehicles, on the other hand...
> Airplanes are responsible for 200-300+ lives at a time, so it’s quite incomparable to road vehicles
Your analogy is wrong. Buses can have 50 people. I would compare a bus to a medium/small airplane.
Also, it is regrettable that cars don't have the FAA-required electronics, software, or integration processes. When I read that a Jeep's braking system was compromised through its entertainment system it was apparent that the aircraft lessons had not been taken aboard by the auto industry.
> We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty
actually ive been thinking this is exactly the win self driving vehicles have been looking for. upgrade, certify, and maintain cross country interstates like I-70 for fully autonomous, no driver vehicles like freight, mail, hell even passengers. maybe that means one lane with high-vis paint and predefined gas station stops and/or some other requirements. i bet the government could even subsidize with an infrastructure spending bill, politics notwithstanding.
there cant possibly be a problem with _predefined_ highways that is harder to solve than neighborhood and city driving with unknown configurations and changing obstacles. i feel like everyones so rabid for fully autonomous Uber that the easier wins and use cases are being overlooked.
Well, to control things, you'd have to have a highway that's only for self-driving vehicles. And then you'd need to get them there - with what, human drivers? (losing the cost savings) Maybe you could use this for self-driving trucks between freight depots.
The problem with this is - why not just use trains at this point? Trains already an economical solution for point to point transportation.
>fully autonomous, no driver vehicles like freight, mail, hell even passengers. maybe that means one lane ... and predefined gas station stops and/or some other requirements. I bet the government could even subsidize with an infrastructure spending bill, politics notwithstanding.
Yup, that's a train. I really do wish US rail hadn't been turned into what amounts to private property. I live along the Amtrak Cardinal line and would love to use it to travel. But the low speed and frequent stops for higher-priority freight mean a trip takes longer than driving and usually costs more than I would pay in gas.
> We aren't going to live in a world where every street has to be (the equivalent of) an FAA certified airport with controls to remove any uncertainty about what the vehicle will encounter when it gets there.
I do sometimes wish we'd just devote one lane of our larger freeways to self-driving cars exclusively. You could let the cars drive as fast as they want, and charge per-mile tolls for any car that uses it when the road is congested. Ideally, you'd charge based on occupancy as well: an empty car should have to pay more than a car with a human in it, and a car with 2+ people might be allowed to use the lane for free.
Cars are vastly more complex to do navigation for than planes too so we need to be even more careful when making auto autos. Plane autopilots are basically dealing with just the physical mechanics of flying the plane which while complex are quite predictable and modellable. All of the obstacle avoidance and collision avoidance takes place outside of autopilots through ATC and the routes are known and for most purposes completely devoid of any obstacles.
Cars have a vastly harder job because they're navigating through an environment that is orders of magnitude more complex because there are other moving objects to deal with.
Which is one reason I come back to thinking that you may see full automation in environments such as limited access highways in good weather but likely not in, say, Manhattan into the indefinite future.
Unexpected things can happen on highways but a lot fewer of them and it's not like humans driving 70 mph are great at avoiding that unexpected deer or erratic driver either.
ADDED: You'd actually think the manufacturers would prefer this from a liability perspective as well. In a busy city, pedestrians and cyclists do crazy stuff all the time (as do drivers) and it's a near certainty that FSD vehicles will get into accidents and kill people that aren't really their fault. That sort of thing is less common on a highway.
This is what I'd be happy with. Something to get me the 20-50 miles between cities (Atlanta<->Winder<->Athens in particular), or through the closed highway loops around them. Driving within them isn't so boring that my focus wanders before I notice it's wandering.
We could just expand MARTA, but the NIMBY crowd won't allow it. People are still hopped up on 1980s fearmongering about the sorts of people who live in cities and don't want them infesting their nice, quiet suburbs that still have the Sherriff posting about huge drug and gun busts.
Public transit isn't a panacea for suburbs/small cities. I'm about a 7 minute drive from the commuter rail into Boston but because of both schedule and time to take a non-express train, it's pretty much impractical to take into the city except for a 9-5 workday schedule, especially if I need to take a subway once I get into town.
For me, it's more the 3-5 hour drive, mostly on highways, to get up to northern New England.
> Which is one reason I come back to thinking that you may see full automation in environments such as limited access highways in good weather but likely not in, say, Manhattan into the indefinite future.
This is why there are SAE levels of automation and specifically Level 4 is what you're describing. Anyone claiming their system will be Level 5 is flat out lying.
Even just in the US, there are some cities that can be fairly challenging for an experienced human driver, especially one unfamiliar with them. And there are plenty of even paved mountain roads in the West which can be a bit stressful as well. And that's in good weather.
Highways are probably the best case scenario for full automation but we've seen scenarios where even that idealized environment has deadly failures in a handful of Tesla crashes.
They're still in a vastly more controlled environment than cars and moving at much lower speeds as well. If an auto taxiing plane has to avoid another airport vehicle something has gone massively wrong. Judging by this video [0] I'm not sure they're even worrying about collisions and are counting on the combination of pilots and ATC ground controllers to avoid issues while taxiing. It looks like the cameras are entirely focused on line following.
I recently avoided a startup prospect because they were looking to build a car OS that wasn't hard real-time. The very idea that they're trying to develop such things that might intermittently pause to do some garbage collection is freaking terrifying.
What has hard real time have to do with garbage collection? You can have concurrent GCs (with [sub]millisecond pauses, or no Stop-the-world at all) but you also need 'hard real time' OS to begin with. Heck, even opening files is far from real-time.
Then you need: not-blocking data strcutures, not just lock-free - that are much easier to develop. Pretty much you need forward guarantees on any data structure you'd use.
You usually need garbage collection because you are allocating in the first place. And allocating and releasing adds some non-determinism. You apparently don't know how much exactly needs to be allocated - otherwise you wouldn't have opted for the allocations and GC. That non-determinism translates to a non-determinism in CPU load, as well as to "I'm not sure whether my program can fulfill the necessary timing constrains anymore".
So I kind of would agree that producing any garbage during runtime makes it much harder to prove that a program can fulfill hard realtime guarantees.
To play devil's advocate...and because I'm just not that educated in the space, is this really a huge deal? If we're talking couple ms at a time delays, isn't that still vastly superior to what a human could achieve?
If you can set a guaranteed maximum on the delays (regardless of what those limits are), you're hard real-time by definition. The horror is that they weren't building a system that could support those guarantees.
What if say, a system is written against an indeterminate timed GC like say, Azul or Go's, but code is written in a way that proves GC times never exceed X, whether by theory or stress testing. Is this still seen as 'unguaranteed'?
It depends on your system model (e.g. do you consider memory corruption to be a valid threat), but it could be. In practice, actually establishing that proof purely in code is almost always impractical or doesn't address the full problem space. You use hardware to help out and limit the amount of code you actually have to validate.
GC pauses can be hundreds of milliseconds. You could perhaps use a particular GC scheme that guarantees you never have more than a couple millisecond pause, but then you have lots of pauses. That might have unintended consequences as well. I'm also not sure that such GCs, like golangs, can really mathematically guarantee a minimum pause time.
Hard real time garbage collectors have existed for decades. Of course you can mathematically guarantee a minimum pause time given a cap on allocation rate. What's stopping you?
You don't even need a cap on allocation rate, GC can have during allocation w/o fully blocking, it'd 'gracefully' degrade the allocation, itself. It'd be limited by CPU/memory latency and throughput.
If ernie the intern decides to use a hashmap to store a class of interesting objects as we drive, you could end up with seconds of GC collection + resizing if it grows big enough.
That doesn't seem especially bad. The car could, for instance, predict whether or not it was safe to do garbage collection. Humans do the same when they decide to look away while driving.
A human that looks around while driving is still taking in a lot of real-time input from the environment. Assuming they're not a terrible driver they examined the upcoming environment and made a prediction that at their current speed there were no turns, obstacles, or apparent dangers before looking away. If they didn't fully turn their head they can quickly bring their eyes back to the road in the middle of their task to update their situation.
If a GC has to pause the world to do its job there's none of that background processing happening while it's "looking away".
That's horrifying on such a deep level. There should be mandatory civil service for programmers, but you just get sent somewhere cold and you gotta write scene demos and motion control software for a year to get your head on straight. :P
To echo this, as someone who has done some work with formal specifications, I have to say it seems like the self-driving car folks are taking a "move fast and break things" approach across the board, which is horrifying.
The mechanism by which those lessons were learned involved many years full of tragedy and many fatalities including many famous celebrities dying in those plane crashes. Obviously, we do not want to follow that same path, but at the moment that's exactly the path we're on.
The US govt isn't going to do anything until there's a public outcry, and historically there won't be a public outcry until there's a bunch of famous victims to point to.
> The US govt isn't going to do anything until ...
I think this attitude is defeatist and absolves us of doing anything. It's a democracy; things happen because citizens act. 'The US government isn't going to do anything' as long as citizens keep saying that to each other.
> it seems to me that all the lessons that we have learned have been chucked out the window with self-driving cars.
I think it’s unfair to lump all self driving car manufacturers together.
The traditional car companies have been doing research for decades (see for example https://en.wikipedia.org/wiki/VaMP), but only slowly brought self-driving features to the market with part of the slowdown because they are aware of the human factors involved. That’s why there’s decades of research on ways to keep drivers paying attention and/or detecting that they don’t.
“Move fast and break things” isn’t their way of working.
These lessons have been chucked out the window by second tier (e.g., GM/Cruise) and third tier (e.g. Tesla and Uber) competitors, who have recognized that the only way they can hope to catch up is by gambling that what happened to Uber won't happen to them.
The car FSD - aircraft autopilot analogy is deeply flawed, and nowhere near instructive. Let's consider some details:
What aircraft autopilot does is following a pre-planned route to the T, with any changes being input by humans. The aircraft autopilot doesn't do its own detection of obstacles, nor of router markings; it follows the flight plan and reacts to conditions of the aircraft. Even when executing automatic take-off and landing, the autopilot doesn't try to detect other vehicles or obstacles - just executes the plan, safe in knowledge that there are humans actively monitoring for safety. There is always at least two humans in the loop: the pilot in command who prepared and inputed the original flight plan and also inputs any route changes when needed (collision and weather avoidance), and an air traffic controller that continuously observes flight paths of several aircrafts and is responsible for ensuring safe separation between aircraft in his zone of responsibility. Beyond that, an ATController has equal influence on all aricraft in his zone of responsibility, and in case one does something unexpected, it can equally well redirect that one or any other one in vicinity. Lastly, due to much less dense traffic, the separation between aircraft is significantly larger than between cars [1] providing time for pilots to perform evasive maneuvers - and that's in 3d space, where there are effectively two axes to evade along.
Conversely with car FSD - the system is tasked both with following the route, and also with continuously updating the route according to markings, traffic, obstacles, and any contingencies encountered. This is a significant difference in quantity from the above - the law and the technology demands one human in the loop, and that human can only really influence his own car at most. Even worse, due to density of traffic, the separation between cars is quite often on the order of seconds of travel time, making hand-over to driver a much more rapid process.
I am moderately hopeful for FSD "getting there" eventually, but at the same time I'm wary of narrative making unwarranted parallels between FSD and aircraft autopilot.
> Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars.
Is that actually the case, though?
I would hope, although perhaps I'm mistaken, that the developers of the actual self-driving systems would be able to effectively reason about what's happening. For example, would a senior dev on Tesla's FSD team look at the video from the article and have an immediate intuitive guess for why the car did what it did? Or better yet, know of an existing issue that triggered the wacky behavior?
Even if not, I'd hope that vehicle logs and metrics would be enough to shed light on the issue.
I don't think I've ever seen a true expert, with access to the full suite of analytic tools and log data, publish a full post-mortem of an issue like this. I'm certain these happen internally at companies, but given how competitive and hyper-secretive the industry is, the public at large never sees them.
They certainly are trying very hard, as far as I can tell. Tesla's efforts on data collection and simulation of their algorithm are incredibly impressive. But part of why it is so necessary is that there is an opaqueness to the ML decision-making that I don't think anyone has quite effectively cracked. I do wonder, for instance, if the decision to go solely with the cameras and no LIDAR will prove to ultimately be a failure. The camera-only solution requires the ML model to accurately account for all obstacles, for example. As crude, and certainly non-human as it is, a LIDAR with super crude rules for "dont hit an actual object" would have even at this point prevented some of their more widely publicized fatal accidents which relied on the algorithm alone.
there are keys difference between automation in e.g. aircraft, and what Tesla at al are failing at,
e.g., how constrained the environment is; and what the exposure is to anomalous conditions is; and what the opportunity window usually is to turn control back over to a human.
The thing I don't understand is, we have a much more comparable environment in ground travel: federal highways.
Innumerable regressions and bugs and lapses aside, I do not understand why so much effort is being wasted on a problem which IMO obviously requires AGI to reach a threshold of safety we are collectively liable to consider reasonable; when we could be putting the same effort into the (also IMO) much more valuable and impactful case of optimizing automated traffic flow in highway travel.
Not only is the problem domain much more constrained, there is a single regulatory body, which could e.g. support and mandate coordination and federated and data sharing/emergent networking, to promote collective behavior to optimize flow in ways that humans limited-information self-interested human drivers cannot.
The benefits are legion.
And,
I would pay 10x as much to be able to cede control at the start of a 3-hour trip to LA, than to be able to get to work each morning. Though for a lot of Americans, that also is highway travel.
Not just this, why not start with the low-hanging case of highway travel, and work out from there onto low-density high-speed multi-lane well-maintained roads? Yes that means Tesla techs who live in Dublin don't get it first. Oh well...
IMO there will never be true, safe FSD in areas like my city (SF) absent something more more comparable to AGI. The problem is literally too hard and the last-20% is not amenable to brute forcing with semantically vacuous ML.
I just don't get it.
Unless we take Occam's razor, and assume it's just grift and snake oil to drive valuation and investment.
Maybe the quiet part and reason for engineering musical chairs is just what you'd think, everyone knows this is not happening; but shhhh the VC.
I'm on my third Tesla. FSD on highways has improved so much in the last 6 years. On my first Tesla, autopilot would regularly try to kill you by running you into a gore point or median (literally once per trip on my usual commute). I now can't even remember the last time I had an issue on the highway.
Anywhere else is basically a parlor trick. Yes, it sorta works, a lot of the time, but you have to monitor it so closely that it isn't really beneficial. As you point out, its going to take some serious advances (which in all likelihood are 30+ years away) for FSD to reliably work in city centers.
I think the issue you've highlighted is one of governance. There's only so much Tesla can do regarding highways. You really need the government to step in to mandate coordination of the type I think you're envisioning. And the government is pretty much guaranteed to fuck it up and adopt some dumb standard that kills all innovation after about 6 months, so it never actually becomes usable.
I think automakers will eventually figure this out themselves. As you say, there are too many benefits for this not to happen organically. Once vehicles can talk to each other, everything will change.
To be fair, it still felt like magic. My car would drive me 20 miles without me really having to do anything, other than make sure it didn't kill me at an interchange.
And I'm now trying to remember, but I think autopilot was initially free (or at least was included with every car I looked at, so it didn't seem like an extra fee). Auotpilot is now standard on all Teslas, but FSD is an extra $10k, which IMO is a joke.
Humans are ridiculously bad at overseeing something that mostly works. That’s why it is insanely more dangerous.
Also, the problem is “easy” for the general case, but the edge cases are almost singularity-requiring. The former is robot vacuum level, the latter is out of our reach for now.
is why there we haven't seen the feds stepping in via the transportation agency to develop and regulate exactly this, with appropriate attention paid to commercial, personal, and emergency vehicle travel.
The opportunities there appear boundless and the mechanisms for stimulating development equally so...
I really don't get it. Then I think about DiFi and I kind of do.
Even lower-level automation for highway driving would be super useful.
I would appreciate a simple "keep-distance-wrt-speed" function for bumper-to-bumper situations. Where worst case scenario, you rear-end a car at relatively low speeds.
I'd happily keep control over steering in this situation and just keep my foot over the brake, though lane-keep assist would probably be handy here as well. A couple radar sensors/cameras/or lidar sensors would probably be enough for basic functionality.
Disable if the steering-wheel is turned more than X degrees - maybe 20 or 25?. Disable if speed goes over X speed - maybe 15mph? Most cruise controls require a minimum speed (like 25mph) to activate.
Trying to do full driving automation, especially in a city like Seattle, is like diving into the ocean to learn how to swim.
As cool as that sounds, I'd trust incremental automation advancements much more.
I would appreciate a simple "keep-distance-wrt-speed" function for bumper-to-bumper situations.
This has been widespread for at least a decade. I'm not even aware of a mainstream auto brand that doesn't offer adaptive cruise control at this point. Every one I've used works in stop and go traffic.
The other features you want are exactly what Tesla has basically perfected in their autopilot system, and work almost exactly as you describe (not the FSD, just the standard autopilot).
I can't say that my experience agrees with this. Maybe some higher-end vehicles had it a decade ago, but it seems to be getting more popular over the past 5 years or so. I still don't see it offered on lower priced vehicles where basic cruise functionality is there, but I doubt ever will be a part of "entry level" considering the tech required.
None of the vehicles my family owns have a "stop-and-go" cruise control function - all newer than 10 years. ACC at higher speeds is available on one, but it will not auto-resume if the vehicle stops.
In this same vein cars with automated parallel/back-in-angle parking.
I think an even more 'sensor fusion' approach needs to be adopted. I think the roads need to be 'smart' or at least 'vocal'. Markers of some kind placed in the roadway to hint cars about things like speed limit, lane edge, etc. Anything that would be put on a sign that matters should be broadcast by the road locally.
Combine that with cameras/lidar/whatever for distance keeping and transient obstacle detection. Then network all the cars cooperatively to minimize further the impacts of things like traffic jams or accident re-routing. Perfect zipper merges around debris in the roadway.
Once a road is fully outfitted with the marker system, then and only then would I be comfortable with a 'full autopilot' style system. Start with the freeway/highway system, get the logistics industry on board with special lanes dedicated to them and it becomes essentially a train where any given car can just detach itself to make it's drop off.
Also, what could possibly save the most lives: simply ML the hell out of people’s faces to notice when they are getting sleepy. That’s almost trivial and should be mandatory in a few years.
A more advanced problem would be safely stopping in case the driver falls asleep/looses consciousness, eg. on a highway. that short amount of self-driving is less error-prone than the alternative.
>Unless we take Occam's razor, and assume it's just grift and snake oil to drive valuation and investment.
I think there's some of that. Some overconfidence because of the advances that have been made. General techno-optimism. And certainly a degree of their jobs depending on a belief.
I know there is a crowd of mostly young urbanites who don't want to own a car and want to be driven around. But I agree. Driving to the grocery store is not a big deal for me. Driving for hours on a highway is a pain. I would totally shell out $10K or more for a full self-driving system even if it only worked on interstates in decent weather.
Highway self driving has been around for decades[1] - and Tesla's general release autopilot can already do all that. As I understand it from ramp on to ramp off, in production vehicles, Tesla can provide an automated experience. I'm not sure how much "better" it can get.
>Highway self driving has been around for decades[1]
Driver assisted highway has been around for years... Level 5 driving requires no human attention. Huge difference.
I think what is wanted by many is level 5 on the highways. I want to sleep, watch a movie, whatever. That is much, much "better" than what we have now. Like many others, I would be most interested in full level 5 on highways and me driving in the city. That is also much easier to implement and test. The scope of the problem is greatly reduced. I think Tesla and others are wasting tremendous resources trying to get the in-city stuff working. It makes cool videos, but being able to do other activities during a 5 hour highway drive has much more value (to me at least) than riding inside a high risk video game on the way to work.
(edit) I get that I am misusing the definition of "level 5" a bit, but I think my meaning is clear. Rated for no human attention for the long highway portion of a trip.
better would be that i can legally go to sleep and let the car drive the highway portion of my trip. then i wake up and complete the final leg of the trip.
as you say, i don’t think this is entirely out of reach (even if it required specialized highway infrastructure or car to car communication). seems like lower hanging fruit than trying to get full self driving working on local/city roads.
i would pay a ton for the highway only capability…
I've watched quite a few FSD videos and in almost every single one the car barely knows what it wants. The route jumps all over the place. I'm pretty sure it's just the nature of their system using cameras and the noisy data that can generate.
The sat nav didn't update the route to go that way. The FSD system decided to. It probably completely lost track of the road ahead because of the unmarked section of the road and just locked on to the nearest road it could find, which was that right turn.
I've seen videos where it cross over unmarked road and then wants to go into the wrong lane on the other side because it saw it first. It seems like it will just panic search for a road to lock onto because the greatest contributor to their navigation algorithm is just the white lines it can follow.
If you watch the Tesla AI day presentation they explain the separation of duties that kind of explains what's happening.
The actual routing comes from something like Google Maps that is constantly evaluating driving conditions and finding the optimal route to the destination. It does this based on the vehicle's precise location but irrespective of the vehicle's velocity or time to next turn.
The actual AI driving the car is trying to find a path along a route that can change suddenly and without consideration. It's like when your passenger is navigating and says "Turn here now!" instead of giving you advanced notice.
That could be selection bias. The [edit, was '99%'] vast majority of the time the car does the boring thing are not click-worthy. Have you driven a Tesla for a reasonably long period of time?
There is a bigger lesson in there: click-driven video selection creates a very warped view of the world. The video stream is dominated by 'interesting' events, ignoring the boringly mundane that overwhelmingly dominates real life. A few recent panics come to mind.
99% perfect is not good enough, it's horrifyingly bad for a car on public roads.
When I drive, I don't spend 1 minute plowing into pedestrians for every 1 hour and 39 minutes I spend driving normally.
If a FSD Tesla spends just 99% of its time doing boring, non-click-worthy things, that is itself interesting in how dangerous it is.
To your point, I'm definitely interested in knowing how many minutes of boring driving tend to elapse between these events. The quantity of these sorts of videos that have been publicized recently gives me the impression that one of these cars would not be able to spend 100 minutes of intervention-free driving around a complex urban environment with pedestrians and construction without probable damage to either a human or an inanimate object.
99% is a an very rough colloquial estimate meaning 'the vast majority of the time' to drive the point. Could well be 99.999999%. What really matters is how it compares with human performance, I don't have data to do that comparison. The only ones that can make the comparison are Tesla, modulo believing data from a megacorp in the wake of the VW scandal.
FYI, that 99.999999 number you quoted is still bad. It means around 30 minutes of the machine actively trying to kill you or others while driving on public roads. I assumed a person driving 2 hours a day for a year.
FSD should not be allowed on the road, or if it is it should be labeled as what it really is: lane assist.
I'm not 'quoting' any numbers. I don't own a Tesla. I don't trust lane assist technology, the probability of catastrophic failure is much larger even compared with dynamic cruise control. I'll steer the wheel thank you very much. I'm not a techno-optimist, rather the contrary. I would like independent verification of the safety claims Tesla makes, or any other claims made by vendors of safety-critical devices.
What I am saying is that selection bias has exploded in the age of viral videos, and this phenomenon doesn't receive anywhere near the attention it deserves. We can't make sound judgements based on online videos, we need quality data.
I have an intersection near where I live where the Tesla cannot go thru in the right lane without wanting to end up in the left lane past the intersection (I guess it's a bit of a twisty local road). At first, it would be going in a straight line, then when it hit the moment of confusion would snap into the left lane so quickly, some 200 ms or something. Never tried it with a car in that spot fortunately. After a nav update, it now panics and drops out of auto-pilot there and has you steer into the correct lane. Nothing to do with poor visibility or anything, just a smooth input that neatly divides its "where is the straight line of this lane" ML model, or whatever.
It's actually fascinating to watch - it just clearly has no semantics of "I'm in a lane, and the lane will keep going forward, and if the paint fades for a bit or I'm goign thrur a big intersection, the lane and the line of the lane is still there."
It also doesn't seem to have any very long view of the road. I got a Tesla at the same time of training my boys to drive, and with them I'm alwys emphasizing when far away things happen that indicate you have increased uncertainty about what's going on down the road. (Why did that car 3 cars ahead break? Is that car edging towards the road going to cut infront? Is there a slowdown past the hill?) The Tesla has quick breaking reflexes but no sign of explicitly reasoning about semantic layer uncertainty
Yes, for 4 years I did, and what they're saying is absolutely true. It desperately tries to lock on to something when it loses whatever it's focused on. Diligent owners just learn those scenarios so you know when they're coming. Others may not be so lucky.
For example, short, wavy hills in the road would often crest just high enough that once it couldn't see on to the other side, it would immediately start veering into the oncoming lane. I have no idea why it happened, but it did, and it still wasn't fixed when I turned in my car in 2019. I drove on these roads constantly so I learned to just turn off AP around them, and it helped traffic on the other side was sparse, but if those weren't true, I'd only have a literal second to response.
EDIT: IMO the best thing it could do in that scenario is just continue on whatever track it was on for some length of time before its focus was lost. Because it "sees" these lines it's following go off to the right/left (such as when you crest a hill, visually the lines curve, or when the lines disappear into a R/L turn) but only in the split second before they disappear. Maybe that idea doesn't fit into their model but that was always my thought about it.
From the admittedly not much footage of Tesla FSD (1-2 hours total maybe) I’ve watched, it seems to be roughly on par with a student driver who occasionally panics for no apparent reason.
While most people see the student vehicle and give it a wide berth, there are those that see it as a "teachable" moment. We've already seen that asshats that screw with cars labeled with Student Driver just to "give 'em a lesson" are already screwing with self driving cars for the same reason: They're assholes.
Assuming by hater you mean "someone who is scared by Full Self Driving including cars veering into pedestrians", it'd help me: I'm in an urban area and don't have a car, and I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not.
Here's my hater take: "Full Self Driving" is an absolutely dangerous and egregious marketing lie. The vehicles are not full self driving and require by law a driver capable of taking over at a moments notice.
I do not believe that Tesla's technology will ever achieve "Full self driving" (levels 4 or 5, where a driver isn't required https://www.nhtsa.gov/technology-innovation/automated-vehicl...), and the labeling of their "conditional automation" system as "full self driving" is an absolute travesty.
People do honestly think they have bought a self driving car. The marketing lie absolutely tricks people.
The problem is that you could apply that argument for a drunk driver, a distressed driver, a driver without a license, a driver that hasn't slept enough, etc.
Any of those could mean a car could randomly turn into the crosswalk.
Humans are horrible at assessing risk. We worry about Teslas because it's on the headlines now, but you're infinitely more likely to get ran over by one of the cited examples than a Tesla just by sheer statistics.
Never mind the driver who can't wait the two to five seconds for the pedrestrian to cross, which is probably far more common than the above, and rather intentional.
TBH I don't think it's ever crossed my mind whether a Tesla was in FSD or not, even when driving alongside one. As long as it doesn't enter my space, I'm fine.
Walking through Manhattan, I'm more worried about the human driver nowadays than the non-human.
"I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not."
Sure - which is why I said I agreed there should be some indication of it being in FSD or not, originally.
...and it is obvious that it will actually ever get more than incrementally technically better, absent AGI.
The problem is too hard for existing tools. Good job on the 80% case; let's look for problems for which the 80% case is good enough and back away from unsupportable claims about "just a little more time" bringing FSD in environments defined by anomaly and noise, in which the cost of error is measured in human life and property damage.
As well as he/she has a professional driver paying close attention to whatever he does. Overseeing something is a really mentally trying thing and “fsd” car drivers will not be able to do that for long times.
You have it backwards. With humans driving there is a constant and unending stream of student drivers, because every new human must learn from scratch. There is no silver lining.
With self driving it takes longer for the one individual to learn, but the difference is that there is only the one individual, and it need not ever die. The learning is transferable between cars, regressions between generations need not be accepted. The problem of bad driving can finally be solved.
Given the number of different people pouring millions to billions of dollars at the problem, I think it's pretty incredible for you to be so certain that that is the case.
I mean... 5 minutes on /r/IdiotsInCars and it's very easy for me to understand how even with bizarre bugs like this, it's better than people.
People say "how could it happen?!" and get angry for a software bug - that can be fixed - and seem to accept how drunk drivers do this (and a lot worse) literally every single day. But since it's a human, that's fine!
There are pretty stringent laws against drunk driving in most places... if Tesla's "self driving" modes are that bad then they should be equally illegal
Genuinely curious: how would you even go about advancing autonomous driving without testing it in the streets?
I'm absolutely certain that they've attempted to simulate millions of hours of driving, yet this odd behavior happened in real life, which now can be studied and fixed.
If we never had the software on the wild, how would you ever really test it?
>Genuinely curious: how would you even go about advancing autonomous driving without testing it in the streets?
Genuine response:
1 don't let customers do the testing, especially if you don't train them (I mean real training about failures not PR videos and tweets and some small letter manual with disclaimers)
2 use employees, train drivers to test, have some cameras to check the driver to make sure he pays attention.
3 postpone testing until the hardware and software is good enough so you don't ignore static objects.
4 make sure you don't do monthly updates that invalidates all your previous tests.
IMO there is so much 'machine learning' in the tesla self driving system is there any way to know a bug is 'fixed' other than just running it through a probably totally boring set of tests that doesn't even approach covering all scenarios?
Yeah, I guess you could always be safer about it, but I'm really not sure it would be enough. If we substitute FSD for any software, you have code tests, QA, the developers test it, and bugs still go through. It's inevitable.
Unfortunately it's always about the incentives and on a capitalist society the only incentive is money. So even if they could be safer, they wouldn't do it unless it's more profitable, specially being a publicly traded company.
In a sense, a self-driving car might actually be easier to test for than complex software - at least parts of it.
After all, normal (complex) software tends to have lots of in depth details you need to test for; and a surface area that's pretty irregular in the sense that it's hard to do generalized testing. Some bits can be fuzz tested, but usually that's pretty hard. It's also quite hard for a generalized test to recognize failure, which is why generalized test systems need lots of clever stuff like property testing and approval testing, and even then you're likely having low coverage.
However, a self-driving car is amendable to testing in a sim. And the sim might be end-to-end, but it needn't be the only sim you use; the FSD system almost certainly has many separate components, and some of those might be easy to sim for too; e.g. if you have a perception layer you could sim just that; if you have a prediction system you might sim just that; etc.
And those sims needed be full-sim runs either; if you have actual data feeds, you might even be able to take existing runs, and the extend them with sims; just to test various scenarios while remaining fairly close to real world.
I'm sure there are tons of complexities involved; I don't mean to imply it's easy - but it's probably tractable enough that given the overall challenge, it's worth creating an absolutely excellent sim - and that's the kind of challenge we actually have tons of software experience for.
A law doesn't prevent anything, it only applies after the fact. You could argue that the prospect of being prosecuted might scare people into not doing the thing that they are not allowed to be doing, but with all the people doing the things they are not allowed to be doing anyway, I doubt a comparative legal argument helps here.
You could make a law that states that your FSD has to be at least as good as humans. That means you have the same post-problem verification but now the parallel with drunk drivers can be made.
Tesla’s “FSD” is only a little bit better than drunk drivers, whom we punish severely whenever caught, even before any accident occurs.
The fact that enforcement is patchy is irrelevant — drunk driving is deemed serious enough to be an automatic infraction.
Also, most of the time drunk drivers are not actually that bad at moment-to-moment driving. That’s why almost everyone worldwide used to do it! You can still do the basics even when reasonably drunk. That doesn’t make you safe to drive. It’s still incredibly dangerous to do.
This assumes FSD-to-drunk-driver analogy means FSD has to be a drunk driver (or a student driver as commented elsewhere) all the time, so always making the bad judgement and slow reaction like a drunk driver would.
I think that some form of responsibility has to be assigned to the FSD in some way (the manufacturer? the user? some other entity or a combination?) regardless but I haven't found any clear case of how that would work.
It also makes me wonder how we would measure or verify a human driver with intermittent drunkenness. Imagine 5 seconds out of every minute you temporarily turn into a drunk driver. That's plenty of time to cause a major accident and kill people, but on the other hand that would mean that the combination of a situation where that would happen and the right timing to not be able to judge that situation has to apply. We do of course have the luxury of not having humans constantly swapping drunk and normal driving, so it isn't a realistic scenario, but it would make for a better computer analogy.
Besides drunk drivers we also have just generally crappy drivers that just happened to get lucky when doing their driving test (although there are places where no meaningful test is required so that's a problem in itself).
I think you’ve missed my point, while adding some additional information.
- drunk drivers are also not uniformly awful drivers: they can drive OK for the most part
- they still drive unacceptably poorly
- we strictly punish them on detection, before any potential accident
- drunk drivers and FSD have more in common with each other than competent drivers and FSD
- why is FSD not held to such a preventative standard?
One can argue that FSD is like a drunk driver driving a student training car with two sets of pedals and two steering wheels, and the Tesla owner/driver is like a driving instructor. But driving instructors are trained and paid to be quite vigilant at all times. Tesla play a sleight of hand and say it’s a labor saving technology, but also you need to be able to behave like a trained and paid driving instructor... that is a conspicuous contradiction.
And I’m ignoring future FSD capabilities because while I’d be happy for it to come about, we should discuss the present situation first, and I don’t believe it’s a good example where sacrificing lives now is acceptable in order to potentially save lives in the future.
Perhaps it is lost in translation; I'm not saying the fact that someone is driving drunk only matters when an accident happens, I'm saying that right until the moment a driver decides to get drunk, the law doesn't do anything. If at the beginning of the day someone decides to start drinking and when they are drunk they get in to a car and start driving, that's when the violation occurs. Not before that like PreCrime would.
The same can't apply to FSD because it isn't consistently 'driving drunk'. That analogy doesn't hold because it is not fixed software like a GPS-based navigation aid would be. Just like humans it does have more or less fixed parameters like the amount of arms and legs you have, that doesn't tend to change depending on your intoxication.
One could make the argument that it's not as much the "haha it is just like a drunk driver zig-zagging", but the uncertainty about the reliability. If a car with some autonomous driving aid drives across an intersections just fine 99 times out of a 100, and that one time it doesn't, that doesn't mean the car software was dunk 100% of the time.
Why FSD is not held to some standard, I don't know. I suppose that depends on how it is defined by local law and how the country it is in allows or disallows its use.
The problem with prevention and detection here is that like humans, the system is not in a static state. The trained neural network might be largely the same with every release, but the context in which it operates isn't, unless the world around it stops completely in which case two trips can be identical and because the input is identical the output can also be identical. Humans do the same, even if well-rested and completely attentive, knee-jerk reactions happen.
Holding FSD to a standard of a drunk driver isn't a valid comparison due to the non-static nature of the state it is in. This isn't even FSD-specific, even lane guidance/keeping assistance and adaptive cruise control isn't static, and those are based on pretty static algorithms. Even the PID-loops used on those will deliver different results on seemingly similar scenarios.
Perhaps we should stop comparing technology to humans since they are simply not the same. The static kind isn't and neither is a NN-based one. We can still explore results or outcomes because those are the ones that have real impact. And let's not fool ourselves, humans are far less reliable in pretty much every man-machine combination. But in human-to-human contexts we factor in those unreliabilities, and with machine-to-human or machine-to-machine we seemingly don't, which is pretty much the same problem you're describing.
This will be an interesting field of development, and if we simply take death toll into account, keep in mind that for some reason seatbelts were thought to have 'two sides of the story' as well when they were first introduced and later required. As with car seats for children and infants, a good idea might start out one way and over time (with the accompanying bodycount) it gets shaped into whatever we expect of it today. Same goes for aerospace, boats and trains, and that's even without taking software into account.
> It also makes me wonder how we would measure or verify a human driver with intermittent drunkenness. Imagine 5 seconds out of every minute you temporarily turn into a drunk driver.
Certain medical conditions are analogous to this. This is handled in various ways, including doctors reporting you to the state DMV and suspending your license if your condition is not very well controlled.
The key difference is human drivers have independent software, whereas the same software powers all FSD Teslas. One human driver getting drunk/otherwise impaired doesn't affect the software of any other human driver; but if your FSD software is as good as a drunk human, then every single one of your FSD vehicles is a road hazard.
That difference is also a benefit, fix one problem, and it's fixed for every instance.
On the other hand, it's not like the software is always bad and always in the same situation. That is a big difference with a human analogy; a drunk driver taking a trip in the car is drunk for the entire trip (unless it's a very long trip etc..), so would be impaired for the entire duration.
There are plenty of people and vehicles that are road hazards and are allowed to drive (or be driven) anyway, so if we really cared about that aspect on its own we could probably do with better tests and rules in general.
What I mean is that you don't see a headline on news outlets or twitter hashtags trending for every drunk driver fatality or crash that happens. Which for me tells that collectively, we are fine with it. We accept it as ordinary, as crazy as it may be.
I regularly see "traffic collision kills [person or people]" stories in my local newspaper. I don't know if it's every one, it might be restricted to the extremely stupid (wrong-way on the highway, drunk, or excessively-high-speed) but "someone got killed" is certainly a common story in the news here.
I've certainly seen many more local "person dies in car accident" non-brand-related stories than "person dies in Tesla accident" stories.
A DUI, in California for instance, means you’re suspended for months (years if you’re a repeat offender). Since it’s basically the same “driver” driving all FSD Teslas, are you arguing that they should be suspended under the same rules for drunken behavior? If so, they will basically be out of commission indefinitely, and we won’t see headlines anymore.
No, my point is that this is overblown just because it's Tesla and people love drama, so this makes headlines and trending topics.
My comparison with drunken drivers is just with the regards to the odd behavior. If you look at a drunk driver (or even someone that fell asleep in the wheel) with a near crash, many times it would resemble this video. But the outrage from the public differs vastly.
My Audi e-tron has this habit of switching the adaptive cruise control to the on-ramp speed limit even when I’m in the middle of the freeway.
It’s something I’ve learned to deal with but the sudden attempt to break from 70 to 55 is pretty bad especially as it’s unexpected for other drivers around you.
While I’m sure Audi are much worse at updating their software to fix known issues than Tesla I find myself skeptical that the mix of hacks cars use to implement these features scale well, especially in construction zones. Hence I’m pretty content with radar based cruise control and some basic lane maintenance and then doing the rest of the driving myself.
I can imagine if my car were slightly better at some of these things I’d be a significantly worse safety driver as I’d start to be lulled into a false sense of security.
As I understood the Tesla AI presentations the path is determined using a Monte Carlo Beamsearch which looks for a feasible path while optimizing an objective that includes minimizing sideways g-forces and keeping the paths derivatives low (smooth path).
Form the videos I have seen this fails often (in that the path doesn’t go straight even if it can). Knowing a bit about randomized met heuristics myself, I am not surprised.
I think, they need to perform some post processing on these paths (a low iteration local search).
I think, they should also start with a guess (like just go straight here, or do a standard turn), and then check if the guess is ok. I think, that could help with a lot of the problems they have.
For anyone interested the Monto Carlo Tree Search used for the planning (that the tentacle shows) is described here (from the AI day video, at the 1h21m50s):
The car almost certainly decided that the road ahead was blocked. For context, this release of FSD is the first to use a new obstacle detection technique, and as a result it is the first one that doesn't drive directly into the pillars on that street without perceiving them at all. So it's very likely that this new obstacle detection system glitched out here.
The guy in the video is intentionally stress testing the system by repeatedly driving in an area he knows is poorly handled. But it's totally fair IMO, this is a real road and while it confuses humans too (as evidenced by all the people here claiming that it's illegal to turn right here), FSD must handle it better than this.
Looking at the video Tesla was probably not closer than 25 feet from any pedestrians. If pedestrians were closer I think their proximity would make the car stop as in other videos.
As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.
Because they are sudden and unpredictable, evasive maneuvers cannot be supervised. It would be unfair to rely on driver to supervise such scenarios.
If evasive maneuver is required it would have to be done in fully autonomous way. For the time of the maneuver system would have to be responsible for driving.
>> As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.
That's a terrible idea. First it's a bandaid over an underlying problem. Second it's a legislative answer to a technical problem, which is IMHO never a good idea.
At the end of the video (0:32 to 0:33) you can see it quickly snap to right turn again. Why is the car attempting multiple right turns while the map indicates a straight line?
I wonder if it's an upgrade bug related to calibration?
There was a video by one Tesla user over the past week that talked about how their Tesla v10 FSD software would attempt to turn into driveways, repeatedly, when driving down a straight road.
The user did a recalibration of their cameras, rebooted the car, and the problem went away.
I don't own or plan to own a Tesla, but if i did, I'm sure I wouldn't use the self driving feature because it seems more inconvenient than simply driving yourself. It's not even a driving assistant, you become the car's assistant.
Edit: to be clear, I plan to buy an EV in the future, but I'll drive it myself. Most of my driving is in a 15Km radius, urban, and babysitting the self-drive seems like more trouble than it's worth.
Speaking as someone who believed all the hype about having fully autonomous cars by 2020, I think the truth is that it is orders of magnitude harder than we thought it was and that we are orders of magnitude less capable than we thought we were.
Maybe a good interview question for experienced developers should be "tell us about a time when you attempted something in software and failed at it." In my case it was being given a 10 or 20 KLOC C server that had memory leaks all over and segfaults all over the place (written by a friend) and told to make it prod ready.
That's a rough one. Mine was fixing some physics/flight aerodynamics software that had been generated by a Fortran-to-C compiler (e.g., http://www.netlib.org/f2c/). It was in inscrutable mess and couple-years-out-of-academic-physics-turned-SE me thought I'd be able to do it easily since I had so much Fortran and C experience. I failed quite hard at that.
On the one hand I applaud Tesla for being so open about what their system is thinking with their visualisations. That could be interpreted to show a deep belief in their system's capabilities.
On the other hand, it's always terrified me how jittery any version of AutoPilot's perception of the world is. Would you let your car be driven by someone with zero object permanence, 10/20 vision and only a vague idea of the existence or properties of any object other than lane markings and number plates?
I test drove a Model 3 yesterday and this was something that really jumped out at me. I didn't try any of the automatic driving features, but driving around Brooklyn watching the way the car was perceiving the world around it did not inspire confidence at all.
Tesla's over the top marketing and hype seems at once to have been a key ingredient in their success, but also so frustrating because their product is genuinely awesome. I've long been kind of a skeptic but I could not have been more impressed with the test drive I took. It had me ready to buy in about 90 seconds. I wish there were actually-competitive competitors with similar range and power that aren't weird looking SUVs, from brands that don't lean into pure hype.
> Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
The extremely poor performance of the visualizations are disturbing to me. It's also completely wasted space on the display that I wish were devoted to driving directions instead of telling me what I can already see outside of the car.
I think whether it's wasted space or not depends entirely on the reliability of the system. For a many-9's system which for all intents and purposes isn't going to fail dangerously, I agree, the visualisation is just peacocking. For a beta-quality system, knowing what the car is thinking is an important driver feedback which gives additional warning before it does something really stupid.
> I applaud Tesla for being so open about what their system is thinking with their visualisations
How do you know that what is on the screen matches what the system is 'thinking'? What reason or obligation would Tesla have to engineer accurate visualizations for the real thing and show them to you (remember the definition of accuracy: correct, complete, consistent)? Would Tesla show its customers something that would make them uncomfortable or show Tesla in a bad light, or even question the excitement Tesla hopes to generate?
I think it's likely that the display is marketing, not engineering.
Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
And orange cones - the orange cone detection team at Tesla is outstanding. Garbage cans are also pretty accurate. Not sure why they detect garbage cans but not parked cars, but that's just me.
Computer-vision-based self-driving is a mistake. Self-driving cars should drive on their own roads with no non-self-driving vehicles. Other vehicles, traffic lights, road hazards, route changes, etc should be signals that are directly sent to the vehicles rather than the vehicles relying on seeing and identifying cars and traffic lights and road signs.
We know how to make computers navigate maps perfectly. We don't know how to make computers see perfectly. The other uncontrollable humans on the map just make it worse.
Yes this'll not work in existing cities. That's fine. That's the point. An inscrutable black box "ML model" that can be fooled by things that humans aren't fooled by should not be in a situation where it can harm humans. I as a pedestrian did not consent to being in danger of being run over by an algorithm that nobody understands and can only tweak. Build new cities that are self-driving first or even self-driving only, where signaling is set up as I wrote in the first paragraph so that a car reliably knows where to drive and how to not drive over people. Take the opportunity to fix the other problems that old cities have like thin roads and not enough walking paths.
Just to point to another possibility, since the drivers right hand is not in shot, even though his left hand is carefully poised over the steering wheel for the whole shot, it also looks like he pulled hard on the steering wheel with his right hand.
That might explain the car trying to re-route on the display, although I'm not even sure it does. To me it looks like the black line disappears into compression artefacts at just the right time.
Mostly I think we don't have enough evidence either way and speculating just expresses our desire for the technology to work or not.
Gotta give you that. He still might have moved the steering wheel with his thigh though ;-)
Crazy how it handled a shitty situation like around 6:55 in https://streamable.com/grihhc but failed there. Guess that shows what most of the ADAS engineers already know, the really hard part are those last few percentage points to make it actually reliable.
Google Maps will update (prompt first) a route in progress depending on traffic. If you don't notice right away it can be very surprising! At least it's not actually driving the car.
Tesla's system is real time, which means that it makes routing decisions on every frame, and the decisions are independent from previous decisions. It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems. For some reason, it thinks that it must turn right.
Situations like these are sent to Tesla to be analyzed, the corresponding data is added to the training data, and the error is corrected in the next versions. This is how the system improves.
After this situation is fixed, there will be an another edge case that the HN crowd panics over.
> It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems.
This is incorrect. Tesla also relies on maps for traffic signs, intersections, stop signs etc. They just don't have additional details like the others do.
> After this situation is fixed, there will be an another edge case that the HN crowd panics over.
Is anything Tesla FSD can't handle an "edge case" now? It's literally an everyday driving scenario.
You're probably right, but from your tone you seem to be implying that this is in any way an acceptable way to develop safety critical software, which is a bit baffling.
It is safe, because there's a driver who is ready to correct any mistakes. There isn't a single case where FSD Beta actually hits something or causes an accident. So, based on actual data, the current testing procedure seems to be safe.
It isn't possible to learn edge cases without a lot of training data, so I don't see any other way.
This person is not a Tesla employee being trained and paid to test this extremely dangerous piece of software that has already killed at least 11 people. This is obviously unacceptable, and no other self-driving company has taken the insane step of letting beta testers try out their barely functional software.
FSD Beta has never killed anyone. Maybe you're confusing it with Autopilot, which is different software, but also has saved much more people than killed.
I'm not saying that safety couldn't be improved, for example by disengaging more easily in situations where it isn't confident. One heuristic would be when it changes route suddenly, like in this scenario.
"In the 1st quarter, we registered one accident for every 4.19 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot but with our active safety features, we registered one accident for every 2.05 million miles driven. For those driving without Autopilot and without our active safety features, we registered one accident for every 978 thousand miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles."
It is an inherently biased statistics, where drivers will let the car drive on parts where they are confident it can do its job, and will take over for the rare, more complex situations.
Also, the NHTSA dataset will contain old as hell cars, comparing it to a fresh out of the factory one will by itself skew the data.
Keep watching the HUD pane - before the driver takes over, it corrects and decides to go straight. In fact, it's turning the wheel to the left before the driver stops the correction.
I think this is a navigation issue. This is exactly what I would have done if I had a passenger yell "WAIT TURN RIGHT TURN RIGHT oh nevermind GO STRAIGHT"
What do you mean by "nagivation issue"? I don't see the navigation system changing momentarily and then back (that would be analogous to your example). If in your phrasing the navigation system includes object recognition then if it istructs the car to suddenly steer right without any real reason, how could we trust that it stops before hitting pedestrians? Even if in this situation it would've corrected itself, I wouldn't say that this is a minor issue.
Right, nothing should ever be built unless it can be 100% correct immediately.
I’m stunned that it hasn’t, so far as we know, actually hit anything yet. I’m not sure how, but clearly they’re doing something right between choosing drivers and writing software.
There were several high publicity collisions with a Tesla colliding into stationary objects -- a highway barricade [1], a blue truck [2], a parked police car[3], two instances of a parked ambulance[4]...
Teslas driving themselves hit objects all the time.
So, devil's advocate. We know the Autopilot ignores stationary objects, so a lane with a parked vehicle (emergency or otherwise) is therefore a reasonable place to travel at the set speed on the cruise control, etc.
But, I believe this discussion is about the new FSD software which is supposed to be more capable. Have we had reports about the new one doing the old tricks?
"autopilot" versus "full self driving* capable" versus "full self driving" versus "autonomous mode" seems like marketing hype instead of actual improvements. After all, "autopilot" was supposed to drive itself, so what's the new one do differently?
The dotted black line coming from the front of the car (which I am assuming is the intended route) quickly snaps from straight ahead, to a hair pin right, to a normal right turn.
Ignoring the fact that the right turn happened to be into a pedestrian crossing with people on it - what was the car even trying to do? The sat-nav shows it should have just continued forwards.
I am astounded that software capable of these outputs is allowed on the roads. When could crossing a junction then taking a hairpin right back across it, ever be the correct thing to do?