Mechanical Turk had a good run, but not surprised it's shutting down. I'm sure the platform was getting flooded with people doing task arbitrage and using lots of AI anyway.
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.
Giving out control of industrial machinery that interacts in the human environment without the physical interlocks (i.e. humanoid robots in a house) to random internet people seems like a problem.
I can imagine a carefully orchestrated plot to assassinate someone by having an embedded agent in the task delegation pool command the laundry bot to punch the target's head off their shoulders.
Giving out control of such things to LLMs is already complete madness, so once the first pleasure bot powered by Grok has dismembered a few thousand users, they'll get sophisticated safety mechanisms.
Though like as not you're still going to be right, after all, Stuxnet happened.
Why does a laundry folding bot have to have the physical strength to be able to kill somebody?
The principle of least privilege has been a hard learned lesson in cybersecurity. Why do we again need to first go through disasters to re-learn it in the physical world?
If a general-purpose humanoid robot has the strength to perform human actions at human speed, it will have that much torque in the motors. If it didn't, it can only move very slowly and/or jerkily. You need fast, high-torque feedback to stabilise the physical control system, especially if the robot is supposed to be able to lift and manipulate objects.
The idea here is Optimus-type robots that can do everything like a human and therefore don't need special infrastructure, as opposed to dedicated, immobile low-torque, low-velocity robots specifically for, say, laundry.
You can layer a safety system on top of the raw motor control but it's very complicated to define what is and isn't safe, especially when you consider what the robot is holding (and what it THINKS it is holding), where the robot is (or thinks it is) at the time, what is around it (what it thinks is about it) and so on.
Which isn't a robot-specific problem. Humans spend years learning how to subconsciously safely interact with their environment and even then fuck it up sometimes. What is robot-specific is being made of metal and having the "safe hand velocity within 10cm of a human head" function bypass be one bug or OTA update away.
giving out unrestricted control that is.
in the example of Waymo, a human can control some aspects of the car manually, but it can never override low level obstacle detection or say open the trunk/door when the car is moving.
This is what I meant by full stack AI companies. I don't think you could get humans into the loop fast enough if they didn't have some idea of the type of task involved. I don't want people to be asked to fold a tshirt one moment and do a difficult traffic merge the next.
There is training systems and validation of skills in mturk iirc: for tshirt folding, you'd be given fake setups to be able to get used to controlling the robot, if you can't do it, you won't ever get assignments to do it. For traffic overrides, you'd be tested on having correct knowledge, and once again given supervised tasks to show you can actually be trusted (and there would be safety systems, elevating tasks that can't be performed at your level to people who can, etc)
The worker needs to do a bunch to opt into any given work group, which makes the (lack of) payments extremely unreasonable on top of everything else
It doesn't really matter what you want though, only what CEOs want and that's low costs. I can see a combined shirt folding/traffic merging platform taking off.
but call center are still mostly single client even though it would be cheaper for any worker to be able to answer to any call. So clearly the expertise and context trade off is too big to be worthwhile
That's a remarkable idea. It could be heavily gamified, it could train models, and it might actually be mentally stimulating since you'd be facing different situations all the time.
Except, I'm a grown adult and I can't fold a t-shirt properly
There are already robotics models that can fold shirts and similar just fine. Progress in VLA models is good, I don't think this would form the foundation of a business. Humanoid robots are going to be another ChatGPT, it's going to seem to happen almost overnight because people aren't paying attention to the underlying research papers.
Most progress in data-driven robotics nowadays are done either in unicorn startups or corporate research labs - so you should follow the industry more than academia. The path to good robot performance isn't really in the models themselves - it's highly dependent on how much you can gather high-quality real-life data.
If their hero image video and the side by side at (5x) with a human at quote "1x" are "solved" I'm not impressed. I'm faster and more accurate and I am the worst folder in my house (kids included). The human looks like they are doing it slow mo to show children how to.
There are already a number of companies providing RLHF and SFT services for AI providers that does a lot of validation/prequalification of people that'd be well placed to take on tasks like that, but the big problem to solve would be latency if you don't have people contracted to carry out a task right now.
I used it before AI coding and it was getting rough. Lots of US interviews with proxy Chinese or Pakistani workers. Lots of bullshit “I’ve done that; I can do this” and instantly apparent that this was untrue. Just the outright lies… whew.
I haven’t touched it since AI coding.
It’s a bad contractor market now. IDK what I would do if I needed a contractor.
It is weird because the last time I've heard about MTurk was about developing countries being rather reliant on it for doing AI grunt work. If am not totally wrong this must mean that the data work has moved to other services.
I think the bigger issue is that a lot of the demand for labelling training data is now in highly-specialised fields (i.e. things like medical imaging), and mechanical turk's focus was on the generalist problems
To the best of my knowledge, that isn't true anymore, and nowadays you'd only get hired to do RLHF if you have particular skills beyond what can be achieved by just running other models against it.
I'm unsure. If it does, it will be work that is too expensive or inaccurate or regulatory for current AI methods. For example, you want a doctor to sign off on some AI output on a diagnosis.
However, I'm not sure a single platform will be how it emerges
A lot of what's been discussed in this thread is what we're tackling at Humwork (YC P26).
We're an MCP/API that connects AI agents to verified domain experts in real time (30s–3 min). Experts are vetted upfront by an AI interviewer that assesses and grades them, then get a mobile notification when a task matches their expertise and chat with the agent directly.
Soon we'll be verifying credentials for doctors, lawyers, CPAs etc on our platform for tasks where people are seeking credentialed experts to sign off and verify ai output.
We do our best to identify AI use and ban those experts - its not perfect just yet. Long term we are thinking of moving towards proctoring experts using their camera and screen capture. Hard to think of another reliable way.
You get matched with an expert in 30s to 3min, which then starts a back and forth with the AI agent and the expert which goes on for 10 to 60mins till the AI is happy.
This works well for many tasks, but for others a more async mechanism where the expert doesn't feel rushed might be better.
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.