new leaf Researchers at the Swiss university EPFL indicate that between 33% and 46% of workers deployed on Amazon Mechanical Turk appear to have “cheated” when performing a certain task they were assigned, as they used tools such as ChatGPT to do some of the work. If this practice were widespread, it could be a very serious problem.
Amazon’s Mechanical Turk has long been a refuge for frustrated developers who want work done by humans. In short, it’s an API that feeds tasks to humans, who perform them and then return the results. These are usually the kind of tasks you wish computers were better at. At Amazon, one example of such a task would be: “Draw bounding boxes to build high-quality datasets for computer vision models, where the task might be too vague for a purely mechanical solution and too vast for even a large team of human experts.”
Data scientists treat datasets differently according to their origin – if they were generated by people or a language large model (LLM). However, the problem here with Mechanical Turk is worse than it seems: AI is now available cheap enough that product managers who choose to use Mechanical Turk over an automated solution are counting on humans being better at something than bots. Poisoning this good amount of data can have serious repercussions.
“Distinguishing between English language and human-generated text is difficult for both machine learning models and humans,” the researchers said. So the researchers devised a methodology to find out whether the text-based content was generated by a human or a machine.
The test involved asking crowdsourcing workers to condense research abstracts from the New England Journal of Medicine into 100-word summaries. It is noteworthy that this is accurately The kind of task that AI generator technologies like ChatGPT are good at.