OpenAI contractors are reading real ChatGPT chats, exposing a hidden privacy risk

The material viewed by 404 Media does not specify which OpenAI model the human reviewers are training, and whether it is a currently available model or one intended for future release. The material 404 Media has seen only references a codename: “Project Lily.”

Next, the reviewers rate each response with a number, with one being the worst — “unacceptable, unusable” — and seven being the best — “would be hard to meaningfully improve.” The instruction guide indicates a response with useful content can still score low if it is, for example, too long or cluttered. Another document marked “Confidential & Proprietary” states the model should “generally match the user’s tone, but slightly less intensely.”

“It should remain natural, restrained, and professional without implying it is human or experiencing emotions,” the document continues. “Flag sycophancy, forced style mimicry, engagement-bait endings, amplification of frustration, or patronizing assumptions when they make the response less trustworthy or natural.” Instead, responses should be, for example, “helpful,” “honest & truthful,” “empowering,” and “smart, but humble.”

Finally, reviewers provide their rationale for their numbered score. Examples in the instruction guide show these can range from a full paragraph to a couple of sentences.

An FAQ section for reviewers states that OpenAI does not expect them to fact-check the responses using external searches. One document asserts that “other project teams handle content verification,” suggesting human reviewers are also working on tasks like fact-checking. However, the company asks reviewers to flag any “factual or correctness issues” they detect and to penalize missing sources for “high-stakes” topics such as medical, legal, and financial responses.

Pay no attention to that man behind the curtain

The person working on the prompts that 404 Media communicated with resides in North America and said they are paid over $50 an hour. They found the job through the recruitment firm Crossing Hurdles, a company that “connects skilled professionals with AI training, evaluation, research, and contributor opportunities across the global AI economy,” according to its website. Its site states, “Human intelligence powers AI progress.” Multiple people on Reddit have reported receiving unsolicited recruitment emails from Crossing Hurdles, with some trying to determine if the company is a scam.

As of writing, the company’s LinkedIn page advertised multiple AI-related jobs, including an AI data reviewer, data annotator, and “chatbot evaluator.” The listing for that job does not mention OpenAI or ChatGPT, but the role responsibilities include “assessing AI responses for personalization, grounding, integration, and helpfulness,” and “comparing model responses side-by-side and evaluating their overall quality.” Its available projects also include contractors recording themselves performing household tasks, a data gathering exercise crucial for developing AI-powered robotics.

Crossing Hurdles, in turn, refers people to Mercor, an AI-training company. This is the company that ultimately pays contractors working on ChatGPT prompts, the worker said. Meta ended its relationship with Mercor in April after the company was involved in a massive data breach.

Reading the prompts can sometimes be “kind of amusing,” the worker mentioned. But overall, the work is “very rote.” They also noted that the job feels “all over the place.” The guidelines frequently change and can seem self-contradictory.

Human reviewers have been an important yet often concealed part of social media content moderation and the enhancement of some artificial intelligence models like those that identify objects in camera feeds. A TIME investigation discovered that OpenAI hired Kenyan workers to data label text pieces to make its platform less toxic. 404 Media’s reporting shows the world’s leading large language model (LLM) companies are also hiring human reviewers to read real users’ conversations to improve their models.

Human reviews of LLM conversations are not exclusive to OpenAI. A disclaimer on Google’s Gemini, for example, states, “Humans review some saved chats to improve Google AI.”

Anthropic told 404 Media that it does employ human review to enhance its models, including improving Claude’s future responses. This applies to users who have enabled the “Help improve our AI models” setting in their privacy settings. Anthropic said it also de-identifies conversations before human review by removing account identifiers like email addresses.

Michal Luria, a senior research fellow at the Center for Democracy & Technology, told 404 Media: “Human review of conversations with chatbots can be essential to safety, especially as companies work to strike the right balance on complex chatbot behaviors. That said, it’s important to bear in mind that current chatbot interfaces automatically create a false sense of intimacy and privacy in what feel like one-on-one interactions, when in fact there may be human reviewers on the other end. This is quite distinct from content moderation on social media, where publishing content already carries expectations of platform moderation and public exposure.”

Sarah T. Roberts, a professor at UCLA and author of Behind the Screen: Content Moderation in the Shadows of Social Media, likened the revelation that OpenAI is using human reviewers to the Wizard of Oz, “where the protagonists discover that the magical kingdom is really a man behind a curtain pulling levers.”

“You don’t have to dig very deep beneath the surface — beneath the mirror — to find that not only are these things built in the image, but usually a pretty bad facsimile thereof, of what human abilities can do. But they require constant, constant intervention from humans," she said.