What data labeling is — who writes the answers an AI learns from
Data labeling is the work of attaching answers to raw data so a model can learn from it — marking what object is in a photograph, transcribing what was said in a recording, or choosing which of two model replies is better. It matters because supervised learning has no route to correctness except through these labels, which means a model's accuracy ceiling is usually set by label quality rather than architecture. The shape of the work has changed: simple annotation is increasingly automated, and the labels people still supply have moved up into expert judgment — above all the preference rankings used to align large language models
The three lines
- Definition — Attaching answers to raw data. It is the only channel through which supervised learning knows what is correct
- Impact — A model's ceiling is usually set by label quality, not architecture. Wrong labels are learned faithfully
- Shift — Simple annotation is automating; human work has moved to expert judgment and preference ranking
Key questions
- What exactly is data labeling?
- **Attaching "this is the answer" to a piece of data.** The form depends on the data. For images it might be a **class label** ("cat"), a **bounding box** drawn around an object, or **segmentation** marking the boundary pixel by pixel. For audio it is **transcription**. For text it may be sentiment or named entities. And one more form has become the most valuable: **a person choosing which of two model-generated replies is better**. That preference data is what steers a large language model toward what people actually want. What they all share is that **the raw data alone gives a model no way to know what is correct.**
- Why do labels matter so much?
- **Because in supervised learning, labels are the only channel through which a model knows the answer.** A model does not treat a label as a suggestion — it **takes the label as ground truth** and pushes its own output toward it. So if the label is wrong, the model learns the wrong thing accurately. The conclusion practitioners reach most often is that **cleaning labels often yields more improvement than changing architectures**. The particular danger is **inconsistency**: if worker A labels a picture "dog" and worker B labels the same picture "wolf," the model cannot learn the boundary and will keep wobbling near it. Worse than random error is **systematic bias** — errors that all lean the same way, because the model learns that lean as a rule.
- AI has improved. Are people still needed?
- **The kind of work needed changed.** Attaching "cat" to a photograph is something models now do themselves, and that layer has largely automated. What remains for people is **judgment models cannot yet make** — assessing the accuracy of answers in medicine, law or code, and writing better ones. That shift rearranged the market. **Amazon Mechanical Turk**, opened in 2005 and the emblematic channel of the simple-task era, **closes on September 30, 2026**. Its demand went to firms recruiting specialists, like Scale AI, Mercor and Prolific. This page covered the move in "Amazon Mechanical Turk shuts down September 30, 2026." **People did not leave the loop — they were replaced by more expensive people.**
When we say a model "recognises cats," there is exactly one route by which it came to.
Someone wrote "cat" on tens of thousands of photographs.
Data labeling is that writing. And it usually sets the ceiling on how good the model gets.
1. What a label does
Model training splits into two broad kinds.
| Approach | Are there answers? | Examples |
|---|---|---|
| Supervised learning | yes (labels) | image classification, speech recognition, translation |
| Unsupervised learning | no | clustering, next-token pretraining |
In supervised learning, labels are the model's only channel to correctness. The model does not consult a label — it assumes the label is right and pushes its output toward it.
Which yields a blunt conclusion. Wrong labels are learned accurately.
2. Kinds of label
| Data | Label form | What it does |
|---|---|---|
| Image | classification | one name for the whole picture |
| Image | bounding box | draw a rectangle around an object |
| Image | segmentation | mark the boundary pixel by pixel |
| Audio | transcription | write down what was said |
| Text | entities, sentiment | mark names, places, tone |
| Model output | preference ranking | choose the better of two replies |
Reading down the table, the work gets more expensive and harder. Drawing a rectangle takes a few hours of training. Deciding which of two medical answers is more accurate requires knowing medicine.
That last row is the label that has appreciated most in the era of large language models. People compare model outputs and rank them, and that preference is what pushes a model toward what humans want.
3. How labels fail
Quality problems are not one thing. They differ in shape and in danger.
| Problem | How it arises | Effect on the model |
|---|---|---|
| Random error | slips, fatigue | washes out somewhat with volume |
| Inconsistency | annotators use different criteria | the boundary cannot be learned; the model wobbles there |
| Systematic bias | the guidelines themselves lean | most dangerous — the lean is learned as a rule |
| Ambiguous guidelines | no rule for edge cases | annotators diverge → becomes inconsistency |
The third row is the one that more data does not fix. Random errors average out as volume grows. Errors that all lean the same way become more firmly established the more of them there are.
This page covered a related consequence of training design in "What AI hallucination is: why models are trained not to say 'I don't know'."
4. How quality is defended
Because people do the work, quality control becomes a statistics problem. The widely used methods:
| Method | How |
|---|---|
| Consensus | give the same item to several annotators and take the majority answer |
| Gold standard | seed items with known answers to measure each annotator's accuracy |
| Inter-annotator agreement | use the rate at which two annotators agree to diagnose guideline clarity |
| Iterated guidelines | add a rule every time an ambiguous case appears |
| Staged review | first pass → second review → sample audit |
The third row is the useful one. If agreement is low, suspect the guidelines before the annotators. When several people diverge on the same item, it usually means nobody wrote down where the boundary goes.
5. The work changed
Through the mid-2020s the shape of this field shifted substantially.
| Before | Now | |
|---|---|---|
| Typical task | classify photos, draw boxes | expert evaluation and rewriting of answers |
| Qualification | generally none | domain knowledge (medicine, law, code) |
| Rate | cents per item | far higher |
| Automation | — | models now do the simple layer |
August 2026 supplied the emblem. Amazon announced that Mechanical Turk would close on September 30. Opened in 2005, it was the defining channel of the simple-task era, and Jeff Bezos called it "artificial artificial intelligence" — it looks automated, and inside, a person is doing it.
The demand did not vanish. Scale AI, Mercor and Prolific took it, along with the workers. Covered in "Amazon Mechanical Turk shuts down September 30, 2026."
6. Common questions
Q. Can models teach models? (synthetic data) Partly, and it is widely done. But the source model's errors and biases replicate, and how far synthetic data can substitute for human labels is unsettled. This page takes no side.
Q. Is training impossible without labels? Supervised training is. But pretraining a large language model runs without labels, predicting the next token in raw text. Labels become decisive afterwards — in the alignment and fine-tuning stages that shape a model to a purpose and to human preference. Covered in "What fine-tuning is."
Q. If I have lots of data, can label quality be sloppy? For random error, somewhat. For systematic bias, the opposite — more data entrenches it.
Q. What does labeling cost? No reliable public statistic was available, so no figure is given here. The direction is clear enough: automation pushed simple-task rates down, and expert-judgment rates up.
7. What could not be confirmed
- Quality-control methods — a summary of common practice, not a verification of any company's operations.
- "Cleaning labels beats changing architectures" — a practitioner heuristic; no quantitative study identified.
- Market size and worker counts — no reliable source found, so no figures given.
- Pay and conditions — international comparisons could not be confirmed.
- Synthetic data — how far it substitutes for human labels remains unresolved.
Sources
- CNBC — Amazon service Bezos once called 'artificial artificial intelligence' is shutting down
- Quartz — Amazon shutting down Mechanical Turk platform on Sept. 30, 2026
- TechCrunch — Amazon will stop accepting new customers for Mechanical Turk
- Google Cloud — What are AI agents? Definition, examples, types
- PYMNTS — Amazon Sunsets Crowd-Sourced Work Platform MTurk