Measuring Human Judgment for AI Safety
Jemma White, COO of Prolific, on why humans ensure AI safety
The strategic shift is that human data is moving from who a person is on paper to how a person behaves under pressure. For AI teams, that means the valuable label is no longer just doctor, lawyer, or coder. It is whether someone stays calm with disturbing content, catches subtle model mistakes, or judges fairness across cultures. Prolific is building around that shift with filters, verification, and tools that surface these traits inside a self serve workflow.
-
This changes the workflow customers buy. Instead of ordering a generic annotation pool, they set criteria for traits, language, background, and task fit, launch studies through Prolific, send participants into survey or evaluation tools, then pay them automatically when the work is done. That makes personality and judgment usable as structured training and eval inputs.
-
The market has already climbed one rung up the quality ladder. Mechanical Turk style crowdwork was cheap and broad. Then companies like Mercor, Handshake, and Office Hours won demand for credentialed experts. The next rung is narrower and more human, where labs want taste, temperament, cultural context, and resilience, not just resumes.
-
This is also why neutral external panels matter even when labs build in house teams. Internal raters are useful for throughput, but outside participants give broader demographics, second opinions, and auditable evidence for safety and compliance work. That is especially valuable as AI evaluation expands beyond frontier labs into AI B2B companies and regulated enterprise use cases.
Over the next few years, the winners in human data will look less like labor brokers and more like systems for measuring human judgment. That pushes Prolific toward becoming an evaluation layer for AI products, where the durable advantage is not just access to people, but the ability to find the right people, capture the right behaviors, and turn those signals into repeatable model testing.