Behavioral Evaluation for AI Safety
Jemma White, COO of Prolific, on why humans ensure AI safety
This marks a shift in human data from teaching models facts to testing whether they behave like capable, trustworthy people. Early AI labeling was about getting armies of workers to tag images or rank answers. Then labs paid doctors, lawyers, and PhDs to push reasoning models forward. Now the harder job is judging tone, cultural fit, judgment, and safety in real situations, which favors platforms like Prolific that profile people on behavior and lived context, not just credentials.
-
The market has moved in clear steps. Scale built the large pool model for basic annotation. Mercor and Handshake grew by supplying expensive experts for math, law, science, and coding tasks. Prolific is positioning for the next layer, where model builders need humans who can evaluate personality, cultural nuance, and trust.
-
Prolific’s product is built for that shift. Customers can filter participants across thousands of attributes, including language, behavior, credentials, experience, and personality traits, then run studies through self serve tools or API workflows. That makes it useful for red teaming, safety checks, localization, and product testing, where the question is less is this fact correct, and more would a real user trust this response.
-
This does not eliminate experts. It changes where experts sit in the workflow. Domain specialists still matter for medicine, law, and frontier reasoning, but once models clear that bar, the bottleneck becomes ongoing human evaluation. The winning vendors are likely to be the ones that combine expert pools with broad, well profiled populations and fast repeat testing.
The next phase of AI data work looks more like continuous user research and safety auditing than one time labeling. As models spread into products used by consumers, employees, and regulated industries, demand should keep shifting toward repeated human checks on how systems sound, decide, and fail, which expands the role of platforms that can supply both expertise and real world human judgment at speed.