Free user votes replace paid raters
$100M/year Nielsen of LLMs
Arena’s core advantage is not better labeler management, it is a different supply engine entirely. Instead of recruiting and paying workers for one task at a time, Arena turns model curiosity into a continuous stream of side by side votes from millions of users who come to try new models. That gives labs live preference data on real prompts, faster than managed annotation workflows and without carrying contractor payout costs inside every evaluation job.
-
Scale and Invisible built human feedback systems around paid operations. Scale grew from classic data labeling into LLM work, while Invisible routes model outputs through trained raters and pays hourly labor. Mercor pushed further upskill, matching labs with doctors, lawyers, and PhDs for $50 to $100 per hour evaluation work.
-
Arena’s product is closer to a consumer ranking site than a staffing platform. Users type their own prompts, compare two anonymous answers, and click the better one. Because prompts are live and subjective, Arena captures taste, preference, and real world usefulness that static benchmarks often miss and that labs cannot fully precompute.
-
This changes the unit economics. Managed vendors often report topline figures that include what gets paid out to contractors, while Arena’s free user loop lets more revenue fall through as software and analytics. That helps explain why Arena reached an estimated $100M annualized revenue only months after launching paid evaluations.
The next step is a split market. Paid expert networks will keep winning where labs need audited domain judgment, safety review, or regulated workflows, while Arena expands where broad user preference is the product. As models spread into coding, images, and agents, the most valuable evaluation companies will be the ones that own the clearest source of human ground truth for each use case.