$100M/year Nielsen of LLMs
Jan-Erik Asplund
TL;DR: Where Scale and Mercor pay armies of contractors and PhDs to grade AI model outputs, Arena gets 5M people a month to do it for free, drawn by access to unreleased frontier models, and sells the resulting rankings and analytics back to the labs. Sacra estimates Arena hit $100M in annualized revenue in June 2026, up from $30M in 2025, valued at $1.7B as of January 2026 for a ~57x multiple. For more, check out our full report and dataset on Arena.
We've covered the post-training human data market extensively, from Scale AI (Jan 2023, Feb 2024) and Mercor to Invisible (May 2025) Surge, Handshake (Sep 2025, Apr 2026), and Prolific (May 2026), including an interview with Prolific’s COO Jemma White. To go deeper on the top player in crowdsourced LLM evals, we researched Arena ($250M raised, Felicis & UC Investments).
Key points via Sacra AI:
- The launch of ChatGPT (November 2022) set off a Cambrian explosion of instruction-tuned chat models from Microsoft’s Bing Chat and Google’s Bard to open-source models like Alpaca, Vicuna, Koala, Dolly, OpenAssistant, and ChatGLM, inspiring the launch of Chatbot Arena (April 2023, now Arena) where human users type a prompt, two anonymous models respond side by side, and the human user votes on a winner, feeding a leaderboard of top models based on chess-style Elo ratings. Common LLM benchmarks like MMLU & GSM8K could test fact-retrieval, math, and coding, but their answers leaked into the training data over time, allowing models to effectively memorize the answers, while Arena tests purely for subjective user preference via live user prompts that can’t be studied in advance.
- Trading free LLM usage in exchange for human evals, Arena hit 5M+ monthly active users across 150 countries by 2025, up from 800K a year earlier, generating $0 of revenue in its first two and a half years before incorporating (April 2025), raising a $100M seed at a $600M valuation (May 2025, a16z & UC Investments), and launching its first paid product, AI Evaluations (September 2025), to sell frontier labs analytics on how their models perform. Where Scale AI built a managed workforce paid per labeling task and Mercor charges labs $60-200/hour for PhDs, doctors & lawyers to grade model outputs, Arena's 5M evaluators work for free, with labs paying for evaluation campaigns metered on volume of battles, votes & prompts consumed, making Arena more like a Nielsen for AI where aggregated user preferences are sold back to labs.
- Expanding from testing pure text prompts into coding, image generation, and vibe coding prompts, and most recently to Agent Mode (June 2026), Sacra estimates Arena went from $0M to $100M in annualized revenue in June 2026, 8 months after launching their paid service, up from $30M in 2025, valued at $1.7B as of its January 2026 Series A (Felicis, UC Investments) for a ~57x multiple. Compare to Mercor at $2B in annualized revenue as of June 2026, valued at $10B as of its October 2025 Series C for a ~20x revenue multiple on ~$500M revenue, Scale AI at $1.5B in annualized revenue, up 97% YoY, with a $29B implied valuation at Meta’s June 2025 investment for a 19.3x multiple, and Handshake at $1.1B in annualized gross revenue in April 2026, up 349% YoY, valued at $3.5B.
For more, check out this other research from our platform:
- Arena (dataset)
- Prolific (dataset)
- Handshake (dataset)
- Handshake vs Mercor
- Invisible (dataset)
- Mercor (dataset)
- Scale AI (dataset)
- Joe Kim, CEO of Office Hours, on the end of crowdwork
- Jemma White, COO of Prolific, on why humans ensure AI safety
- LinkedIn for data labelers
- Invisible vs Mercor
- Scale at $760M ARR
- Scale: the $290M/year Mechanical Turk of machine learning
