Vals AI embeds evals into developer workflow

This shift turns evals from a periodic model bakeoff into part of the software shipping loop. Once Vals builds tests from a team’s own merged pull requests and repo logic, it is no longer grading abstract model quality. It is checking whether a new model, prompt, or code change still solves the exact tasks that matter inside that codebase, on every commit, pull request, and live output.

• The workflow changes from researcher owned to engineer owned. Generic benchmarks are useful for picking a model once. Repo derived benchmarks create regression tests that live next to code, which makes eval spend recurring and ties the product to CI/CD budgets instead of research budgets.
  1 sacra 2 vals 5 braintrust
• The closest comparables are tools like LangSmith and Braintrust, which also push evals from offline experiments into production monitoring and pull request gates. The winning pattern is not best leaderboard scores. It is becoming the system that catches bad model changes before they ship.
  3 langchain 4 langchain 5 braintrust
• Using merged pull requests as tasks and hidden tests as pass fail logic makes coding evals concrete. The benchmark is built from work engineers already did, so teams do not need to invent synthetic tasks from scratch. That lowers setup friction and makes results easier to trust internally.
  1 sacra 2 vals

The next step is deeper insertion into the release path. As eval vendors connect repository history, pull requests, deployment gates, and production scoring into one loop, they start to look less like benchmarking tools and more like quality infrastructure for AI generated software and AI features across the stack.