Supabase Open-Sources Supabase Evals to Test AI Coding Agents.

Supabase is putting AI coding agents through a more realistic test. The company has released Supabase Evals, an open-source benchmark and testing framework designed to measure how well agents build, debug, and secure projects on its platform.

The framework runs agents such as Claude Code, Codex, and OpenCode against real Supabase tasks. Those include creating database schemas, debugging failed Edge Functions, and repairing broken Row Level Security policies. Supabase then scores the resulting work, giving developers and AI tool makers a standardized way to see where agent-generated projects succeed or break down.

Supabase said in its official announcement that the framework supports both its public benchmark and an internal regression suite monitored daily. The code is available through the open-source supabase/evals repository.

Supabase Evals is intended to evaluate more than whether an agent can produce plausible code. Each scenario runs against a real Supabase environment, allowing the framework to inspect what the agent actually does when working with databases, authentication, platform tooling, and security controls.

That distinction matters. An AI assistant can generate syntactically correct SQL while still creating the wrong schema, misconfiguring permissions, or overlooking a failed deployment. Database development also involves stateful systems and security policies, making simple text-based code comparisons a weak measure of whether an agent completed a job safely.

Supabase organized its tests around three dimensions:

• Product areas, including Database and Auth
• Cross-product topics, such as SDK usage and observability
• Builder stages, including creating an application and resolving failures

The scenarios are based on support tickets, bug reports, and GitHub issues rather than purely artificial programming exercises. Supabase selected a relatively small set intended to cover each dimension while reflecting problems developers encounter in production workflows.

The release follows Supabase’s broader push into agent-assisted development, including its OpenCode integration for agentic database management. Agents can already interact with Supabase through its command-line interface, MCP server, documentation, and agent skills. Evals gives the company a common measurement layer across those surfaces.

The framework separates tests into benchmark and regression suites, which serve different purposes.

Benchmark scenarios prioritize breadth. Supabase runs a compact but varied set of tasks using multiple harness configurations, including both more capable and smaller AI models, then publishes the results. This can help developers compare how different agent setups handle the Supabase development journey rather than relying on general coding benchmark scores.

Regression scenarios go deeper into known failure modes. These tests do not affect the public benchmark score, allowing Supabase to add cases from newly discovered bugs or evaluate changes without distorting the headline comparison. Monitoring them daily should also make it easier to determine whether updates to documentation, tools, or platform behavior fix an agent failure and whether that fix persists.

For small teams, the immediate value is visibility rather than a new application-building interface. A founder relying on an agent to configure authentication or RLS can use the published results to better understand the limits of a particular setup. AI coding tool developers can run the open-source framework themselves, investigate failed tasks, and test improvements before releasing them.

There are caveats. Supabase describes the published results as a snapshot because models, agent harnesses, and developer tools change rapidly. The live benchmark is therefore more useful than treating a single score from the launch announcement as a permanent ranking. Performance on the benchmark also does not remove the need to review migrations, access policies, and generated application code before production deployment.

What is Supabase Supabase Evals?

Supabase Evals is an open-source benchmark and testing framework for measuring how well AI coding agents complete real Supabase development tasks. It tests work such as building schemas, debugging Edge Functions, and fixing broken Row Level Security policies in actual Supabase environments.

Which AI coding agents does Supabase Evals test?

Supabase says the framework runs coding agents including Claude Code, Codex, and OpenCode. Its public benchmark uses multiple harness configurations and a mix of more capable and smaller models.

Is Supabase Evals available now, and what does it cost?

Supabase open-sourced the framework on July 31, 2026, and publishes results on its live Evals page. The announcement does not specify a paid plan or separate price for using the open-source repository.