Supabase Open-Sources Supabase Evals for Real-World AI Agent Testing.

Supabase has released an open-source benchmark designed to answer a question that increasingly matters to developers: can an AI coding agent actually build and repair a production-style backend, rather than merely generate plausible code?

Supabase Evals tests agents including Claude Code, Codex, and OpenCode against real Supabase tasks. Instead of grading text responses in isolation, the benchmark places agents inside working Supabase environments, where they interact with the company’s actual Model Context Protocol server and command-line interface.

The company announced the release on LinkedIn, alongside an official blog post. It is available now, with its scenarios and scoring criteria open for inspection.

Supabase says AI agents have become a primary route through which people build on its platform. Those agents can use Supabase documentation, agent skills, the CLI, and its MCP server to inspect projects and perform development work. Until now, the company evaluated that experience largely through manual testing and intuition.

Supabase Evals replaces some of that guesswork with repeatable scenarios. Tasks include:

• Building a database schema
• Debugging a failed Edge Function
• Repairing a broken row-level security, or RLS, policy
• Using Supabase’s real MCP and CLI interfaces to complete the work

Crucially, the scenarios are based on support tickets, bug reports, and GitHub issues. That should make the benchmark more representative of the messy problems encountered in active projects, where an agent must diagnose context, choose an appropriate tool, and leave the environment in a working state.

The distinction matters. An agent can produce SQL that looks correct while introducing an unsafe access policy, or suggest an Edge Function fix without successfully deploying it. By running against a real environment, Supabase Evals can assess the resulting system rather than relying only on the agent’s explanation.

Scoring combines deterministic checks with an LLM-as-judge approach. Deterministic tests can verify concrete outcomes, such as whether a schema exists or a policy behaves as required. An AI judge can assess elements that are harder to reduce to a simple pass-or-fail assertion.

That hybrid model offers broader coverage, although LLM-based judging can introduce variability and model bias. Open sourcing the scenarios and criteria gives developers and model providers a way to examine how results were produced instead of treating a leaderboard score as an opaque claim.

Results can be browsed and filtered by model, product, development stage, or individual evaluation. This could help Supabase identify whether an agent struggles with a particular part of the platform, while agent makers can use the same evidence to improve tool calling, instructions, and error recovery.

For small teams, the immediate value is less about choosing a universal “best” coding agent and more about understanding reliability for specific jobs. A model that performs well at schema creation may still struggle with security policies or debugging deployed functions. Builders should therefore read results at the task level and continue reviewing database migrations, credentials, and authorization rules before using agent-generated changes in production.

Supabase Evals also reflects a broader shift in developer tooling. Documentation is increasingly consumed by machines as well as people, while CLIs and MCP servers are becoming interfaces through which agents act directly. Supabase’s recent work bringing Postgres into agent workflows, including its connector for Perplexity Computer, makes systematic testing of those interfaces more important.

What is Supabase Evals?

Supabase Evals is an open-source benchmark for measuring how well AI coding agents complete real Supabase development tasks. It runs agents against working Supabase environments and evaluates the changes they actually make.

Which AI coding agents does Supabase Evals test?

Supabase specifically names Claude Code, Codex, and OpenCode. Results can be filtered by model, product, development stage, or individual evaluation.

Is Supabase Evals available now?

Yes. Supabase says the benchmark is available now and has open-sourced its scenarios and scoring criteria. The announcement does not specify a paid plan or separate price for accessing it.