Netlify Launches AXIS, Open Source Framework to Measure Agent Experience With APIs.

Netlify has launched AXIS (Agent Experience Index Score), an open source framework designed to measure how well APIs and services work with AI agents. The tool, released under the MIT license and available as an npm package, addresses a growing problem: as AI agents increasingly interact with developer tools and services, most platforms have no way to measure or improve that experience.

AXIS runs a defined set of scenarios against a live endpoint and produces structured scores across four dimensions: goal achievement, environment setup, service usability, and agent decision quality. The framework is not a pass/fail test. Instead, it provides a baseline and a direction for improvement, similar to how Lighthouse scores web performance.

The scoring system breaks down into weighted categories. Goal achievement carries the most weight at 40 percent, measuring whether the agent actually completed the task against a defined rubric evaluated by an LLM judge. Environment scores (20 percent) assess whether project structure, shell commands, and build tools trip agents up. Service scores (20 percent) evaluate API usability, including MCP tools and third-party integrations. Agent scores (20 percent) capture the quality of the agent's own planning and tool selection.

This structure allows teams to isolate specific pain points. A low environment score might indicate confusing project layouts or missing documentation. A low service score could point to unclear error messages or poor API design. The framework supports testing with over 20 coding agents, including Claude Code, Cursor, Gemini, and Copilot.

AXIS is designed to fit into existing development pipelines. Teams can run it locally during development, baseline scores over time, or gate CI builds on minimum thresholds. The tool outputs detailed reports in HTML and JSON, making it easy to track improvements or regressions across releases.

Installation is straightforward:

npm i -g @netlify/axis
axis init
axis run

The framework can test any combination of APIs, skills, MCP servers, or CLI tools. It runs in any environment and does not require cloud access or proprietary infrastructure.

Agent experience is not identical to user experience. Agents authenticate differently, interpret error messages literally, and decide whether a service is worth using based on factors that are hard for human developers to predict. AXIS makes those factors visible.

Matt Biilmann, CEO of Netlify, previously wrote that agent experience is moving upstream, becoming a core consideration in platform design. AXIS reflects that shift by giving teams a concrete way to measure and improve how agents interact with their infrastructure.

Netlify is developing AXIS in the open with founding contributors who are among the first services to run the benchmark and publish their results. The company published a detailed blog post explaining how it measures its own agent experience, and an O'Reilly Radar article explores the reasoning behind the framework.

The tool is available now at axis.run, with the full repository on GitHub. Teams can start scoring their APIs immediately and contribute to the framework's development.