PostHog Highlights LLM Analytics For Tracing AI Agent Runs.

PostHog shared a case study on using its LLM Analytics tooling to better understand how AI agents behave in production. In the example, a standup-writing agent appeared to work, but tracing exposed how it was making decisions, which tools it called, and where inefficiencies were hiding.

Key details from the post include:

• Full trace visibility for each agent run, including tool call order, arguments passed, returned data, generated output, and token usage by step.
• Detection of redundant behaviour, such as repeated tool calls fetching the same pull request details multiple times.
• Prompt version tracking through PostHog prompt management, allowing prompt changes without redeploying and making it easier to link odd outputs back to a specific prompt version.
• Automatic AI-generated trace summaries to help users review runs without inspecting every span manually.
• A linked build story from Andy Maguire showing how tracing improved debugging and iteration for a real workflow.

For small business owners and builders, this matters because AI automations can appear functional while still wasting tokens, repeating work, or producing inconsistent results. Better tracing and prompt visibility can make agent-based workflows easier to debug, improve, and trust.