Experimentation's Second Act

The biggest software companies have experimentation at their core. Google, Meta, Amazon, and Uber run millions of experiments each year. Nearly every change is tested on a subset of users before it ships everywhere, and the results help teams decide what to build next. The constant stream of real-world feedback is one reason they have been dominant for decades.

But almost nobody else does it well. Experimentation is hard: you need engineering capacity to build variants, infrastructure to deploy them, data scientists to design and read the tests, and enough traffic to reach significance. Few companies have all four. Optimizely's median customer runs only 34 tests a year, and over 90% are simple funnel metrics like clicks and checkout. The first generation of standalone experimentation platforms has been absorbed: Datadog bought Eppo, OpenAI bought Statsig.

Agents are about to bring experimentation to everyone else. As we wrote in When Software Engineering is Cheap, Product Judgment is Priceless, near-unlimited engineering capacity makes deciding what to ship the whole game. Experiments give software teams the data to make those calls. Every AI-native software company will need a strong experimentation foundation, and we’ll need a new generation of platforms to build it.

Why Agents Break Today's Experimentation Stack

Today's experimentation and analytics platforms assume a few dozen tests, human operators, and human users. Agents break all three assumptions.

1. Scale and complexity: In the time an engineer builds one version of a feature, an agent can build five, ship them, learn from feedback, and ship five more. AI features are also harder to evaluate: instead of comparing clicks on two versions of buttons, you might need to compare two versions of an agent workflow that call different tools and produce subtly different behaviors. Companies will have thousands of features to test, and will need better ways to judge them than simple conversion rates.
2. Agent users: Agents will become the largest users of software. It’s already true for developer tools: agents create around 75% of databases on Neon and Supabase. Consumer applications are next, kickstarted by apps like Muse and Instinct. Agents don’t behave like people: humans might browse aimlessly on a website while an agent hones straight in on a goal; humans prefer short, concise instructions, while an agent likes pages of full documentation. The meaning of key metrics changes: time on page might be good news from humans, but bad from agents. And agent behaviors change fast: human usage drifts over months, but agents can change dramatically with each new model or harness release.
3. Agent experimenters: It's hard for a person to design, run, monitor and evaluate one experiment well, let alone hundreds in parallel. Agents don't have that limit. They can build the variants, run the tests and read the results, while humans decide what's worth testing, define what better means, and bring in outside context that doesn’t show up in telemetry.

What Does an Agent-Native Experimentation Platform Look Like?

Today’s observability and product analytics platforms have added MCPs and in-app AI assistants, but the underlying products haven't changed: they’re built for humans to ask questions about human users, not for agents to run experiments on other agents. We think the next generation will be built around four things:

1. Thousands of experiments run by agents: Agents will be the primary operators, running thousands of experiments while humans set priorities and constraints. That means APIs before dashboards, many variants instead of a single A/B split, and humans reviewing summaries and exceptions instead of individual results.
2. Complex interactions: Today's experiments mostly measure clicks and conversions. As products shift to personalized experiences and agents doing long-running tasks, evaluating a change means reading agent traces, inspecting tool calls, and using AI judges to decide whether a feature is doing its job well.
3. Agents and humans as first-class users: The platform has to be built natively for two classes of users: agents and humans. Agents change with every model release, so it has to catch behavior shifts and regressions in real time: more like anomaly detection than weekly reports. Because today’s platforms were designed for humans, analytics for agent users is a strong wedge. We think this will matter even more than AEO: AEO helps you get discovered, but a great agent experience is ultimately what drives conversion and referrals.
4. Simulation: Coding agents fix the engineering bottleneck, but not the traffic one. Most companies don’t have the user base to run hundreds of concurrent experiments. Even those that do will want to screen ideas first: test 10 variants in simulation, then take the best 2 to real users. For agent users, simulation is close to exact, since you can run the actual agents against a staging build. Simulating humans based on telemetry seems possible too, at least for a first pass. And every live experiment calibrates for future simulations, so a platform that runs both gets better with every test.

Agent-written software will change constantly, and experimentation is the feedback loop that keeps humans and agents building in the right direction. Companies that run that loop well will see compounding benefits, just like Google does today.

If you're building agent-native experimentation and analytics platforms, I'd love to hear from you: at@theoryvc.com.