The general-purpose agent trap
Our first experiment was to hand the full production of a CRO audit to a single agent: UX analysis of heatmaps, interpretation of behavioral data, building an A/B test plan, writing the client recommendations. A broad scope, with apparent efficiency.
The result quickly raised a red flag. The audit lacked depth. The insights could have been produced without analyzing the site's data. The test plan looked like a compilation of generic best practices rather than a contextualized analysis.
Our first reflex was to refine the prompt: more context, more precise instructions, a more directive structure. Results improved slightly, but the problem persisted.
The diagnosis: we were asking a single agent to do what actually amounts to four distinct skills. Analyzing heatmaps to spot UX friction requires a specific eye. Interpreting GA4 data calls for another kind of expertise. Prioritizing tests by impact draws on a different strategic logic. Synthesizing it all into client recommendations requires yet another skill set. A CRO expert naturally shifts stance at each step. The single agent was trying to handle everything with the same approach.
It could do everything, but nothing at the level of expertise each task demands. The problem was the architecture, not the instructions.
Our answer: five specialized agents and an orchestrator
We completely rethought how the work was organized. Instead of a general-purpose agent, we built a pipeline of five agents with strict scopes, each calibrated for a specific type of reasoning:
Analysis of heatmaps, scrollmaps and screenshots to identify user friction points and underperforming areas in the journey.
Dedicated processing of analytics data exports (GA4, AT Internet) to extract the five most significant actionable insights, separating meaningful signals from noise.
Building the A/B test plan by cross-referencing the outputs of agents 01 and 02, with an effort/impact score for each proposed test.
Writing the executive summary and the final recommendation sheets, calibrated to the sponsor's reading level.
Keeps deliverables flowing smoothly between each step and checks overall consistency. Triggers the agents in the right order, manages dependencies, flags anomalies.
This organization may look more complex. It is markedly more effective. Each agent has a narrow scope, which makes it easier to control, improve and maintain without destabilizing the whole pipeline.
Co-building with business experts
The most demanding part of this project was not technical. It was co-building with Niji's CRO experts.
We ran workshops with our business teams to calibrate each agent. They were the ones who challenged the hypotheses produced, explained why certain friction points identified were not priorities in a real-world context, or why certain wording lacked operational precision.
Three principles guided this build phase:
- Testing on real engagements, not textbook cases. An agent that works on a clean file tells you nothing about its ability to handle incomplete data, non-standard formats or apparent contradictions in the numbers.
- Short iteration cycles. Four weeks per cycle, with systematic field feedback after each batch of engagements. Adjustments add up fast when they are grounded in real cases.
- Constant human validation. Every deliverable is still reviewed by an expert. This review makes it possible to spot gradual drift and adjust the system before it takes hold.
Measured results
The 25 to 30 hours saved per engagement relate to operational tasks: structuring data, formatting reports, rewording recommendations. Strategic analysis is not automated: it remains the core of human added value. The time freed up lets experts focus on it fully.
Two additional results emerged over the engagements. First, greater consistency of deliverables from one expert to another, since the structure is carried by the pipeline rather than reinvented each time. Second, robustness to imperfect data: when a step cannot be automated, an expert manually fills in what is missing without blocking the rest of the process.
A consistent finding: across every scope where we replicated this approach (SEO/SEA, lead qualification, CRM audit), the best results always came from specialized agents coordinated by an orchestrator, never from a single agent expected to handle everything.
What it means for your organization
This approach rests on a simple principle: whenever a process involves several distinct types of reasoning, one agent per type outperforms a single agent.
The starting point for identifying your opportunities: map the steps of your processes and the skills they involve. Wherever you see your experts shift stance from one step to the next, you have probably found a boundary between two agents.
Implementation requires business calibration work. Each agent must be built on your specific context, with your tools and data formats. That work looks like what we have described: workshops with your experts, tests on your real cases, continuous iteration, constant human validation. The architecture is transferable; the calibration is specific to each organization.
