Five agencies, around 70 agents enrolled, five sessions over Zoom every two weeks. Between sessions, participants used EVA in their real client work. A survey before the first session, a short pulse survey after each one, and product analytics throughout.
I used AI to help draft the pilot materials: the discussion guides, the pulse surveys, the KPIs, and later the readiness scorecard. It gave me solid first drafts fast, and I shaped each one around what that session actually needed to prove.
Every session was recorded and transcribed in Marvin. For the ten-week pilot and the first open-pilot round, I used Copilot to turn those transcripts into structured summaries in Confluence: friction points, success moments, and verbatim quotes, tagged by feature. From the round after that, I switched to Claude for the same job. I reviewed every summary before it went into the findings. A synthesis tool can be confidently wrong in the same way EVA can, and that is the exact risk this whole project was about.
One constraint shaped everything: no code changes during the pilot. The version deployed on day one was the version used in week ten. Only the quality of EVA’s answers could be tuned. That sounds limiting, and it was, but it also meant every finding was about the same product rather than a moving target.
I told every participant the same thing up front: we are not testing you, we are testing EVA, and “I never used it” is a useful answer. That one line got me far more honest data than any incentive would have.
