01
A useful conversation still has to work as a journey
I built an AI-based questionnaire from the ground up. The challenge was larger than making the responses sound natural. It had to gather useful information, help someone understand their options, and support a next step they were actually willing to take.
A conversation introduces variability that a fixed form does not. People answer indirectly, skip questions, ask for more information, or decline to share contact details. I needed to understand those behaviors across versions without reducing every session to a simple success or failure.
The supplied Hearo screens show a related conversational questionnaire direction. I include them to make the interface explorations visible; they are not identified as the v2 or v5 screens used in the comparison below.
02
Build the measurement alongside the experience
The analysis work groups messages into sessions and reconstructs the order of the conversation. Separate workflows examine the journey, question dynamics, phone requests, segments, word patterns, and time spent. The dashboard brings those perspectives together.
That separation matters. A drop at a phone request is different from a person who never reaches it. A longer successful conversation is different from a long conversation that goes nowhere. I wanted the team to be able to investigate those differences before changing the prompt.
The code also handles test-lead filtering, version data, and links between conversation and performance sources. Those are the less glamorous parts of an AI product, but they determine whether the feedback is useful.
- 01ConversationVersioned messages and responses
- 02AnalysisSession paths, question dynamics, timing
- 03DecisionCompare behavior with business outcomes
03
One metric improved while the bigger picture got worse
The version comparison makes the tradeoff visible. In v2, 216 of 481 people asked for a phone number gave one, about 44.9%. In v5, that became 149 of 232, about 64.2%. On that measure, the later experience looked stronger.
But overall lead conversion was still below the corresponding baseline: about 16.9% lower for v2 and 43.6% lower for v5. The willingness of the smaller group who reached a phone request did not mean the full journey was performing better.
Those are separate test cohorts, so this is not a controlled head-to-head comparison of v2 and v5. It is a practical example of why I read an intermediate metric alongside the full funnel.
| Measure | v2 | v5 |
|---|---|---|
| Gave phone / asked for phone | 216 / 481 | 149 / 232 |
| Phone-sharing rate | 44.9% | 64.2% |
| Overall lead conversion | 1.42% | 0.92% |
| Change vs. respective baseline | −16.9% | −43.6% |
04
Treat hesitation as information
The sixth version introduced multiple paths: phone, email, scheduling, reminders, and quote requests. The flow documentation distinguishes people ready to act from people who need more information or are not ready yet.
That change also changed the measurement problem. Counting only phone leads would miss other responses. Counting a phone submission and its downstream appointment as two conversions would overstate progress. The workbook explicitly treats downstream scheduling as part of the same conversion path.
The supplied v6 comparison contains inconsistent session and entry figures, including more entries than sessions. I do not use that row to claim an overall uplift. The next step is to reconcile the cohorts and definitions before making a version-level performance claim.
- 01ReadyPhone or scheduling
- 02Tell me moreInformation, quote, or reminder
- 03Not readyA choice about the next step
05
Test behavior, including refusal
The QA workflow includes simulated users with different communication styles and comfort levels. One is quick to act. Another asks about email. Another declines to share contact information.
That gives the team repeatable scenarios for checking flow, tone, and outcome. It is useful for catching regressions, especially when a prompt changes. Simulated conversations still need to be checked against real behavior; they cannot establish usability or conversion performance by themselves.
06
The outcome, and the work that remains
The project produced an AI questionnaire, a versioned analysis workflow, and a way to inspect the customer journey in more detail than a single completion rate. The documented experiments also exposed shortcomings, including weaker overall conversion in several versions.
For me, the useful result is an accountable product loop: change the experience, inspect the behavior, check the business outcome, and decide what to change next. I would tighten event definitions earlier and add an explicit contact-refusal check to every QA run.