What We Got Wrong About Onboarding
A post-mortem on the SupportPages onboarding flow: why the generated question set was cut entirely, and what it taught us about asking anything at all.
The onboarding flow used to have two sets of questions. The first was static - five things we ask everyone. The second was generated: the analyzer read your codebase, worked out what it could not infer, and produced questions specific to your project. It was the most sophisticated thing in the product and it shipped in the first version, because it was obviously a good idea.
We deleted it on 5 June, and the reason it had to go is more embarrassing than "users did not like it".
The answers arrived with the questions deleted
The generated questions were stored keyed by opaque identifiers. q1, q2, q3. When those answers were passed downstream to the recommendation generator, they went as the identifier and the selected value - and the question text and option labels stayed behind.
So the generator received something with roughly the semantic content of q2: "option B".
It had no idea what had been asked. It could not tell whether q2 was about audience, deployment model, or which of three components mattered most, and it had no way to recover that. Every user answering those questions was donating careful thought to a field that was discarded at the next hop.
Nobody noticed for months, because the failure is invisible from both ends. The user sees a thoughtful question and answers it. The generator produces a reasonable article, because it has plenty of other context. Nothing errors, nothing looks wrong, and the only symptom is that a feature you believe is contributing is contributing nothing at all.
The questions were also asking the wrong thing
When we went to fix the plumbing, we read a sample of what had actually been generated. That is when the second problem showed up, and it turned out to be the one that killed the feature.
The generated questions skewed heavily toward ranking things the analyzer had already found. Which of these three components matters most. Which of these five flows is most important. Reasonable-sounding, and precisely backwards - if the analyzer found it, the analyzer can weigh it. The entire justification for asking a human is to capture what is not inferable from the code, and the generated set was systematically asking about the inferable part.
That reframed the decision. We were not looking at a broken feature worth repairing. We were looking at a feature whose output would have been low value even with the plumbing fixed, and the repair was a re-plumb of the whole path.
So it was cut, along with the "a few more questions" gate between analysis and section generation. Five static questions remain: who is reading, what kind of product it is, how people use it, what the docs should help with, and what tone fits.
What we'd do differently
The plumbing bug and the design bug have the same root, and it is worth naming because it generalises.
Both come from treating a question as a value rather than as a pair. A question is its text plus its answer; separating them destroys the answer, and we separated them at a serialisation boundary where it looked like a reasonable normalisation. If the identifier had never existed - if the payload had carried the question text alongside the response - the bug would have been impossible to write and the low quality of the questions would have been obvious on the first read of a payload.
The second thing I would change is when we looked. We shipped generated questions and then did not read the generated questions for months. There was no step in the process where a person examined what the system was producing, because it worked - the flow completed, articles appeared. A weekly habit of reading ten real payloads would have caught both problems inside a fortnight.
That habit is now the thing I would put in place before any generative feature ships. Not a metric. Somebody reading the actual output, regularly, with the specific question "is this contributing anything".
The general test
For any question your product asks, trace the answer to the place it is consumed and check what arrives.
If the thing consuming it cannot tell what was asked, you are not collecting data. You are collecting keystrokes.
About the Author
Simon Moxon has been building SaaS companies since 2009. He's scaled teams to 20+ people, hit £2m ARR, and learned most of the lessons the hard way. Now he's building SupportPages.io - help docs that write themselves when you ship code. He writes about what's changed in SaaS and what hasn't.