Never ask a synthetic user if your design is any good

Not yet familiar with the concept of "synthetic users"? First read our blog article introducing the topic.

There are human traits AI simply cannot replicate. Real data gets you a long way: the more conversations and transcripts you feed in, the richer the source your synthetic can draw from (and the more authentic the picture you get back). However, a few vulnerabilities remain.

• People are unpredictable. They sometimes do things that don't fit their own logic. They drop off for reasons they can't even name themselves. A synthetic doesn't do that — it stays neatly within the character you've given it.
• People have flaws. They're tired, rushed, distracted, or simply don't read. Synthetic users never get tired, unless you ask them to.
• Synthetics tell you what you want to hear. They're naturally inclined toward coherent, positive answers. A synthetic will rarely tell you your assumption makes no sense.
• Synthetics don't feel priority. They can tell you that something is confusing, but not whether it's confusing enough to make someone drop off, call, or abandon the whole process.
• Synthetics only know what you know. You get an answer that's consistent within the logic you fed it. And because it comes back in different words, it sounds like evidence rather than an echo.

Despite that list, synthetics are strong at exploring. Think of it as a crash-test dummy: the reason a car has passed a hundred tests before anyone actually gets in. Useful if you want to stress-test ideas.

We deploy them early in the process, for exploration. In the design process, they help generate hypotheses, investigate exceptions, and test first sketches and ideas. The payoff is speed and breadth. Not certainty.

How you name the outcome determines what happens with it later. Call it a synthetic signal, not evidence. So not "users find this," but "this suggests something we should test." Otherwise, before you know it, an assumption ends up in a presentation that nobody dares to question anymore.

As soon as the decision carries enough weight that you need strong evidence. Think of establishing user needs, prioritizing problems, estimating adoption, understanding emotional or cultural context.

The standard for human-centered design (ISO 9241-210) is clear on this: evaluate designs with real users, because user feedback is a critical source of information and reduces the risk of missing real needs or practical problems.

Before deploying synthetics, ask two questions: how close to the truth does the outcome need to be, and what are the costs if you're wrong? The higher the cost of being wrong, the less you can rely on synthetic evidence alone.

We use a matrix with two axes for this:

• Required fidelity: how precisely does the response need to reflect real human cognition, emotion, or behavior?
• Impact of a wrong conclusion: what happens if the output is incorrect and you act on it anyway?

Low fidelity, low impact: go for it. You're exploring, not yet building anything. Brainstorming, probing, concept sketches. A synthetic suffices (for now).
Low fidelity, high impact: synthetic as preparation. Let the synthetic put options and risks on the table, but leave the decision to real data. You use it to better focus your research, not to replace it.
High fidelity, low impact: allowed, with a grain of salt. Being wrong here is cheap, so exploring is fine. Treat every answer as a hypothesis and write it down as such.
High fidelity, high impact: real people, without a doubt. Decision guidance, accessibility, vulnerable target groups, major decisions. Anything that's hard to undo.