Where AI Speeds Up Discovery Work, and Where It Still Gets It Wrong

Discovery is the part of a project where speed and accuracy are usually in tension. Move fast and you risk building on assumptions nobody checked. Move carefully and stakeholders start asking why the team is still "just doing research" three weeks in. AI tools have changed that trade-off, but not evenly. Some parts of discovery genuinely get faster without losing much. Others get faster and lose something that matters, usually without anyone noticing until later in the project.

Where AI actually helps

A few discovery tasks are mostly mechanical, and that's exactly where AI earns its keep.

• Getting a study off the ground. Drafting screener questions, recruitment emails, interview guides, and test plans is repetitive work that AI can do a fast first pass on, leaving the researcher to edit rather than start from a blank page (NN/g).
• Desk research and landscape scanning. AI is a reasonable starting point for exploring a problem space or pulling together existing findings from inside or outside the organization, as long as someone goes back and checks the primary sources it points to.
• First-pass synthesis. Tools that cluster sticky notes or suggest codes for interview transcripts won't get it exactly right, but having AI take a first swing at organizing raw notes speeds up the part of analysis that's mostly sorting, not judgment.
• Prototyping around a hypothesis. Once a team has a working theory from real interviews, AI can generate variations to test quickly, which shortens the loop between "we think this" and "let's find out."

The common thread: AI is strongest on the front end of discovery, where the task is organizing information, and weakest the moment the task becomes understanding a person.

Where it still gets it wrong

The clearest failure mode right now is treating AI-generated people as a substitute for real ones. Nielsen Norman Group ran a direct comparison, testing AI-generated "synthetic users" against three studies it had already conducted with real participants, and found the synthetic responses too shallow to be useful for most research activities (NN/g).

A few specific problems keep showing up:

• Sycophancy. AI-generated personas tend to want to please. In one NN/g study, synthetic users reported successfully completing training they would realistically have abandoned, which is close to useless for anything involving prioritization.
• Missing nuance. Synthetic responses are consistently one-dimensional. Real interviews surface contradictions, hesitations, and half-formed opinions. Synthetic ones tend to hand back a tidy, confident answer regardless of the question.
• Outperforming real users on tasks they'd actually struggle with. Separate research on AI-generated participants in tree testing found the AI far outperformed most real people, which defeats the purpose of testing in the first place.
• False confidence in front of stakeholders. A polished AI-generated report can look like validated research even when nobody has talked to an actual customer. That's arguably the more dangerous failure, because it's invisible until a launch underperforms.

The line that matters

The useful distinction isn't "AI in research, yes or no." It's whether AI is standing in for real human data or helping someone process real human data faster.

• AI-moderated interviews, where an AI runs the conversation but the participant is a real, verified person, keep the data real while automating the collecting.
• Synthetic users, where the AI invents both the questions and the answers, replace the human entirely. This is the version that consistently falls apart under scrutiny.

Confusing these two is where teams get into trouble. A vendor pitching "AI research" doesn't always make clear which one they're selling.

What this means for discovery work

None of this is an argument against using AI in discovery. It's an argument for being specific about which part of the process it touches. AI makes us faster. It doesn't replace the judgment about which findings are trustworthy enough to build a roadmap on, and that judgment still depends on having talked to real people.

The practical split looks something like this: let AI handle the logistics, the first-pass organizing, and the exploratory groundwork. Keep the actual conversations, the interpretation of what people say and don't say, and the go or no-go calls in human hands. Discovery exists to catch wrong assumptions before they get expensive. A shortcut that quietly reintroduces those assumptions isn't actually a shortcut.