Nobody experiences a p95 number. People experience staring at a spinner with no idea whether anything is happening. Those are different problems, and we keep trying to solve the second one with infra budget meant for the first.
Streaming tokens, showing the plan before executing it, narrating tool calls, and letting the user interrupt all make a slow system feel responsive without making it faster. Often that's the higher-leverage fix.
The infra side still matters — see the latency tax of multi-agent systems — and a lot of it traces back to how you spend the context budget. But start by asking whether the user needs it faster or just needs to not feel abandoned.