PTN—15
Streaming Response
Render output token by token as it is generated, so the wait becomes part of the answer.
A language model produces text incrementally, and the interface should too. Streaming turns dead air into signal: within a second the person sees the answer forming, can judge its direction, and can stop it if it is wrong. Buffering the full response and dropping it in at once throws away the single biggest latency advantage this technology has — perceived time-to-first-value measured in hundreds of milliseconds, not tens of seconds.
Streaming is not just faster; it changes behavior. When people watch a response unfold they read along, catch misunderstandings early, and interrupt instead of waiting out a bad answer. That makes streaming a control surface, not a loading treatment. Every streaming view needs a stop affordance in reach, because the moment you show partial output you have invited judgment on it.
The hard part is rendering structure mid-flight. Markdown, code blocks, and tables arrive half-formed, and a naive renderer flashes broken syntax before the closing fence lands. Good implementations parse optimistically — style an unclosed code block as code immediately — and keep layout stable so text does not reflow under the reader’s eyes. A cursor or subtle shimmer at the write head tells people generation is live versus stalled.
Streaming became table stakes the week ChatGPT shipped, but its maturity curve runs toward selectivity. Token-by-token rendering flatters prose and punishes structure: a table assembling cell by cell, or a JSON payload twitching into shape, is worse than a short deliberate wait. Mature products stream where reading along has value — explanations, drafts, code — and buffer where the output is consumed as a whole, the way image models present finished frames rather than scan lines. The tradeoff is honest latency against legibility, and the tell of craft is a team that has decided, surface by surface, which one wins.
Anatomy
Which customers should we call about overdue invoices?
THINKING
A response streaming into a chat surface, with live structure and a stop control.
- 1Write head. A cursor marks where new tokens land, distinguishing live generation from a stall.
- 2Optimistic formatting. Markdown and code blocks style as they open, never flashing raw syntax.
- 3Stable layout. The container grows downward only; already-read text never reflows.
- 4Stop control. A stop button replaces send during generation — interruption is one click.
When to use
- Any conversational response longer than a sentence or two.
- Generation tasks where people benefit from reading along — drafts, explanations, code.
- When time-to-full-response exceeds roughly two seconds and a spinner would otherwise own the screen.
- When you want interruption to be a real option, not an emergency exit.
Design considerations
- 01Get first tokens on screen inside one second; front-load anything that delays them.
- 02Swap the send button for a stop button for the entire duration of generation.
- 03Parse markdown optimistically so unclosed code fences and lists render styled mid-stream.
- 04Pin scroll to the bottom while streaming, but release the pin the instant the user scrolls up.
- 05Keep already-rendered text pixel-stable — no reflowing, restyling, or re-wrapping of prior lines.
- 06On interruption, keep the partial output in the transcript rather than deleting it.
- 07Batch token updates to animation frames; per-token DOM writes wreck long responses.
Pitfalls
- ✕Auto-scroll that fights the reader — the person scrolls up to reread and the stream yanks them back down.
- ✕Raw markdown flashing before the renderer catches up, making the product feel like a debug console.
- ✕Streaming into a layout that reflows, so the paragraph someone is reading jumps mid-sentence.
- ✕No stop control, turning a wrong answer into a thirty-second hostage situation.
- ✕Streaming reasoning or tool chatter into the same visual channel as the answer, burying the result.
In the wild
- ChatGPT
- Streams every response with a stop-generating control, and made token-by-token rendering the default expectation for the category.
- Claude
- Streams prose and code with live markdown rendering, so code blocks appear styled while still being written.
- GitHub Copilot
- Ghost-text completions appear as you pause typing — streaming compressed to a single inline suggestion you accept with Tab.
- Perplexity
- Streams the synthesized answer after briefly showing source-gathering steps, so the wait is narrated rather than blank.