Rethink the Interface for Work Agents

I believe a promising next step for work agents is to redesign the interface through which people collaborate with them. Models are taking on longer, more complex tasks. How much of that capability people can use still depends on how easily they can understand progress, contribute judgment, and change direction.

Many tasks cannot be fully specified at the outset. You ask an agent to analyze sales, then realize after seeing the first chart that overseas revenue should be excluded. You review a few product proposals before discovering which tradeoffs matter to you. Intent becomes clearer as the work develops. These tasks need an accessible way to intervene: a quick “wait, use a different definition” should change what happens next while preserving the useful work already done.

There is a substantial commercial opportunity here. If people can spend less effort checking, revising, and resuming tasks, the same attention could support more work, including work of greater complexity. The interface that supports this collaboration could become a primary interface for everyday work. That possibility depends on actually reducing the effort of understanding and correcting an agent. An interface that requires more frequent supervision would defeat its purpose.

An interaction model is an important technical path toward that interface. The aim is to build decisions about interaction into the model itself. It needs an ongoing understanding of what the user is attending to, which intentions remain unsettled, and where the task stands. This informs when to respond, when to clarify, and when to pass a new requirement to the execution system. Speech, screens, text, and interactive controls can all contribute to this process.

This approach already has concrete implementations. Thinking Machines Lab’s research preview demonstrates continuous audio and video perception, proactive interjections, and tool use during conversation. Its real-time interaction model operates separately from a background model handling longer tasks. Demo and technical description GPT Live also separates full duplex voice interaction from background reasoning and tool execution, allowing conversation to continue while work proceeds. GPT Live documentation This division lets immediate responses and sustained reasoning operate on different timescales.

Gemini Live’s camera and screen sharing make it easier to point to the subject of a conversation, reducing the need to describe context again. Product introduction But progress across interaction capabilities is uneven. In the current Live API, 2.5 Flash Live Preview supports non-blocking tool calls, while 3.1 Flash Live Preview must still wait for tool results before responding. Tool-use documentation For work, accurate understanding, timely responses, and accepting new instructions during execution each need their own validation.

For real-time voice collaboration, I think full duplex should be a priority. A model needs to understand new input while producing output and adjust its behavior accordingly. A user’s “mm-hmm” may be an acknowledgment; “wait” may withdraw an instruction. These require different responses. Stopping playback whenever speech is detected cannot, by itself, make that distinction. Training and evaluation should also consider how these behaviors affect the task: responding too early may prevent a user from finishing a thought, while responding too late may miss the opportunity to correct course.

In the full duplex approach we are considering, incoming speech segments and generated text are interleaved in a continuously advancing sequence. The model uses listen and speak to decide its interaction behavior, while a separate Talker produces speech. New audio continues to enter the context during response generation. We distinguish eos, which marks the end of text generation, from eotk, which marks the Talker’s actual playback completion. The playback system supplies the latter as an external event.

This distinction matters for collaboration. The model may have generated a whole sentence when the user interrupts halfway through playback. The system needs to know what has been played and what remains unplayed. An eotk event alone only marks completion; recovering from an interruption still requires aligning playback progress with text. For a work agent, finishing text generation, finishing speech playback, and completing a background task are three separate facts. After speech stops, whether the task continues or adopts the revised instruction still needs to be handled separately.

Prioritizing full duplex does not require waiting for it to mature before testing agentic capabilities. I would pursue two tracks. One would test continuous understanding during output, recognizing when the user has finished speaking, acknowledgments, interruptions, and playback state. The other would connect an existing execution agent through text and event streams to test clarification, delegation, task revision, and result delivery. This should help reveal earlier whether a failure comes from interaction timing or from understanding the task and the user’s intent.

Separate validation must eventually give way to joint testing in real collaboration. Unfinished phrases, self-corrections, and overlapping speech expose problems that text tests miss. Once a user changes a requirement, a late result from the old task must not override it. Actions that have already happened must also be reported accurately. Beyond response latency, I care about how quickly revisions take effect, whether outdated results are used, and how many additional explanations a user needs to correct one misunderstanding.

I am optimistic about this direction because it could make more model capability usable in real work. But commercial opportunity and a lasting competitive advantage are separate questions. Natural speech or a two-model architecture is not enough to secure that advantage. It must come from reliability on actual tasks, integration into workflows, and a continuing reduction in the effort of collaboration. An interface that lets people confidently delegate more work while retaining an easy way to steer it deserves sustained investment in its own right.