AI 助手如何以上下文为界面重构跨应用体验

The future of design lies in what happens between products

Imagine you receive a message from a friend recommending a restaurant. You want to check where it is, see whether you are free that evening, and reply with a time that works for both of you. You search through your messages, open the link, check the location, switch to your calendar, return to the conversation, and finally write your reply.

Each step is simple on its own, but together, they create a fragmented experience. You have to carry the context from one product to another, remembering what you were doing and what you need to do next. This is a familiar interaction. We have designed products around individual tasks for years, leaving people to connect those tasks themselves. But when AI can interpret an intention and help carry it across products, the experience starts to change.

The question becomes how we design what happens between interfaces.

From applications to context

The app became the dominant model for digital experiences because it gave software a clear structure. Each product has its own interface, features, and way of managing information. People move between them to complete a task.

The problem is that people’s intentions rarely follow these boundaries. Planning a trip involves conversations, calendars, maps, bookings, and payments. Organizing a meeting requires information about people, availability, locations, and commitments that may live in several different systems.

Each product can work perfectly well on its own while the overall experience remains fragmented. AI makes it possible to approach these activities from another direction. Instead of asking people to identify the right application and navigate its features, an assistant can start with their intention and determine which information and capabilities are relevant.

Apple’s work on Siri and Apple Intelligence illustrates this direction. Through its App Intents framework, Apple allows developers to describe actions and content that can be made available to supported system experiences. This gives Siri and other supported features a structured way to discover what an application can do.

Imagine asking Siri to find a restaurant a friend recommended in an earlier conversation, check your availability for Friday, and help prepare a reservation. Depending on the available integrations and permissions, the assistant could connect information that would otherwise require several manual searches. The applications remain responsible for their data and functionality, but their capabilities can be accessed through an experience that begins somewhere else.

This changes the design problem, because the challenge is no longer limited to helping someone navigate an application. It also involves making the right information and capabilities available in the right context, at the moment they become useful.

For a practical explanation of how this works, Apple’s developer documentation on Apple Intelligence, Siri, and App Intents describes how app entities, schemas, and onscreen context help the system interpret and use app capabilities.

Context is a design material

Context has always been part of UX. It helps a system understand what a person is trying to do, what information is relevant, and what should happen next. It can include the current conversation, a person’s preferences, previous decisions, or the state of an ongoing task. It can also include constraints, such as whether a calendar is available or whether an action requires permission.

When someone asks for a restaurant recommendation, the request alone may not be enough. The assistant may need to understand the occasion, the people involved, the location, the available time, and any preferences that matter. Some of this information may already be available. Other details may need to be requested.

Designing for context means deciding which information matters at each stage and how it should influence the experience. It also means deciding what should be remembered, what should be shared between products, and what should remain private.

Memory becomes particularly important when a task spans time or products. An assistant might remember that a friend recommended a particular restaurant, that Friday was the preferred evening, or that the user usually avoids noisy places. But remembering information is not the same as knowing whether it is still relevant. The system should be able to surface the context it intends to use, especially when that context influences a decision. Showing someone that a restaurant was recommended by a friend six months ago gives them an opportunity to confirm or correct the information before the assistant acts on it. Memory should help maintain continuity without turning past behaviour into an unquestioned assumption.

Unfortunately, context is rarely complete. A person may say, “Find somewhere nearby for dinner,” without specifying a budget, cuisine, or exact time. The system has to decide whether it has enough information to proceed or whether it should ask a question.

This has implications for how products structure their information. In Apple’s App Intents framework, developers define entities, parameters, and actions so that supported system experiences can understand what an application contains and how its capabilities can be used. These definitions influence whether a system can identify the correct item, interpret a request, and perform an action reliably.

These decisions shape the interaction as much as the interface does. They determine whether the experience feels relevant, whether the user understands what is happening, and whether the system can act without making assumptions that matter.

Context therefore becomes a design material in its own right. It is part of the structure that makes an experience coherent, and it needs to be structured, interpreted, and actively designed rather than simply collected.

UX governance defines the boundaries

When an assistant connects information and actions across products, someone needs to determine how that information can be used and what the system is allowed to do with it.

Consider an assistant arranging a meeting. It finds an available time, retrieves the participants’ contact details, and prepares an invitation. Should it send the invitation automatically, or ask for confirmation? Which calendar should it treat as authoritative? What happens if availability changes before the invitation is sent?

These are questions of UX governance: the shared principles and rules that determine how an experience behaves across products and contexts. Governance establishes who can access information, which actions can be delegated, when approval is required, and how mistakes can be identified and corrected. Without these shared rules, an experience that crosses several products can become inconsistent, even if each application behaves as intended.

The NIST AI Risk Management Framework offers a useful foundation for thinking about these responsibilities. Its four functions, Govern, Map, Measure, and Manage, provide a way to identify risks, establish accountability, evaluate system behaviour, and manage issues over time.

The Model Context Protocol, or MCP, is another relevant development. It provides a standard way for AI applications to connect with external tools and data sources. This creates opportunities for systems to work together, while making questions about access, permissions, and responsibility important parts of the design.

For designers, these questions are part of the experience itself. If an assistant can take action across products, the boundaries of that action need to be understandable. People should know what the system can access, which tools it intends to use, and when their approval is required.

This also means making tool use visible. When an assistant searches a calendar, retrieves information from a message, or sends a request to another service, people should have a way to understand what happened and why. The aim is not to expose every technical operation, but to provide enough information for people to follow the process and remain confident in the result.

Governance cannot sit with one design team alone. It requires collaboration between design, product, engineering, security, legal, and the teams responsible for the systems involved. The NIST AI Risk Management Framework Playbook can help translate these responsibilities into practical activities throughout the development process.

Designing for uncertainty

Even well governed systems can misunderstand context. An assistant might find a restaurant recommendation from six months ago and interpret it as a current preference. It might identify a date in a message but misunderstand which event it refers to. The information is available, but its meaning depends on circumstances the system may not fully understand.

This also affects how information is presented. People need enough visibility to understand what the assistant has inferred, which details it is using, and what it plans to do. They should be able to correct an assumption without having to restart the entire task.

A useful design must account for this uncertainty instead of treating every inference as fact. One practical approach is to distinguish between information explicitly provided by the user and information inferred from context. Another is to consider the consequences of an incorrect decision when determining how much confirmation is necessary.

A restaurant suggestion can be presented as an option, with a brief explanation of why it was selected. A booking should make the date, time, and party size clear before confirmation if those details have not already been authorized. A conflicting calendar entry may require a question rather than an automatic choice.

The same principle applies to how an assistant communicates its plan. Before carrying out a task that involves several steps, it can explain what it intends to do, which information it needs, and where the user may need to make a decision. For a simple request, this might require only a short explanation. For a more consequential task, it may mean presenting a sequence of actions and waiting for approval before proceeding.

Planning and visibility become especially important when the task continues across products. If an assistant is finding a restaurant, checking availability, and preparing a reservation, the user should be able to see what has been completed, what is still in progress, and what remains to be decided. Without this visibility, even a successful process can feel unpredictable.

For designers, the practical questions are straightforward. What does the system know, and where did that information come from? What assumptions is it making? What does it plan to do next? How costly would a mistake be? Can the user understand and correct the result?

These questions help teams design for the full journey, including moments when the system cannot confidently proceed, rather than treating uncertainty as an edge case. They also make the experience easier to understand and recover from when something goes wrong.

The product team no longer owns the entire experience

When a task moves across several products, no single interface necessarily contains the full experience. A user may begin in a conversation, use information from a calendar, rely on a recommendation service, and complete an action through another product.

Each product still has a role. Its value also depends on how well it participates in the wider journey. This changes the questions product teams need to ask. They need to understand which information their product makes available, which actions other systems can request, and how those actions fit into a task that may begin somewhere else.

This creates a different relationship between products and their users, because a product can contribute to a successful outcome without the person visiting its interface at all.

For product teams, that means app sessions and screen time are incomplete measures of value. A service might help someone complete a reservation through an assistant, even if the user never opens its home screen. This does not make interface design less important. It changes the scope of the work around it.

It also means that teams need to consider how their products communicate progress and responsibility to systems outside their own interfaces. If an action is started by one product and completed by another, the transition should not leave the user wondering what happened or who is responsible for the outcome.

The experience needs to remain coherent even when its parts belong to different teams. This requires shared expectations about what each product can do, how its actions are reported, and what happens when the next step cannot be completed.

What designers should work on next

Designers can begin by mapping the context a task requires. This includes the information a system needs, where that information comes from, how it changes over time, and which products are involved. They also need to define how the system should behave when context is missing, incomplete, or ambiguous.

Microsoft’s Guidelines for Human-AI Interaction provide a useful foundation for this work. Building on these principles, we can identify a set of recurring patterns for designing agentic experiences and workflows, regardless of their specific domain.

Planning and visibility

Teams need to decide when an assistant should explain its intended steps, when it should show progress, and how people can understand the status of a task that spans several products. The level of detail should depend on the complexity and consequences of the task. People need enough visibility to understand what the system plans to do, what it has completed, and what still requires their attention.

Memory surfacing

Designers should consider how the system uses past conversations, preferences, and decisions to inform the current experience. People need ways to understand which information is influencing a recommendation or action, and to correct it when it is outdated or no longer relevant. The aim is to provide continuity without turning past behaviour into an unquestioned assumption.

Tool use disclosure

When an assistant uses external tools or accesses information across products, people should be able to understand what it has accessed and which actions it has taken. Teams need to distinguish between a suggestion, an attempted action, and a completed outcome. This makes the system’s behaviour easier to understand and helps people know when they need to intervene.

Multi step workflow tracking

A task that spans several products should feel like a continuous journey rather than a series of disconnected operations. Designers can map its stages, the information passed between them, and the decisions that require user involvement. People should be able to see what has been completed, what is in progress, and what remains to be done without having to reconstruct the process themselves.

Recovery routing

Not every task will go as planned. An assistant may encounter missing information, a failed action, or a decision it cannot confidently make. Designers need to define how the system responds in these situations, whether by asking a question, returning to an earlier step, suggesting an alternative, or handing control back to the user. Recovery should preserve the context already established and help people continue without having to start again.

Autonomy

Teams also need to decide how much autonomy to give the system. Autonomy is not a choice between a fully manual experience and an assistant that does everything independently. It is a spectrum.

An assistant might suggest a restaurant without taking any action, prepare a reservation for review, or complete a booking once it has the necessary permission. The appropriate level depends on the task, the user’s preferences, the reliability of the available information, and the consequences of a mistake.

Designing for autonomy means making the transition from recommendation to preparation and action understandable. People should be able to delegate routine steps while retaining control over decisions that matter to them. They should also be able to change their minds, interrupt a process, or take over when the system reaches its limits.

Testing

Testing should follow the task across products. Instead of evaluating each interface separately, teams can examine the full journey from the initial request to the final outcome. They can test what happens when information is unavailable, the system makes the wrong assumption, an action fails, or a user changes their mind. This helps reveal gaps between products that may not be visible when each interface is tested in isolation.

Collaboration

Designing these experiences requires collaboration across product boundaries. Designers need to work with engineers and other teams to understand system capabilities, data access, permissions, and limitations, then translate these constraints into experiences people can understand and trust.

The aim is to make the connections between products feel coherent while keeping the underlying actions visible, understandable, and under the user’s control.

Beyond the interface

The restaurant example began with a simple recommendation. Completing the task required information, decisions, and actions that lived across several products. As AI systems become able to work with more of these connections, the experience will depend on how well context is carried from one step to another.

The design challenge is to make that process useful, understandable, and respectful of the boundaries between systems. People need to know what the assistant remembers, how it plans to proceed, which tools it uses, and what happens when a task cannot be completed as expected. They also need control over how much they delegate and the ability to step in when necessary.

Interfaces remain where people see, guide, and correct what is happening. But more of the experience is shaped by the context that surrounds them, the decisions a system makes, and the rules that determine what it can do.

Designing for this future means looking beyond individual screens and considering the relationships between products. It means treating context, memory, planning, permissions, recovery, and governance as part of the experience from the beginning.

The future of design lies in what happens between products.

References

1. Apple Developer Documentation. Apple Intelligence and Siri AI. Technical guidance on exposing app content and actions, transferable entities, schemas, and onscreen context to Siri and Apple Intelligence.
2. Apple Developer Documentation. Getting started with the App Intents framework. An introduction to making app actions and content available outside the app through system experiences.
3. Apple Developer Documentation. Providing contextual cues to Apple Intelligence and Siri. Guidance on associating app entities with onscreen content so that Siri can interpret contextual references.
4. Techstrong AI article on the MCP AI governance gap. The article highlights issues enterprises need to address as they connect AI systems with tools and organisational data.
5. National Institute of Standards and Technology (NIST). Artificial Intelligence Risk Management Framework (AI RMF 1.0). A voluntary framework for managing AI risks and supporting trustworthy AI systems.
6. National Institute of Standards and Technology (NIST). NIST AI RMF Playbook. Practical suggestions aligned with the framework’s Govern, Map, Measure, and Manage functions.
7. Amershi, S. et al. (2019). Guidelines for Human-AI Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
8. Microsoft Research and Microsoft’s UX design for agents .

From AI × Taste

A monthly series exploring the economics of taste in the age of AI:

• AI made everyone a creator, not a designer
  How AI made generation abundant, making design judgment and taste the new competitive advantage.
• The organizational costs of low taste
  How weak design judgment compounds across products, teams and organizations.
• Taste cannot be delegated
  Why AI can generate endless possibilities, but deciding what should exist remains fundamentally human.
• Designing for the Proxy
  Why AI is changing who reads first, and what happens when we optimize for the intermediary instead of the person.
• When the canvas starts acting, who’s really in control?
  How intelligent interfaces are changing what designers control, from interactions to agency, orchestration and attention.
• The economics of attention in the age of AI. When machines pay attention, what deserves ours?
• Context is the new interface. The future of design lies in what happens between products

Context is the new interface was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.