The New Trust Boundary: Humans, Agents and Systems

The trust boundary has moved inside work

Enterprise security has spent years moving away from simple perimeter thinking. NIST zero trust architecture reflects that shift: access decisions should not rely on implicit network trust, and policy should be enforced close to resources using identity, device, context, and other signals. AI agents push this evolution further. The boundary is no longer only about whether a user or workload can access a resource. It is also about whether a delegated AI action should be trusted.

In an agentic workflow, trust moves through several handoffs. A human gives an instruction. The agent interprets it. A model produces text or a plan. Retrieval systems supply context. Tools expose actions. Enterprise systems accept updates. A human may review part of the workflow. Each handoff can introduce error, misuse, manipulation, ambiguity, or over-trust.

This means governance must inspect the work itself. The enterprise cannot assume that a valid login, secure network path, or approved application makes the final action trustworthy. The control question is whether the chain from intent to action remained inside approved boundaries and whether evidence exists for review.

Zero trust is necessary but not enough

Zero trust principles are highly relevant to AI agents. They encourage continuous evaluation, least privilege, explicit policy, and resource-focused protection. Those principles help reduce the risk that an agent or user has broad unmanaged access. They also support a more dynamic view of access decisions than older perimeter models.

But zero trust architecture was not designed to answer every AI governance question. It can help decide whether a request to a resource should be allowed. It may not know whether a model output is reliable, whether an agent interpreted intent correctly, whether an AI recommendation is fair, whether a human approval was meaningful, or whether a vendor model change affected a workflow. These are trust questions beyond access.

The right approach is to extend the zero trust mindset into agent governance. Do not trust an agent merely because it is inside the enterprise. Do not trust a model output merely because the model is approved. Do not trust a tool call merely because the agent has permission. Evaluate context, purpose, risk, action, and evidence at the point of use.

The first handoff is human intent

The first trust boundary is between human intent and agent interpretation. A person may ask an agent to solve a problem, draft a response, analyse a document, update a record, or prepare a workflow. The instruction may be clear, vague, incomplete, rushed, or outside authority. The agent then converts that instruction into steps.

Governance should not treat human instruction as automatically valid. A user may be authenticated and still request something outside policy. They may ask for a prohibited data use, an unauthorised external communication, a shortcut around review, or an action that affects another team’s responsibility. The agent should not inherit unlimited trust from the user’s convenience.

A governed handoff binds the instruction to user, role, approved use case, data class, agent capability, and autonomy mode. If the intent is unclear or material, the agent should ask for clarification, require approval, or route for review. Trust begins with human intent, but governance verifies whether that intent is authorised and specific enough.

The second handoff is model interpretation

The second boundary is between the agent and the model. Models can generate useful analysis, summaries, plans, and recommendations, but they can also produce errors, unsupported statements, inappropriate confidence, or outputs that conflict with policy. NIST describes trustworthy AI through characteristics such as validity, reliability, safety, security, accountability, transparency, explainability, privacy, and fairness, and the OECD AI Principles set out complementary values including transparency, accountability, and human-centred design. Those characteristics depend on context.

A model output should therefore be treated as a proposal, not an unquestioned fact. The control requirements should depend on the use case. Internal brainstorming may tolerate more uncertainty. Customer advice, regulated workflows, security decisions, healthcare, insurance, public-sector services, or operational actions require stronger validation, source traceability, human review, and evidence.

The trust boundary here is not solved by choosing a reputable model. Governance must decide when model output can be used, when it must be checked, what sources support it, what confidence or limitations should be shown, and whether the output can trigger action. Trust is contextual, not brand-based.

The third handoff is retrieval and context

Many AI systems use retrieval to ground outputs in enterprise documents, records, tickets, emails, code, policies, or knowledge bases. Retrieval can improve relevance, but it also creates a trust boundary. Retrieved material may be stale, confidential, biased, incomplete, misclassified, or maliciously crafted. The model may treat retrieved text as instruction rather than evidence.

Prompt injection is one reason this boundary matters. OWASP highlights prompt injection, catalogued as LLM01:2025 Prompt Injection at the top of its Top 10 for Large Language Model Applications, as a major LLM application risk. Indirect prompt injection can occur when untrusted content contains instructions that influence the model or agent. In an enterprise workflow, the untrusted content may be a webpage, email, ticket, document, comment, or external file.

Governance should classify retrieval sources by trust level and data class. It should restrict which sources can influence which workflows, separate instructions from content where possible, validate outputs before action, and record which sources were used. Retrieval makes AI useful, but it also moves the trust boundary into the knowledge environment.

The fourth handoff is tool authority

Tools are where AI output becomes operational. An agent may call a search tool, write to a CRM, update a ticket, send a message, open a pull request, change configuration, query a database, or trigger a workflow. Tool use turns an uncertain output into an action with consequences.

This is why tool authority needs stronger governance than model access alone. An agent that can only draft has a different risk profile from an agent that can send, update, delete, approve, deploy, or change access. The trust boundary sits where the agent attempts to use a tool. An AI control plane should evaluate purpose, data, action type, reversibility, user authority, approval rule, and policy state before the call is executed.

OWASP’s excessive agency category, catalogued as LLM06:2025 Excessive Agency in the OWASP Top 10 for Large Language Model Applications, captures this risk in security language. From a governance perspective, the answer is bounded delegation. Give agents enough authority to help, but not so much that a confused, manipulated, or misused agent can create disproportionate harm without review.

Output handling is a separate boundary

Even when a model output is not allowed to call a tool directly, the output may still cross into another system. A human may copy it into a customer email, paste it into a record, use it in a board paper, add it to code, or rely on it in a decision. The boundary between generated output and downstream use is often overlooked.

Governance should define output handling rules. Some outputs are drafts only. Some require source verification. Some require legal, clinical, security, or compliance review. Some should not be used externally. Some should carry warnings, citations, or confidence limits. Treating all AI output as the same creates too much trust in low-assurance material and too much friction for low-risk material.

This boundary is where user training and workflow design meet technical control. A system can label output, restrict copying, require review before send, preserve source context, or block external use. The aim is not to eliminate human judgement. It is to help people understand when generated content can safely leave the AI environment.

Cross-system handoffs create hidden risk

Agentic workflows often span several systems. A support agent may read documentation, inspect a ticket, query a customer record, draft a response, and update the case. A developer agent may read an issue, inspect code, run tests, open a pull request, and notify a channel. Each system may have its own controls, but the handoff across systems creates the governance risk.

The risk is hidden because each individual step can look acceptable. The document read is allowed. The ticket update is allowed. The message is allowed. The combined chain may still violate policy by moving information across audiences, creating an unsupported commitment, changing a record without review, or triggering a downstream process. Governance has to evaluate the chain, not only isolated calls.

A strong control model records the workflow path. It should show which systems were touched, what data moved, where the output landed, and whether the action crossed an approval threshold. This is how the trust boundary becomes visible across applications that were never designed as one system.

Source trust must be explicit

AI systems often mix sources with different levels of trust. Public webpages, internal policies, old tickets, customer notes, code comments, vendor documentation, email threads, and knowledge-base articles can all become context. If the system treats these sources equally, weak or untrusted material can influence important output.

Source trust should be explicit. Authoritative policies should carry more weight than informal notes. Untrusted external content should not be allowed to instruct the agent. Stale material should be flagged. Sensitive sources should be restricted by use case. If a model uses a source, the user or reviewer should be able to see enough context to judge whether reliance is reasonable.

This does not require perfect source scoring. It requires a governance habit: do not let AI systems silently blend all context into one answer. The trust boundary around source material should reflect authority, freshness, sensitivity, and suitability for the task.

Rollback and containment are part of trust

A trust boundary is stronger when the organisation can contain or reverse a bad action. If an agent drafts text, the harm may be limited until a human sends it. If an agent changes a record, grants access, triggers a payment, closes an incident, or deploys code, the organisation needs a plan for detection, rollback, and containment.

Reversibility should influence governance. Irreversible or hard-to-reverse actions deserve stricter approval and evidence. Reversible actions may be allowed with monitoring and review. Actions that affect customers, safety, legal rights, financial outcomes, infrastructure, or regulated obligations should be treated with care even if they are technically easy to execute.

Containment also helps adoption. Teams are more comfortable using agents when they know mistakes can be detected, paused, rolled back, and reviewed. Governance should therefore define stop conditions, kill switches, rollback owners, incident thresholds, and post-action review for material workflows.

Monitoring should follow the boundary

Monitoring AI agents only at the infrastructure layer misses important signals. The organisation should monitor where trust shifts: unusual data sources, unexpected tool calls, repeated approval overrides, policy blocks, high-risk outputs, vendor feature changes, and actions outside normal workflow patterns. These signals are governance events as much as security events.

The monitoring design should be proportional. Low-risk drafting may need light telemetry and user feedback. High-impact agent workflows may need alerts, sampling, approval analytics, blocked-action review, drift monitoring, and incident triggers. The enterprise should not collect data without purpose, but it should collect enough to know whether controls are working.

Monitoring should feed back into governance decisions. If an agent repeatedly triggers warnings, the workflow may need better design. If users approve too many exceptions, policy may be unclear or business pressure may be too high. If blocked actions cluster around a data source, classification may need improvement. Monitoring makes the boundary adaptive.

Vendor-hosted AI changes control placement

When AI runs inside a vendor-hosted product, the enterprise may not control the full architecture. It may have limited visibility into model behaviour, logging, feature changes, retention, source use, or automated actions. The trust boundary then sits partly outside the organisation’s direct control.

This does not mean vendor AI cannot be used. It means governance should understand which controls the vendor provides and which controls remain with the enterprise. Administrative settings, feature toggles, data restrictions, contract terms, user training, monitoring, and exportable evidence all matter. So does the ability to disable or restrict features if they no longer fit the approved use case.

Vendor-hosted AI should be reviewed as a living capability, not a one-time procurement item. If the vendor adds agentic action, changes a model, expands data use, or introduces a new workflow, the trust boundary may move. The enterprise needs a process to notice and respond.

Trust reporting should show boundary health

Board and executive reporting should not reduce AI trust to the number of tools approved. Leaders need to understand boundary health: which workflows cross from advice to action, which agents have tool authority, which data sources are used, which actions require approval, which exceptions remain open, and which incidents changed controls.

A useful report distinguishes low-risk productivity use from material AI operations. It should identify high-impact agent workflows, unresolved control gaps, vendor-hosted AI dependencies, missing evidence, unusual exception trends, and areas where human review is weak. This helps leaders understand where trust is being assumed and where it is being governed.

The reporting should also connect to decisions. If boundary health is weak in a workflow, someone should own remediation. If evidence is missing, controls should be improved. If a vendor feature moves the boundary, procurement and risk owners should review it. Reporting without ownership does not strengthen trust.

The fifth handoff is human review

Human review is often presented as the safety valve for AI systems. It can be, but only when designed well. A human reviewer may over-trust an AI output, lack time to inspect sources, miss a subtle data issue, or assume another control already checked the action. Human involvement is not automatically meaningful oversight.

A useful review interface should show the objective, agent action, source context, data warnings, policy status, uncertainty, alternatives, and consequence of approval. The reviewer should be able to edit, reject, escalate, or request more information. The record should show what the reviewer saw and what they decided.

This is a trust boundary because the enterprise is shifting from AI recommendation to human-backed action. If the human cannot understand the decision, cannot change the outcome, or is pressured to approve quickly, the boundary is weak. Governance should design review around actual decision quality, not merely the presence of a person.

The sixth handoff is system action

System action is the moment an AI-assisted workflow affects a record, message, customer, employee, infrastructure, account, codebase, payment, case, or operational process. This is where errors become consequences. It is also where governance evidence becomes most valuable.

Before system action, the enterprise should know whether the action is reversible, external, customer-impacting, regulated, security-sensitive, privacy-relevant, or operationally material. High-impact actions should require stronger controls than low-risk drafting. If the action crosses a trust boundary, the system should require approval, block, or escalate.

After system action, the record should be retained. Who initiated the action? Which agent acted? Which model, data, source, and tool were involved? Which policy decision applied? Was there an approval or exception? What changed in the system? Without this record, the organisation may know that an action occurred but not whether the trust boundary was managed.

Trust boundaries are also social

AI trust boundaries are not only technical. They are social and organisational. People may trust AI output because it sounds fluent, because it is embedded in an official tool, because a senior leader endorsed it, or because the work is urgent. Teams may assume that another function has reviewed a feature. Business owners may assume that security approval means governance approval.

These assumptions create risk. A confident model answer can mask uncertainty. An official vendor feature can feel approved even if no use-case review occurred. A human reviewer can become a rubber stamp. An agent can look like a helpful colleague while operating under broad authority. Governance should make trust assumptions visible and testable.

This is why training, communication, and workflow design matter. Users should understand which AI outputs can be relied on, which require verification, which data is restricted, and when approval is required. Trustworthy AI is not only about model behaviour. It is also about the human environment around the model.

Testing should probe the boundary, not only the model

AI testing often focuses on model output quality. That matters, but agentic trust testing should also probe the boundary. Can hostile retrieved content influence a tool call? Can a vague human instruction produce an unauthorised action? Can the agent combine low-risk steps into a high-risk outcome? Can a reviewer approve without seeing enough context?

Boundary testing should include misuse cases, prompt injection scenarios, excessive agency, data-class errors, source confusion, approval fatigue, vendor feature changes, and rollback failure. The aim is not to prove that no error can happen. The aim is to understand where controls fail and how much harm can occur before detection.

This kind of testing is especially useful before agents receive write access, external messaging authority, privileged workflow access, or customer-impacting roles. It helps teams tune autonomy, approval rules, tool boundaries, monitoring, and evidence before the agent becomes operational.

Accountability should sit on both sides of the boundary

A trust boundary fails when accountability sits only on one side. If technical teams own the agent but business teams own the outcome, both perspectives must appear in the governance record. If a human approves an AI recommendation, the system should show what the human approved and which controls prepared the recommendation.

Accountability should include the use-case owner, technical owner, risk or compliance owner, data owner where relevant, and the person or role approving material actions. These owners do not all make every decision, but their responsibilities should be clear before the workflow runs. Ambiguity after an incident is a sign that the boundary was not governed well.

The practical test is simple: after a material AI action, can the organisation name who owned the purpose, who owned the system, who approved the action, who accepted any exception, and who is responsible for remediation if something goes wrong? If not, the trust boundary is still too blurry.

The boundary changes by use case

There is no single AI trust boundary that fits every workflow. A writing assistant for internal brainstorming has a different boundary from an agent that updates customer records. A code assistant has a different boundary from a healthcare triage tool. A public-sector service chatbot has a different boundary from an internal analyst summarising market research.

Governance should classify the boundary by impact. Data sensitivity, external exposure, autonomy, affected people, reversibility, regulatory relevance, vendor dependency, and operational consequence all matter. Low-risk use can move faster with lighter controls. High-impact use needs stronger validation, human oversight, monitoring, and evidence.

This proportional approach keeps governance usable. If every AI action is treated as high risk, teams will resist or bypass controls. If every AI action is trusted by default, the enterprise will learn about risk only after harm occurs. The trust boundary should be risk-based and explicit.

Evidence makes the boundary reviewable

A trust boundary that cannot be reviewed is mostly a hope. Evidence makes the boundary testable. The record should show the handoffs: human to agent, agent to model, model to context, model to tool, agent to human, and agent to system. It should preserve the policy outcome, approval state, exception, and final action.

This evidence should avoid unnecessary exposure. Enterprises do not need to store every sensitive detail forever. They do need enough structured context to support investigation, audit, privacy review, risk reporting, and control improvement. Evidence design should balance accountability with data minimisation and security.

When evidence is strong, assurance teams can test whether the trust boundary worked. Did the system block prohibited tool calls? Did high-impact actions receive approval? Were source warnings visible? Were exceptions time bound? Did incidents lead to control changes? These questions turn AI trust from belief into management practice.

Conclusion: Helixar perspective

Helixar’s view is that enterprises can manage the new trust boundary by evaluating AI activity against policy during operational use and preserving evidence around delegated action. This framing is useful for the handoffs that matter: human intent, agent capability, model or provider use, data class, tool call, approval, exception, and system action.

For security teams, the governance pattern can include least-privilege agent boundaries, sensitive-data restrictions, provider rules, tool-call controls, and incident signals. For governance, risk, privacy, legal, compliance, audit, and business owners, it helps create a shared record that explains why an AI action was allowed, approved, blocked, or escalated.

Helixar does not replace zero trust architecture, identity systems, legal advice, privacy assessment, model evaluation, or human judgement. It complements those disciplines by making AI trust boundaries operational and reviewable. The goal is to avoid treating agentic work as trusted simply because it happened inside an approved tool.

Concretely, Helixar is an AI control plane that sits at each of the handoffs this piece describes, enforcing policy at the moment of every AI or agent action across every model provider, in front of or in place of an AI gateway. At the tool-call and system-action boundaries where excessive agency and irreversible harm concentrate, it verifies user and agent identity and context, evaluates the intended action against policy, and applies a graduated response of observe, alert, require approval, block, or contain, so a confused, manipulated, or misused agent cannot cross from advice to action without review. It is fail-closed by default and enforces organisation-wide cost caps, and every one of these decisions is written to a tamper-evident, independently verifiable evidence trail that captures the user, agent, model or provider, data source, tool call, and policy outcome that the evidence and reporting sections here call for. That trail rolls up into framework-aligned evidence packs, with SOC 2 and ISO 27001 available today and ISO 42001, EU DORA, PCI DSS v4, APRA CPS 234, RBNZ BS-11, and the NZ Privacy Act 2020 mapped and delivered at implementation.

What leaders should ask

Leaders should ask where the organisation’s AI trust boundaries are. Which AI workflows move from recommendation to action? Which agents can call tools? Which data sources can influence outputs? Which external communications, record changes, access changes, or customer-impacting actions require approval? Which vendor features operate automatically?

They should also ask whether trust is supported by evidence. Can the organisation show which policy applied at the point of action? Can it identify when a human reviewed an AI recommendation and what they saw? Can it detect when an agent crosses from low-risk assistance to material action? Can assurance test those boundaries over time?

The new boundary is not a reason to stop AI adoption. It is a reason to govern the points where trust shifts. Enterprises that can see and manage those points will be better positioned to use AI agents confidently, proportionately, and with evidence.