Bounded Autonomy

AI agents that operate within explicit operational boundaries.

The Autonomy Spectrum

AI autonomy exists on a spectrum — from fully manual (AI offers no assistance) to fully autonomous (AI decides everything, ignores the human). Gartner predicts that by 2028, at least 15% of work decisions will be made autonomously by agentic AI, up from effectively 0% in 2024. Yet 40% of enterprise applications are expected to embed task-specific AI agents by end of 2026.

The question for regulated industries is not whether to adopt AI agents — it is where on the spectrum each operational decision should sit. Bounded autonomy is the answer: real autonomy within defined operational boundaries, with specific triggers that force escalation to human decision-makers.

Why Unrestricted AI Agents Are Unacceptable

EU GMP Annex 22 (draft July 2025) states it plainly: "AI can assist, draft, summarize, or route work, but it cannot make autonomous GxP decisions." The compliance model is: "AI prepares. Humans decide. Systems execute."

The risks of unrestricted AI agents in regulated environments are specific and consequential:

  • Non-deterministic outputs — LLMs produce probabilistic results; the same input can yield different outputs. This is fundamentally incompatible with GxP requirements for reproducible, traceable decisions
  • Hallucination and fabrication — generative AI can invent information rather than reference validated sources, which is catastrophic in contexts requiring regulatory traceability
  • Validation impossibility — traditional IQ/OQ/PQ protocols validate systems that behave identically every time. Adaptive AI agents do not fit deterministic validation frameworks
  • Accountability void — under 21 CFR Part 211, GMP compliance responsibility rests with named human roles (Quality Unit, Qualified Person). There is no legal mechanism for AI to bear regulatory accountability

The FDA's first AI-related warning letter (April 2026) confirmed this: when Purolea Cosmetics Lab used AI agents to generate drug specifications and SOPs without adequate human oversight, the company's defense — "the AI agent did not identify it" — was rejected outright.

Four Types of Boundaries

Policy Boundaries

Non-negotiable rules encoded from regulations, SOPs, and organizational policies. These function like physical safety interlocks that cannot be violated — defining what an agent is permitted to do, prohibited from doing, and obligated to do.

  • An AI deviation triage system can never close a critical deviation without QA head sign-off, regardless of confidence level
  • An AI batch review system cannot approve product release — that authority rests solely with the Qualified Person
  • Regulatory constraints from FDA 21 CFR Parts 210/211, ICH Q7-Q10, and EU GMP Annex 11/22 become executable constraints, not just reference documents

Evidence Boundaries

Minimum data requirements before the AI can take any action. The system must verify that all required inputs are present and validated before proceeding.

  • A CAPA recommendation engine will not generate a recommendation unless the dataset includes: deviation description, affected batch records, laboratory results, historical similar deviations, and process parameter data
  • Missing any required input triggers a "data incomplete" flag rather than a partial recommendation
  • Under Annex 22 principles, AI outputs must reference validated sources, not generate ungrounded information

Confidence Boundaries

The AI's own certainty scores determine the appropriate action mode. Tiered thresholds govern behavior:

  • Above 95% confidence — execute autonomously
  • 85-95% — execute with notification to human
  • 70-85% — execute with mandatory human review within 24 hours
  • 50-70% — require approval before execution
  • Below 50% — escalate to human immediately

When statistical drift exceeds predefined thresholds, automated detection triggers an alert, the Model Risk Owner is notified, and a human decides whether to roll back or revalidate. Statistical anomalies do not automatically cascade into unvalidated production modifications.

Oversight Boundaries

Mandatory human intervention points regardless of AI confidence. These are absolute gates:

  • Final batch release always requires Qualified Person sign-off
  • Safety-critical parameter changes require production supervisor approval
  • Compliance documentation alterations require QA validation
  • Model updates require Model Risk Owner approval before production deployment

Three Modes of Operation in Practice

Act: Deviation Triage

A pharmaceutical manufacturer processes 200-400 deviations per month. The AI system operates autonomously for routine classification:

  • Auto-categorizes deviations by type (process, documentation, equipment, environmental) when confidence exceeds 95%
  • Auto-assigns severity for clearly low-risk deviations (minor documentation errors, typographical corrections)
  • Auto-routes to the appropriate department and generates investigation templates pre-populated with historical data

AI-assisted triage speeds initial classification by 15-30%. But autonomous action applies only to the routine — never to the complex or high-risk.

Recommend: CAPA Generation

The CAPA recommendation engine never operates in Act mode for final decisions. The maximum autonomy level is Recommend:

  • Searches historical CAPA database for precedent-based recommendations
  • Performs statistical pattern analysis across deviation history
  • Proposes CAPA actions ranked by predicted effectiveness based on historical outcomes
  • Generates draft documentation for human review

Pilot studies show AI-enabled CAPA processing achieves a 75% reduction in processing time (from 6 hours to 1.5 hours per record), with AI-generated documentation drafts reducing writing time by 40-60%. But the human always makes the final CAPA decision — because CAPA approval is a regulatory commitment that AI cannot bear.

Escalate: Batch Impact Decisions

An environmental monitoring system detects a temperature excursion during active manufacturing. The AI:

  • Acts — auto-generates alerts, logs the excursion, notifies personnel via escalation chain
  • Recommends — assesses potential batch impact using product-specific risk profiles, recommends whether affected batches should be held or sampled additionally
  • Escalates — any decision to release, reject, or rework an affected batch requires human sign-off by the Qualified Person with identity-linked, timestamped approval

Under 21 CFR 211.22(c), the Quality Unit must approve or reject all procedures affecting product quality. Batch disposition is explicitly a human accountability — no exception.

The Operational Envelope

The concept of operational envelopes — borrowed from aerospace engineering, where a flight envelope defines the boundaries of safe aircraft operation — translates directly to bounded autonomy in pharma:

  • Parameter space definition — just as a flight envelope defines safe airspeed and altitude ranges, a pharma AI operational envelope defines the data types, deviation categories, and decision types within which the AI is validated to operate
  • Boundary monitoring — real-time monitoring of whether the AI is operating within its defined envelope. Novel deviation types, data quality below threshold, or unprecedented process conditions automatically restrict autonomy
  • Graceful degradation — when boundary conditions are approached, autonomy degrades gracefully rather than failing catastrophically. The AI progressively increases human involvement as it approaches its operational limits

Guardrails vs. Bounded Autonomy

These are often confused but fundamentally different:

  • Guardrails are reactive, external constraints — input/output filtering, rule-based veto conditions, monitoring after the fact. They catch problems at the boundary but do not shape the system's fundamental behavior
  • Bounded autonomy is a proactive architectural design principle — governance built into the system's core from the ground up, not retrofitted with safety checks

The analogy: guardrails are a fence around a cliff. Bounded autonomy is building the road away from the cliff. True safety emerges not from the absence of autonomy but from its careful design.

The Human Factor

Research on human-AI interaction reveals a critical risk: automation complacency. When automation is consistently reliable, operators detect only about 30% of automation failures (Parasuraman et al., 1993). In aviation maintenance, over-trust develops within 60-90 days of consistent AI tool use.

Bounded autonomy addresses this by design:

  • Maintains operator engagement — requiring active human decisions at oversight boundaries prevents the complete disengagement that leads to complacency
  • Provides explainability — transparency in AI reasoning helps maintain vigilance
  • Prevents skill degradation — keeping humans involved in complex decisions ensures they maintain the cognitive skills needed for novel situations
  • Structures accountability — explicit review points, escalation pathways, and the ability to override system outputs

This is the operational model that makes AI trustworthy in regulated environments — not unrestricted autonomy, not reactive guardrails, but governed, evidence-backed, boundary-aware intelligence that knows when to act, when to recommend, and when to hand the decision to a human.