For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

See how Agent Substrate runs, suspends, and resumes the Actors behind every Session.

ActorTemplate

An ActorTemplate is the definition that Agent Substrate creates an Actor from. It states the agent’s container image, the command that starts the agent, the agent’s environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller compiles an AgentAgentA Kubernetes custom resource that pairs one AgentTemplate with one Harness. Each side takes either an inline spec or a reference to an existing resource, and the controller compiles the pair into a revision.Learn more and the resources that it names into an ActorTemplate.

Agent Substrate rejects any change to an ActorTemplate’s spec after it is created, so a running Session stays on a fixed definition. The controller creates a new ActorTemplate for every compiled revisionRevisionThe compiled, immutable output of one Agent, identified by a content digest. A Session runs the revision it was created from for its whole life, so editing the Agent affects only sessions created afterward. instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent references one for as long as the revision is its desired or its latest successful one, and a Session references one for as long as it runs on that revision.

Workers and WorkerPools

Each Actor runs on a Worker, a pod that Agent Substrate starts in advance. A Session that needs an Actor gets one on a Worker that is already running, so no Actor waits for a new pod. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it.

WorkerPools

Workers come from a WorkerPool, a Kubernetes custom resource that an operator provisions before any Agent can compile. A WorkerPool declares how many Workers to keep running and which sandbox technology those Workers use.

An operator never creates a Worker directly. Agent Substrate manages them and keeps enough ready in each WorkerPool for an Actor to start or resume immediately.

A WorkerPool exposes a Kubernetes scale subresource, so a HorizontalPodAutoscaler can size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches this number adds Workers before the pool fills, instead of after placements fail.

Worker capacity

One Worker hosts many Actors at the same time. Each Worker reports its maximum Actor capacity when it starts. A Worker at that capacity refuses further Actors, and Agent Substrate places them on another Worker in the pool. The Worker’s Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors.

Actor placement

Agent Substrate checks two things before it places an Actor on a Worker:

• Whether the Worker has a free Actor slot. This check always runs.
• Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate states how much CPU and memory one Actor needs.

An ActorTemplate that the kagent controller compiles states no per-Actor CPU or memory, so the free Actor slot check is the only one that decides where a Session’s Actor lands. Worker resource limits still matter, because a Worker with no limits reports its whole node as its own capacity.

Agent Substrate first narrows the pool to the Workers that can take the Actor. It then picks two of those Workers at random and places the Actor on the less loaded of the two. A Worker’s load is the higher of two shares: the share of its Actor slots that are filled, and the share of its CPU and memory that is committed. Sampling two Workers keeps the decision cheap, and it avoids the pile-up that follows when every placement aims at the Worker that looks emptiest.

Atespaces

An atespace is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding.

An atespace resembles a Kubernetes namespace, but it is not one. Agent Substrate defines it as its own cluster-wide resource. No namespace contains it, a namespace’s RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it.

Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Agent Substrate places on it, and those Actors can come from different atespaces.

kagent names each atespace after the Kubernetes namespace of the Agent that owns the Actor, and creates the atespace the first time a Session in that namespace needs an Actor. The Actor’s own name is session- followed by the Session’s identifier. A Session on an Agent in the kagent namespace therefore runs on an Actor in the kagent atespace. Both halves of this identity appear in the address that traffic uses to reach the Actor, which Sandboxing covers.

Sandboxing

Because an Actor often runs a model-directed agent that calls tools and executes commands, Agent Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool’s sandboxClass field selects the sandbox technology for its Workers: gVisor, or a micro-VM that runs the workload under Cloud Hypervisor with a Kata Containers kernel and root image. Both technologies isolate an Actor from its Worker’s host kernel, and both support suspend and resume operations.

kagent compiles each ActorTemplate to the sandbox class of the WorkerPool that hosts it, so a default installation runs kagent agents in a gVisorgVisorA user-space kernel that isolates a workload from the host kernel by intercepting its system calls. A default kagent installation compiles ActorTemplates to the gvisor sandbox class.Learn more sandbox, and a pool on the microvm class runs them under the micro-VM. A WorkerPool that backs kagent Harnesses must run the Worker build that matches its class. For what each class isolates, see Sandboxing.

Suspend, snapshot, and resume

An Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond. Agent Substrate suspends an idle Actor, reclaims the capacity that it held, and resumes the Actor when traffic arrives. A WorkerPool can therefore run far more Actors than it has room for at any one moment.

The following diagram traces an Actor through one suspend-and-resume cycle, and shows the second path that opens up once the resulting snapshot is tagged.

    flowchart LR
    pool["WorkerPool"] --> worker1["Worker"]
    worker1 -->|hosts| actor["Actor<br>(running)"]
    actor -->|suspend| snapshot["ActorSnapshot<br>(immutable)"]
    snapshot -->|resume| worker2["Any Worker with room<br>in the pool"]
    snapshot -->|pinned by| tag["ActorSnapshotTag<br>(retention pin)"]
    tag -->|seeds| newactor["New Actor"]
  

A WorkerPool keeps Workers running and ready, and a Worker hosts the Actor alongside other Actors while its Session is active. Suspending this Actor writes its full state to an immutable ActorSnapshot and releases the capacity that it held on its Worker.

The diagram forks at this snapshot, because a snapshot serves two purposes.

• Resume restores the same Actor onto any Worker in the pool with room, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor’s full state, the conversation continues from where it left off. Every idle agent takes this path.
• A TagTagAn Agent Substrate resource that gives one snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. It also acts as a retention pin: Agent Substrate does not collect a snapshot while a tag names it, and only the tag's visibility scope can change afterward.Learn more pins this snapshot so that Agent Substrate does not collect it. Seeding a New Actor from a tag is unavailable in this release. Resuming an existing Actor never goes through a tag.

A tag gives a snapshot a stable name, so callers do not track Agent Substrate’s internal snapshot identity. The tag names one snapshot permanently, and only its visibility scope can change afterward. A tag is also a retention pin, so Agent Substrate does not delete a snapshot while a tag still names it.

For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a checkpoint tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Agent Substrate collects this snapshot once a newer one supersedes it.