workbooks docs

Engine internals

The engine is the BEAM runtime in runtime/host/*.ex — the trusted host behind the Dock membrane. It runs WASM, gates capabilities, brokers the I/O WASM cannot do, builds source to WASM in-sandbox, signs artifacts, and persists state. This page is the internals map: every fact carries a :SRC: anchor and an honest maturity tier per subsystem.

This is reference, not explanation. For why the design is shaped this way see the deep pages under /learn; for the external RCP/HTTP API see the nexus reference; this page is the host code.

Maturity legend: ships-today (verifiable now at a cited file:line) · partial (primitive exists, feature as described is scoped) · north-star (intended, code may be stub/absent) · wall (impossible-as-described under the architecture).

Subsystem map

clusterwhat it ownsprimary source
Agent loopLLM ↔ tool ↔ VFS loop; sessions; defsagent.ex agent_session.ex agent_def.ex
Isolation & policycap profiles, memory/CPU caps, tier ladderpolicy.ex isolation.ex instance.ex
Brokers (Dock caps)host-mediated net/exec/storage/serve, SSRF guard*_broker.ex net_guard.ex instance/imports.ex
Build & compilerssource → WASM in-sandbox, package managercompilers.ex package_manager.ex command_registry.ex
Provenance & signingEd25519 did:key/did:web, artifact manifestsgit.ex did.ex manifest.ex
Storage & VFSper-instance SQLite, bundles, libraryvfs.ex bundle.ex library.ex
Fabric & kernelfan-out (width × tier), hot kernel ABIfabric.ex kernel.ex isolation_node.ex

---

Agent loop

An agent is a BEAM loop: call the LLM → if it requests tools, run them → append results → loop, until the model stops calling tools or signals done. Bounded by max_steps, resumable because state lives in the VFS.

The loop

Workbooks.Agent.run/3 takes (system, task, opts) and returns a run record %{result, steps, events, log}. The loop drives Llm.complete/2; an empty tool_calls or a done tool ends it (agent.ex:172-179). Default max_steps = 12 (agent.ex:148), raised to 40 by the session wrapper (agent_session.ex:60).

optdefaultsource
:model(caller / def)agent.ex:143
:max_steps12agent.ex:148
:vfsVFS.open(":memory:")agent.ex:138
:tenant"dev"agent.ex:147
:execfalseagent.ex:154
:workdirSystem.tmp_dir!()agent.ex:158
:on_stepno-opagent.ex:161

Tool surface

There is no OS shell — the native run bash hatch was deleted (wb-9ja). The agent cannot execute native code by construction; the test seam __tool_names_for_test__/1 (agent.ex:117) asserts this.

toolgrantsource
shellbaseagent.ex:58-65
searchbaseagent.ex:66-70
wbbaseagent.ex:71-75
fetchbaseagent.ex:76-80
web_searchbaseagent.ex:81-85
file_issuebaseagent.ex:86-90
vfs_writebaseagent.ex:91-95
vfs_readbaseagent.ex:96-100
donebaseagent.ex:101-105
gitexec-onlyagent.ex:34-40
publishexec-onlyagent.ex:41-45
imageexec-onlyagent.ex:46-54

shell runs in-WASM over Workbooks.Shell + the CommandRegistry (coreutils, jq, grep, pipes, ; && ||, vars, redirection) — agent.ex:60. The exec tools are host-brokered: trusted host Elixir performs a fixed git commit && push or a constrained File.cp, never an agent-chosen command line (agent.ex:26-32).

:exec is a TRUST grant (host-brokered git/publish/image + OS-workdir routing), not native execution. The run hatch is gone. — agent.ex:151-154

Bounds & telemetry

Every tool call is wall-clock bounded at 150s via Task.async + Task.yield; on overrun the task is Task.shutdown(:brutal_kill) and the model sees a tool error (agent.ex:228-238). Every step is appended to <workdir>/_steps.jsonl (agent.ex:216-221) and to an org event log; the image tool is capped at 2 calls/run (agent.ex:156-157).

Sessions (long-horizon, streamed)

Workbooks.AgentSession runs the agent in a DynamicSupervisor child; the HTTP caller starts it and polls (status/1) or subscribes (subscribe/1, agent_session.ex:89-91) for live :agent_step / :agent_done messages (brandnana-style telemetry). It carries a CTK human-in-the-loop review queue (put_review/2 / take_review/1, agent_session.ex:96-116), persisted per-run to JSONL (agent_session.ex:105-111). The run's own VFS holds resumable state.

Agent definitions (authored as org)

Workbooks.AgentDef.parse/1 finds the first :agent: node and reads :ID: / :MODEL: / :TOOLKITS: / :TAGLINE: props plus the system prompt.

The system prompt is read ONLY from under the ** System prompt heading (agent_def.ex:43-44); with no such heading it falls back to the node body — "an agent authored as a plain def shouldn't silently run prompt-less."

run/3 auto-injects a compact toolkit index from :TOOLKITS: into the prompt (progressive disclosure tier 1; skill bodies stay on-demand via the wb tool — agent_def.ex:28-37).

---

Isolation & policy

Capability profiles

A profile maps to a memory cap, a capability set (which Dock imports of the workbooks:engine world are reachable), and a wall-clock CPU timeout.

profilememorytimeoutcaps
compute64 MiB5 svfs
minimal64 MiB5 svfs commands exec kv secrets queue tcp udp tls
network128 MiB30 sminimal + net llm browse
posix256 MiB60 snetwork + posix parallel

Source: policy.ex:28-37. compute is the fail-closed default: an unknown/typo'd profile resolves to compute (vfs-only), never the over-granting minimal (policy.ex:69-71).

minimal grants the SSRF-brokered raw sockets (tcp/udp/tls) but not high-level egress (net/llm/browse). "minimal network", not "minimal caps" — pick compute for a true sandbox. — policy.ex:22-26

Memory & CPU caps

  • Memory — enforced by Wasmex.StoreLimits from the profile; a component growing past it traps (policy.ex:41-44, instance.ex:72).

  • CPU — a component call runs on a tokio thread; on overrun the boundary call times out and the caller gets {:error, :cpu_timeout}, BEAM stays responsive (instance.ex:101-105). This is a trap, not preemption: the runaway worker thread is only freed by epoch interruption, which is not yet built (instance.ex:9-13).

Network gate

allow_http?/1 is DERIVED from caps: only profiles granting net or browse get WasiP2Options.allow_http, the single stock-wasmex switch gating wasi:http linking AND inherit_network() + DNS (policy.ex:51-67, instance.ex:59). A non-network profile reaches no host network stack at all. Per-instance raw-socket scope is resolve-then-pinned to public IPs host-side (closes DNS-exfil; instance.ex:60-66).

Isolation tier ladder

The depth knob of the (width, tier) surface. tier_for_shape/1 maps an #+EXEC shape to its tier; the tier is determined by shape, not a free knob (isolation.ex:92-100).

tierboundaryshape mappingreality
:instancelinear memorykernel, componentlive — persistent in-VM Wasmex instance
:os_processOS processcommand, posixlive — per-call run_wasmtime subprocess (-W fuel/timeout)
:nodeBEAM VM(escalation target)module present (isolation_node.ex), needs distributed runtime
:containerOS kernelmodule present (isolation_container.ex), needs container image

Default tier comes from #+TRUST: first-party → :instance, third-party → :node (isolation.ex:80-81). Third-party escalates one rung toward stronger isolation (effective_tier/2, isolation.ex:112-120). resolve/1 returns WHY a non-live tier failed rather than a bare error (isolation.ex:127-138).

Instance lifecycle

Workbooks.Instance is one Wasmex.Components component under a GenServer. init/1 owns a VFS, builds the typed Dock imports from the profile's caps (Instance.Imports.for_caps/4, instance.ex:46), and disables stdio inheritance (the typed Dock is the real surface — instance.ex:73-77). A fault returns an error tuple; the BEAM survives. API: call/3, vfs_put/4, vfs_get/3, info/1 (instance.ex:21-31).

---

Brokers (the Dock capability surface)

The Dock membrane: a sandboxed guest names a privileged op; trusted host Elixir performs it under a uniform security cadence. The guest never opens a socket, forks a process, or holds a credential.

Dock world imports (in-VM components)

Instance.Imports.for_caps/4 links one import per granted cap into the workbooks:engine world:

world importcapsource
vfs-queryvfsinstance/imports.ex:34
run-commandcommandsinstance/imports.ex:38
llm-completellminstance/imports.ex:42
browse-fetchbrowseinstance/imports.ex:48
run-command-manyparallelinstance/imports.ex:57

Importing an ungranted cap fails to instantiate. (Compiled-Rust core modules use a parallel env.host_* surface via rust_dock.ex; compiled-JS via js_dock.ex.)

Common security cadence

Every broker enforces: default-deny (a cap must grant it) · per-principal revocation (revocation.ex, checked every call, owned by a never-dying process revocation.ex:9-15) · per-principal rate limit (rate_limiter.ex, atomic time-bucketed counter, default 120_000/60_000ms rate_limiter.ex:18) · size-capped output · audit on denial (broker_audit.ex, record/3, stats/0).

The brokers

brokercapability deliveredkey guardsource
NetGuardSSRF floor for host-mediated egressdenies loopback/RFC1918/link-local (169.254.169.254)/CGNAT/ULA; deny on resolve failnet_guard.ex:7-13
CappedHttpstreaming GET with hard body + deadline cap:stream accumulate to max_bytes, cancel on overflowcapped_http.ex:2-13
ExecBrokerexec(cmd) → in-sandbox CommandRegistryregistered-only, structural argv (no shell), depth ≤ 8, ≤ 64 concurrent/principalexec_broker.ex:19-92
ProcessBrokerfork-exec model (spawn/await/kill)per-principal ≤ 32 live processes (fork-bomb)process_broker.ex:16-17
ParallelBrokerbrokered data-parallelism (map over N):max_inputs 1024, per-task timeout, default-denyparallel_broker.ex:16-19
StorageBrokerdurable per-tenant K/V (survives instance)tenant isolation; value ≤ 1 MiB, ≤ 10k keysstorage_broker.ex:21-23
QueueBrokerinter-guest FIFO message queueper-tenant; msg ≤ 256 KiB, depth ≤ 1000queue_broker.ex:16-18
SecretBrokersign-with-secret (host holds the key)NO read op exists; only sign/3; tenant-scopedsecret_broker.ex:9-12
TcpBrokerraw-TCP request/responseSSRF + resolve-then-pin (connect pinned IP)tcp_broker.ex:7-13
UdpBrokerUDP send/recv-oneSSRF + resolve-then-pinudp_broker.ex:7-9
TlsBrokerTLS request/response (cert-verified)SSRF + pin + SNI = hostname + verifypeertls_broker.ex:7-11
ServeBrokerinbound server flip (host listens, guest handles)per-serve_id request channel, response size-cappedserve_broker.ex:2-13
TcpServeBrokerraw-TCP inbound listener flipper-client rate, request byte cap, mid-flight revocationtcp_serve_broker.ex:2-16
BuildBrokercompile-source-then-run (build other tools)default-deny, source/output caps, sandboxed compile+runbuild_broker.ex:11-17

ExecBroker wire protocol

A guest writes a length-prefixed little-endian request into wasm memory for host_exec:

[name_len:u32][name][argc:u32][ (arg_len:u32)(arg) ]*[stdin_len:u32][stdin]

parse_request/1 returns {:ok, name, argv, stdin} or :error (exec_broker.ex:103-111). Binary-safe; trailing bytes tolerated. The unified fork-bomb defense (@max_concurrent_exec 64) is acquired here — the single choke point all exec callers pass through (exec_broker.ex:71-91).

Brokered CLI tools

Workbooks.BrokeredTools registers real commands (http, pip-fetch) at boot as :pynet commands — CPython with the brokered transport, so every request goes through NetGuard and every exec through ExecBroker. This delivers the curl/httpie class without a per-binary port (brokered_tools.ex:1-9,11-13).

---

Build & compilers

The canon: untrusted source never compiles or runs natively

Every language lane compiles AND runs untrusted source entirely in the wasm sandbox (wasmtime), zero native compilation. The compiler wasms themselves are built once by trusted provisioning.

lanepathhelpersource
Cclang.wasm + wasm-ldCompilers.compile_cpackage_manager.ex:8
Zigzig1.wasm → clang.wasmCompilers.zig_compile…package_manager.ex:9
Rustmrustc.wasm → clang.wasm + stdCompilers.rust_compile…package_manager.ex:10
JSQuickJS-ng built by clang.wasmCompilers.js_compile…package_manager.ex:11
Goyaegi interpreter (yaegi-run.wasm)package_manager.ex:12
TStsc inside QuickJS → JS laneCompilers.ts_compile…package_manager.ex:13

Compiler kinds

Workbooks.Compilers recognizes three kinds (compilers.ex:11-21):

  • compile-and-run — compiler reads source and executes it (compile_run/4).

  • compile-to-c — emits C, fed through the C lane → wasm → run (compile/4).

  • compile-to-wasm — emits an artifact .wasm we then run.

Lanes reuse the command path (CommandRegistry + PackageManager.run, which enables -W exceptions + memory64).

Command registry

A command is a CLI converted to a runnable WASM module (stdin → stdout), invoked by name via run-command (command_registry.ex:1-8). list/0 returns registered names (command_registry.ex:93); run/5 executes one with argv, dir preopens, and depth=/ropts (=command_registry.ex:976).

Proc-macro host bridge

Workbooks.ProcMacroHost replaces a proc-macro executable's native posix_spawn with two host imports under the workbooks module — pm_expand(...) runs the proc-macro server wasm and stashes the result, pm_read(...) copies it back into guest memory (proc_macro_host.ex:5-13).

BuildBroker

See the brokers table — build_broker.ex wraps the proven in-sandbox compile recipes in the standard default-deny / size-cap / rate / revocation cadence so a tool can build other tools without native exec (build_broker.ex:2-17).

---

Provenance & signing

Tenant identity (Ed25519 did:key)

Workbooks.Git idempotently ensures a per-tenant Ed25519 keypair under .workbooks/ (git.ex:27,55). did_key/1 encodes the public key as a real did:key:z6Mk… (multicodec 0xED 0x01 + base58btc — git.ex:103-110). sign/2 signs with the private half (git.ex:117); verify/3 decodes the pubkey straight from the DID, no registry (git.ex:124). WB_SIGNING_KEY (base64 of the 32-byte seed) overrides the stored key (git.ex:81).

did:web resolution

Workbooks.Did republishes the same Ed25519 key as a did:web document resolvable at https://<host>/.well-known/did.json. did:key and did:web wrap one key, so a signature verifies under either (did.ex:8-13). "Self-host for now": no Radicle/PLC/registry.

Artifact signing (workbooks-c2pa)

Workbooks.Manifest.sign/4 embeds <script type="application/workbooks-c2pa+json" id="wb-c2pa-manifest"> carrying an Ed25519-over-canonical-JSON manifest + signature into the published HTML (manifest.ex:16,30,40-53). verify/1 checks two things — signature validity AND asset integrity — and returns %{signature, asset_integrity, valid} (manifest.ex:57-66); the asset half needs no key (pure read). This is the did:key scheme, NOT a full COSE/X.509 C2PA claim — the X.509 cert-chain spike is superseded (manifest.ex:8-15).

---

Storage & VFS

Per-instance VFS

A Workbook's VFS is SQLite-backed, one db file per Instance, with three named volumes (vfs.ex:14):

volumerolelifetime
workspacethe working tree (default)persists
memoryagent long-term memorypersists across freeze/resume
tmpscratchcleared on resume

A volume is the (volume, path) key in one table — one file, one connection. Off-box durability is Litestream's job. The raw vfs-query Dock import sees the volume column, so a component can scope a query to any volume (vfs.ex:7-11).

StorageBroker (durable K/V)

Durable, tenant-isolated K/V that survives the instance — see the brokers table (storage_broker.ex). Distinct from the ephemeral in-instance VFS.

Bundle (egress format)

Workbooks.Bundle packs the Workbook HTML + its SQLite VFS (all volumes) + a manifest into one portable .wbundle zip — any tool can open it (bundle.ex:2-8,11-14). ship → restore round-trips it.

Library (access graph)

Workbooks.Library is an index + access layer over an identity's workspaces, not new machinery: workspaces ← workspace.org manifests in the tenant git repo; provenance ← did:key (library.ex:2-17).

---

Fabric & kernel

Fabric (width × tier fan-out)

Workbooks.Fabric fans a command/kernel over N inputs across isolated WASM instances at a chosen (width, tier). Defaults: width = 16, per-worker timeout = 60_000ms, tier = :instance (fabric.ex:28-29,39-41). :instance runs each input in its own wasmtime instance across BEAM schedulers (strong + cheap); heavier tiers are not yet wired (fabric.ex:11-15).

Kernel ABI (hot path)

Workbooks.Kernel instantiates a bytes → bytes wasm kernel once and calls it many times, reusing a fixed in/out arena — the #+EXEC: kernel shape. The module exports memory and an entry (default process); per call the host writes input at :in_off, calls process(in_len) -> out_len, reads back out_len (kernel.ex:11-17).

paramdefaultsource
:entryprocesskernel.ex:38
:in_off1024kernel.ex:32,39
:out_off65_536kernel.ex:33,40
timeout60_000kernel.ex:34

Per-call cost is one function call + two memory copies — no fresh Store, no WASI stdio round-trip (what command shape pays — kernel.ex:6-9).

Heavier isolation tiers

  • :node — run a command on a separate :peer BEAM VM; full VM isolation AND cross-machine scale (the render-farm case). One peer started lazily and reused (isolation_node.ex:2-16).

  • :container — run the content-addressed .wasm in a throwaway container with --network none, mem/CPU/pids ceilings, read-only rootfs; image WB_CONTAINER_IMAGE (default wb-wasmtime:latest), runtime WB_CONTAINER_RUNTIME (default docker, podman/krunvm drop-in — isolation_container.ex:2-12).

---

See also