LLM cache¶

Record and replay LLM responses via the llm_cache infra middleware — attach a kind: InfraMiddleware manifest with --infra-ref or workspace infra_refs / --infra-ref, write responses to a JSON file, then play them back without calling the provider.

Reference (parameters, pipeline rules, implementation): llm-cache.md

Prerequisites: Tutorial 0 (task install, .venv on PATH). Commands below assume the repository root as the working directory.

Overview¶

The llm_cache middleware caches every LLM_CALL the runtime makes to the provider — initial answers, tool-call turns (next_step: TOOL_CALL / PARALLEL_TOOL_CALLS), and post-tool completions. Each call is one cache entry keyed by the assembled conversation preview at that point in the run.

Caching is configured in a separate infra manifest, not on the agent:

Provider (standard:openai, standard:openai, …) — where the model runs.
Cache middleware (middleware: llm_cache) — read/write/replay policy and cache_path.
Pipeline — middleware wraps the provider; see Pipeline order.

MAS also has a built-in engine disk cache (policy from merged RuntimeEngine manifests — runtime_refs / --runtime-ref, or CLI --cache-read / --cache-write on mas-ctl chat). When an infra pipeline includes llm_cache, the built-in cache is off for that session. See references/llm-cache.md.

End-to-end workflow (record → fixture → replay)¶

Pick or copy infra manifests from library-samples/infra/ or bundled standard:llm-cache-* refs (write + replay pair, shared cache_path).
Attach provider + write manifest on a chat or experiment run (--infra-ref).
Run the scenario (single-turn or multi-step with tools). Every LLM_CALL appends to the JSON file at cache_path.
Check in the cache file (e.g. library-samples/infra/cache/demo.llm-cache.json).
Replay with the replay manifest — no provider network calls; with raise_on_miss: true, any drift from the recorded trajectory fails fast.

Optional: set include_preview: true on the write manifest so each entry stores the conversation preview (including the user query) for manual inspection and pruning — see Known limitations.

Sample manifests¶

Manifest	Ref / path	Purpose
Write	`library-samples/infra/llm-cache-write.yaml`	`cache/demo.llm-cache.json` (resolved under that infra dir)
Replay	`library-samples/infra/llm-cache-replay.yaml`	Strict replay; same `cache_path`
Read+write	`standard:llm-cache`	XDG default; both read and write
Provider + cache	`standard:llm-proxy-cached`	`InfraBundle` (cache + OpenAI)

Write/replay YAML lives in library-samples/infra/. Bundled standard:llm-cache and standard:llm-proxy-cached live under library-standard/src/mas/library/standard/libs/standard/.

Write manifest (record)¶

library-samples/infra/llm-cache-write.yaml:

apiVersion: infra/v1
kind: InfraMiddleware

metadata:
  name: llm-cache-write

spec:
  middleware: llm_cache
  applies_to:
    - LLM_CALL
  params:
    allow_read: false
    allow_write: true
    raise_on_miss: false
    cache_path: cache/demo.llm-cache.json

allow_read: false — always call the provider on a miss; still write responses. Use the same cache_path in the replay manifest.

Replay manifest (playback)¶

library-samples/infra/llm-cache-replay.yaml:

apiVersion: infra/v1
kind: InfraMiddleware

metadata:
  name: llm-cache-replay

spec:
  middleware: llm_cache
  applies_to:
    - LLM_CALL
  params:
    allow_read: true
    allow_write: false
    raise_on_miss: true
    cache_path: cache/demo.llm-cache.json

On a hit the provider is not called. On a miss, raise_on_miss: true errors instead of falling through to a live call.

Parameter reference: references/llm-cache.md.

Pipeline order¶

Only InfraMiddleware refs add pipeline steps. Provider refs do not — they only supply api_base / model_access.

Refs merge: workspace infra_refs → user default → CLI --infra-ref (not from agent/MAS YAML) (left to right). Pipeline steps append in that order; first step = outermost (cache runs before the provider).

One cache + one provider¶

Either CLI order works — only the middleware ref adds a pipeline step:

--infra-ref standard:openai --infra-ref library-samples/infra/llm-cache-write.yaml
# same as
--infra-ref library-samples/infra/llm-cache-write.yaml --infra-ref standard:openai

Provider plus cache middleware¶

--infra-ref standard:openai --infra-ref library-samples/infra/llm-cache-write.yaml merges to pipeline [llm_cache] wrapping the OpenAI provider.

Multiple middleware¶

Order matters — first ref is outermost:

--infra-ref standard:llm-cache --infra-ref standard:chaos-lite

Formal rules: references/llm-cache.md — Pipeline model.

Mini walkthrough (Tutorial 1 agent + library-samples infra)¶

This reuses the Tutorial 1 agent and overlays — no extra files under the tutorial tree. Cache manifests come from library-samples/infra/.

export PATH="$(git rev-parse --show-toplevel)/.venv/bin:$PATH"
AGENT=docs/tutorials/01-building-an-agent/agent.yaml

Record

rm -f library-samples/infra/cache/demo.llm-cache.json

mas-ctl chat "$AGENT" \
  -q "Say hello in exactly three words." \
  --infra-ref standard:openai \
  --infra-ref library-samples/infra/llm-cache-write.yaml \
  --single-turn

Replay (same query; swap middleware ref)

mas-ctl chat "$AGENT" \
  -q "Say hello in exactly three words." \
  --infra-ref standard:openai \
  --infra-ref library-samples/infra/llm-cache-replay.yaml \
  --single-turn

Prove raise_on_miss (different query → error)

mas-ctl chat "$AGENT" \
  -q "Say goodbye in exactly three words." \
  --infra-ref standard:openai \
  --infra-ref library-samples/infra/llm-cache-replay.yaml \
  --single-turn

Live provider (OpenAI)¶

Requires OPENAI_API_KEY. Bundled refs share $XDG_CACHE_HOME/mas/llm_cache.json.

mas-ctl chat "$AGENT" \
  -q "Say hello in one sentence." \
  --infra-ref standard:openai \
  --infra-ref library-samples/infra/llm-cache-write.yaml \
  --single-turn

mas-ctl chat "$AGENT" \
  -q "Say hello in one sentence." \
  --infra-ref standard:openai \
  --infra-ref library-samples/infra/llm-cache-replay.yaml \
  --single-turn

Or --infra-ref standard:llm-proxy-cached for read-through caching.

Tool-using runs¶

Every LLM_CALL is cached, including responses that schedule tools:

Turn	Typical `next_step`	Cached?
Model requests a tool	`TOOL_CALL` or `PARALLEL_TOOL_CALLS`	Yes
Model answers after tool results	`STOP`	Yes

Each turn is a separate cache entry (the preview includes prior tool results). On replay, the runtime walks the same LLM-call sequence; with deterministic tools (same tool output for the same inputs), the trajectory matches the recording and you get cache hits on every LLM_CALL until the run completes — no live provider calls.

Non-deterministic tool-call IDs change the preview text between runs, so strict replay (raise_on_miss: true) may miss on post-tool turns. Prefer a live recording for fixture-grade replay, and deterministic tools for CI.

Uses Tutorial 1's overlays/tools.yaml (same agent as above):

TOOLS=docs/tutorials/01-building-an-agent/overlays/tools.yaml

# Record
mas-ctl chat "$AGENT" \
  -q "Who is POTUS?" \
  --infra-ref standard:openai \
  -o "$TOOLS" \
  --infra-ref library-samples/infra/llm-cache-write.yaml \
  --single-turn

# Replay
mas-ctl chat "$AGENT" \
  -q "Who is POTUS?" \
  --infra-ref standard:openai \
  -o "$TOOLS" \
  --infra-ref library-samples/infra/llm-cache-replay.yaml \
  --single-turn

Experiments and CI fixtures¶

Treat the cache file as a run fixture: record once, commit cache_path, replay in CI with raise_on_miss: true so tests never hit a live LLM.

Record:

mas-lab benchmark run experiment.yaml \
  --infra-ref standard:openai \
  --infra-ref library-samples/infra/llm-cache-write.yaml

Replay:

mas-lab benchmark run experiment.yaml \
  --infra-ref library-samples/infra/llm-cache-replay.yaml

There is currently no API to stop writing mid-experiment or to prune entries in-process — see Known limitations.

See Tutorial 3.