Semantic cache short-circuit

When an eligible request matches a cached one - either exactly or by semantic similarity - DeepIntShield can return the cached answer without a provider call. A hit still incurs gateway lookup, scoping, and policy-fingerprint work. Savings depend on hit rate, embedding cost, payload size, and provider pricing; measure the complete path with your workload.

The request path is ordered for both latency and enforcement:

1. Exact lookup runs before request coalescing. The direct gate never claims a coalescing leadership slot.
2. A warm exact L1 hit returns with zero provider, semantic, or guard-runtime calls and without durable-store, network, or embedding work.
3. After an exact miss, the current input guard runs before semantic/paraphrase embedding and vector lookup.
4. A semantic hit gets one current validation, including the reverse output-policy barrier. Only its final governed response is promoted to exact reuse and inserted into the process L1.

Default behavior

Fresh eligible workspaces are seeded with the semantic-cache plugin enabled, but the feature is plan-gated, requires a working vector store and embedding path, and can be disabled per workspace or Virtual Key. Self-managed operators must also configure the plugin and its dependencies. Confirm a miss, write, and subsequent hit in Request Trace before relying on it.

The exact-response L1 is shared process-wide across workspaces and split plugin stages. It is bounded to 10,000 entries and 64 MiB by default, and uses a one-second TTL. The local lookup microbenchmark measured about 166 ns. A five-call live acceptance fixture of established exact repeats measured 2.5–11.0 ms gateway time and 0.18–1.45 ms direct-lookup attribution; provider, semantic-cache, and guard-wait phases were all zero. These figures are not an end-to-end latency guarantee.

# Disable the local exact-response tier when remote deletes must be visible
# immediately on every replica.
DEEPINTSHIELD_SEMCACHE_DIRECT_L1_ENABLED=false

# Optional startup-time bounds (defaults shown).
DEEPINTSHIELD_SEMCACHE_DIRECT_L1_MAX_ENTRIES=10000
DEEPINTSHIELD_SEMCACHE_DIRECT_L1_MAX_BYTES=67108864
DEEPINTSHIELD_SEMCACHE_DIRECT_L1_TTL_SECONDS=1

Local invalidation clears all cache stages in the same process. Without a cross-replica invalidation bus, an out-of-process delete can remain visible for at most the configured L1 TTL. Disabling L1 gives strict remote-delete visibility but restores durable-store latency on every exact lookup. Cold work is never a zero-latency path.

A 16-request established-exact burst measured 7.92 ms server p50 / 22.75 ms maximum and 0.107 ms direct-lookup p50, with one response content digest across all responses. This is concurrency and consistency evidence for that fixture, not a throughput guarantee.

Why a cache hit is safe

A cached response carries the guardrail fingerprint captured when it was stored. The exact-hit path requires the current fingerprint plus the complete request scope (tenant, workspace, Virtual Key, provider/model settings, request shape, and stream mode), so a policy or boundary change misses structurally. Semantic reuse is evaluated only after the current input decision, and the reused answer still receives applicable output checks. Evidence is cloned for each request so concurrent hits cannot mutate shared state.

These mechanisms preserve request-path enforcement; they are not proof that a detector is perfectly accurate. Test policy invalidation, Virtual-Key isolation, two-user boundaries, concurrent mutation, TTL expiry, and semantic false-positive rates in staging, and keep a conservative durable cache TTL for sensitive workloads.

Discovered gateway-injected MCP schemas are canonicalized once at discovery, and injected tools use deterministic order. Caller-supplied tool order and duplicates are preserved. This stabilizes exact request identity without request-time discovery or sorting.

Per-virtual-key isolation

Cache entries are scoped per virtual key. Two virtual keys with different policy attachments never share cache entries, even when their content matches.

Illustrative planning ranges

These ranges are workload examples, not benchmark results or savings guarantees. Establish your own hit-rate and cost baseline before rollout.

Workload	Typical hit rate	Cost saved
FAQ bot, customer support templates	40–60%	~50%
Internal copilot, repeated dev questions	25–40%	~30%
Long-form RAG, ad-hoc creative prompts	5–15%	~10%
Streaming code completion	<5%	Minimal

Manage the cache under Cost Optimization. Disable it for Virtual Keys or request classes that must reach the provider every time, such as freshness-critical research queries.