Embedded guard runtime
Default: On. Guards can evaluate locally without a separate guard-runtime network hop. The cited ~20–300ms saving is workload and deployment dependent; measure it against the separated-runtime topology you would otherwise use.
Performance & Cost
DeepIntShield ships five controls that can reduce guardrail latency or provider cost while preserving the documented enforcement mode. Some are enabled by default and others are explicit opt-ins. Validate effective configuration and measure the complete path before adopting a latency, throughput, or savings target.
2.5–11msLive exact-repeat fixture
Up to 90%Workload-dependent ceiling
max(g, m)Speculative-mode model
~30–50%Example p99 reduction
The five knobs
Section titled “The five knobs”Speculative dispatch
Default: On for eligible non-streaming requests. Run cache/provider work alongside input guards. The raw guard compute overlaps; only its uncovered release-barrier tail is additive.
Async post-guards
Default: On (auto). When no output policy needs to block or redact, the response ships immediately and output checks run in the background.
Per-category timeouts
Default: Opt-in. Set a separate time budget per check class - PII <150ms,
toxicity ~600ms, jailbreak ~1200ms - so one slow classifier no longer pulls p99
up to a flat 1500ms ceiling.
Semantic cache short-circuit
Default: On when configured and entitled. Exact lookup occurs before coalescing. Semantic/paraphrase lookup runs after the current input guard. A hit skips the provider while preserving the output-policy barrier. Some repetitive chatbot examples have shown up to 60% cost reduction; hit rate and safe cache eligibility determine the actual result.
Published example results
Section titled “Published example results”| Metric | Example result or model | Context |
|---|---|---|
| Warm exact L1 lookup | ~166ns | Local microbenchmark; process-local lookup only, not an end-to-end SLO. |
| Live established exact repeats | 2.5–11.0ms gateway / 0.18–1.45ms direct | Five-call acceptance fixture; provider, semantic, and guard-wait phases were zero. |
| 16-way established exact burst | 7.92ms server p50 / 22.75ms max | Direct lookup p50 was 0.107ms and all responses shared one content digest. Fixture evidence, not a guarantee. |
| Deterministic local guard | ~1.42µs decision reuse | Local microbenchmark with no I/O; external model, webhook, attachment, MCP, and RAG checks remain fresh. |
| Allow-path total latency | Critical path plus uncovered guard tail | In speculative mode, raw guardrail_input overlaps cache/provider work; guardrail_input_wait is the additive tail. |
| LLM cost saved | Up to 90% | Upper-bound marketing example across multiple controls, not a universal outcome. |
| Tail latency (p99) | ~30–50% lower | Example comparison using per-category rather than one flat timeout. |