Speculative dispatch

Speculative dispatch runs eligible cache/provider work in parallel with input-guard evaluation. The response is never released until the input verdict settles. On an allow path, most guard compute overlaps instead of being added to the provider or cache phases.

For most workloads this is the single biggest latency win available.

Default behavior

On for eligible non-streaming requests. Disable per deployment or plugin configuration when strict sequencing is required.

# Plugin config (preferred, live-reloadable):
{ "speculative_input_guards": true }

# Or env var:
DEEPINTSHIELD_GUARD_SPECULATIVE_INPUT_GUARDS=true

# Force strict input-guard-before-provider sequencing:
DEEPINTSHIELD_GUARD_SPECULATIVE_INPUT_GUARDS=false

What changes

With this enabled, the provider call and input-guard checks run at the same time. The response is only released once the guard verdict is in.

• On the allow path: total latency follows the request critical path. Raw guard compute can overlap exact/semantic-cache and provider work; only the unfinished guard tail at the release barrier is additive.
• On the deny path: the model’s response is discarded and the user sees the standard guardrail_blocked error. You pay for one wasted provider call - that’s the safety vs. latency tradeoff.

The two safety semantics

Read this before enabling

Speculative dispatch changes one behavior compared to strict sequential mode:

1. Deny verdicts - the model has already responded by the time the deny lands. DeepIntShield discards the response and returns guardrail_blocked. The wasted provider call is the cost of the latency win.
2. Input redact verdicts - the model has already seen the unredacted input by the time the redact lands. Serving the model’s response would defeat the redactor, so DeepIntShield promotes input-redact verdicts to block in speculative mode. If you rely heavily on input redaction (e.g. PII scrubbing on prompts), leave speculative dispatch off.

Reading latency telemetry

• guardrail_input is the raw input-guard compute time.
• guardrail_input_wait is the exact time the response barrier had to wait after other eligible work completed.
• The critical-path breakdown uses the latter for added guard latency. Do not sum raw guardrail_input with all cache/provider phases or overlapping work will be counted twice.
• Eligible deterministic local decision reuse measured about 1.42 µs in the local microbenchmark and performs no I/O. External model, webhook, attachment, MCP, and RAG checks run fresh; their inference or I/O latency is irreducible. No guard path is promised to take literally zero time.

When to keep it enabled

• Latency-sensitive chat and copilot UIs where a 200–800ms reduction is visible.
• Workloads with low deny rates (typical: <5%) so the wasted-provider-call cost is small.
• Workloads that don’t rely on input redaction for safety.

When to disable it

• Audit-heavy workloads where every guarded request must complete before the provider call even starts (some regulated environments).
• Workloads with high deny rates (>20%) where the wasted-call cost is real.
• Streaming responses - speculative dispatch is automatically disabled for streaming, since there’s no clean way to discard tokens after the first one hits the wire.