Durable webhooks and object storage

The durable webhook subsystem provides tenant- and workspace-scoped, at-least-once delivery for control-plane events. Enqueue writes the event and matching delivery rows in one database transaction, then bounded workers send the network requests asynchronously. Slow destinations do not hold the enqueue request open.

Enable the subsystem

export DEEPINTSHIELD_WEBHOOKS_ENABLED=true

When disabled, the gateway does not create workers or an object-store backend. Ordinary inference never performs a webhook database, network, or object-store operation.

The subsystem requires the configured SQL store. Hosted management requests require workspace or tenant administrator permission; an unauthenticated local self-host control plane retains its existing local-administrator behavior.

Create a signed subscription

Signing secrets are referenced by environment-variable name and are not stored in the database:

export CUSTOMER_WEBHOOK_SECRET='replace-with-a-random-secret'

curl "$DEEPINTSHIELD_URL/api/webhooks/subscriptions" \
  -H "Authorization: Bearer $DEEPINTSHIELD_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "response-events",
    "url": "https://customer.example/hooks/deepintshield",
    "event_types": ["response.completed", "response.failed"],
    "signing_secret_env": "CUSTOMER_WEBHOOK_SECRET"
  }'

Webhook URLs must use HTTPS. HTTP is accepted only for loopback development, and redirects are not followed.

Enqueue idempotently

curl "$DEEPINTSHIELD_URL/api/webhooks/events" \
  -H "Authorization: Bearer $DEEPINTSHIELD_MANAGEMENT_TOKEN" \
  -H "Idempotency-Key: response-resp_123-completed" \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "response.completed",
    "data": {"response_id": "resp_123", "status": "completed"}
  }'

An accepted event returns 202. A byte-equivalent repeat in the same tenant returns the original event with 200 and does not add delivery rows. Reusing the key with a different event type, JSON payload, or content type returns 409 Conflict.

Delivery and verification

Every delivery includes:

X-DeepIntShield-Event-ID
X-DeepIntShield-Delivery-ID
X-DeepIntShield-Event-Type
X-DeepIntShield-Timestamp
X-DeepIntShield-Signature: v1=<hex hmac-sha256>

The signature input is <timestamp>.<raw request body>. Verify it with the shared secret using a constant-time comparison, reject stale timestamps, and deduplicate by delivery ID.

2xx completes a delivery. Timeouts, 409, 425, 429, and 5xx retry with deterministic exponential jitter; a bounded Retry-After is honored. Other 4xx responses and exhausted attempts move to the dead-letter state.

GET  /api/webhooks/deliveries?limit=100
POST /api/webhooks/deliveries/{delivery_id}/replay

Leases and multi-replica safety

Workers claim one delivery at a time with a conditional database lease. The lease is renewed every third of its duration, and all finish updates are fenced by the current owner. If a worker crashes or loses ownership, another replica can reclaim the delivery after expiry without allowing the stale worker to commit its result.

This coordinates replicas through SQL; it does not promise exactly-once network effects at the receiver.

Payload object storage

Payloads up to 64 KiB stay inline. Larger payloads are written to object storage and verified against the immutable SHA-256 checksum before delivery. The default maximum payload is 16 MiB; DEEPINTSHIELD_WEBHOOK_MAX_PAYLOAD_BYTES can set a value from 1 byte through 64 MiB.

The default atomic file backend is suitable for one gateway node. Use an S3-compatible backend for replicated deployments:

export DEEPINTSHIELD_WEBHOOK_OBJECT_STORE=s3
export DEEPINTSHIELD_WEBHOOK_S3_BUCKET=deepintshield-webhooks
export DEEPINTSHIELD_WEBHOOK_S3_REGION=us-east-1
export DEEPINTSHIELD_WEBHOOK_S3_PREFIX=production

# Optional for an S3-compatible service:
export DEEPINTSHIELD_WEBHOOK_S3_ENDPOINT=https://objects.example.com
export DEEPINTSHIELD_WEBHOOK_S3_FORCE_PATH_STYLE=true

The default AWS credential chain is used when static credentials are absent. If required, set DEEPINTSHIELD_WEBHOOK_S3_ACCESS_KEY_ID, DEEPINTSHIELD_WEBHOOK_S3_SECRET_ACCESS_KEY, and optionally DEEPINTSHIELD_WEBHOOK_S3_SESSION_TOKEN.

Retention

Only events whose deliveries are all delivered or dead are eligible for pruning. Pending, retrying, and in-progress work is retained. Failed object deletions become durable cleanup intents and retry independently.

export DEEPINTSHIELD_WEBHOOK_TERMINAL_RETENTION=720h # 1h..8760h
export DEEPINTSHIELD_WEBHOOK_PRUNE_INTERVAL=1h       # 1m..24h
export DEEPINTSHIELD_WEBHOOK_PRUNE_BATCH_SIZE=100    # 1..1000

Management routes

Method and route	Purpose
`GET /api/webhooks/subscriptions`	List subscriptions in the active scope
`POST /api/webhooks/subscriptions`	Create a subscription
`PUT /api/webhooks/subscriptions/{id}`	Replace its configuration
`DELETE /api/webhooks/subscriptions/{id}`	Disable it
`POST /api/webhooks/events`	Enqueue an event
`GET /api/webhooks/deliveries`	Inspect recent deliveries
`POST /api/webhooks/deliveries/{id}/replay`	Requeue a delivery that is not currently `in_progress`

See Protocol operations for the inference-side resource APIs that can produce events in an application workflow.