Multimodal inference

SDK 2.8.3 includes the native OpenAI Python client for gateway inference. Use shield.openai() or shield.async_openai() and select a configured provider/model-id on each request. Optional provider SDK extras are not required for this path.

The gateway’s 29 provider identities expose different operations. A working text request does not establish that the same model accepts images, PDFs, uploaded files, audio, or video. Check the selected provider/model capability, account entitlement, virtual-key permissions, and gateway support in the operation guide.

Choose the native operation

Input or task	Native method	Requirement
Text	chat.completions.create	Chat-compatible model
Image understanding	chat.completions.create with image_url content	Vision-capable Chat model and supported image encoding
Inline PDF	responses.create with input_file.file_data	Model and gateway adapter supporting PDF input through Responses
Image generation/editing	images.generate / images.edit	Supported image model and operation-specific fields
Speech generation	audio.speech.create	Supported speech model, voice, and provider-specific language settings where required
Transcription	audio.transcriptions.create	Supported transcription model and audio file format
Video generation	videos.create, retrieve, download_content	Supported video model; some require a reference image
Uploaded PDF	files.create, responses.create, files.delete	Provider support for upload, Responses file input, and deletion

Pass additional fields using the native method’s request arguments or extra_body where required. The SDK does not invent universal sampling, reasoning, image-size, voice, or video-duration values. Keep nested model IDs, Azure deployment names, and Bedrock inference-profile IDs intact.

Inline image and PDF examples

This example uses a local image as a Chat data URI:

import base64
from pathlib import Path
from deepintshield import DeepintShield

image_data = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")

with DeepintShield.from_env() as shield:
    with shield.openai() as client:
        response = client.chat.completions.create(
            model="openai/gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image."},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/png;base64,{image_data}",
                    }},
                ],
            }],
        )
        print(response.choices[0].message.content)

For inline PDF input, use the Responses content shape:

import base64
from pathlib import Path
from deepintshield import DeepintShield

pdf_data = base64.b64encode(Path("document.pdf").read_bytes()).decode("ascii")

with DeepintShield.from_env() as shield, shield.openai() as client:
    response = client.responses.create(
        model="openai/gpt-4o-mini",
        input=[{
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Summarize this document."},
                {"type": "input_file", "filename": "document.pdf",
                 "file_data": f"data:application/pdf;base64,{pdf_data}"},
            ],
        }],
        store=False,
    )
    if response.status != "completed":
        raise RuntimeError(f"Response did not complete: {response.status}")
    print(response.output_text)

Use a model enabled in your workspace. These calls submit the supplied bytes to the selected provider. Apply file-size and format checks before constructing requests; provider limits can be lower than your application’s upload limits.

Run the example suite

Download the multimodal runner served with this documentation release. It provides text, vision, PDF, image, speech, transcription, video and file examples across applicable provider identities. The download manifest records the SDK version and script SHA-256; the license accompanies it.

Install the matching SDK, download the script, then select an operation:

pip install "deepintshield==2.8.3"
curl --fail --silent --show-error \
  https://aidocs.deepintshield.com/examples/multimodal/run.py \
  --output deepintshield-multimodal.py
export DEEPINTSHIELD_VIRTUAL_KEY="<virtual-key>"
# Optional; the default is https://app.deepintshield.com.
export DEEPINTSHIELD_BASE_URL="http://localhost:8080"

python deepintshield-multimodal.py --list
python deepintshield-multimodal.py \
  --provider openai --operation vision --model gpt-4o-mini --dry-run
python deepintshield-multimodal.py \
  --provider openai --operation vision --model gpt-4o-mini
python deepintshield-multimodal.py \
  --provider anthropic --operation pdf --model claude-sonnet-4-5 --stream

--list and --dry-run require no credentials, import no provider SDK, make no requests, and write no files. A real run requires an explicit model. The runner does not discover or substitute one. The script is distributed with these docs; pip install deepintshield does not install an examples command.

Without --input, the vision example generates a blue PNG in memory and the PDF/file examples generate a one-page PDF with a blue square. They check model input handling using synthetic content. Use --input and --prompt for your own inputs; transcription requires an audio file, and speech requires an explicit --voice. Sarvam speech additionally needs the selected language in --parameters '{"extra_body":{"language_code":"en-IN"}}'.

The runner accepts up to 32 MiB of local input and 128 MiB of binary output; provider limits may be lower. --output writes only to a new path. Image URLs are reported without fetching them. Video runs poll for up to 180 seconds, then download completed content through the gateway; a timeout does not cancel the upstream job. The file example attempts to delete its own uploaded file in finally, including after inference failures, and reports unconfirmed cleanup. A forced process termination cannot run that cleanup.

Streaming completion and delivery

The runner supports --stream for text, vision, and inline PDF examples, consumes the stream, and prints the final result. It rejects failed, incomplete, interrupted, or empty text results. Dedicated media operations use their normal native methods.

In an application, consume Chat deltas and terminal finish_reason, or Responses events through response.completed. Handle response.failed, response.incomplete, and error events explicitly. An HTTP success or an initial text delta does not establish completed inference. Tool, reasoning, and refusal events need their own handling when present; see Chat and guardrails.

Output policy inspection is incremental. Scan cadence can release text before evaluation, and already delivered deltas cannot be recalled. Redaction of a final response snapshot does not sanitize earlier stream chunks. Use buffered inference or a separate application buffering boundary when the complete verdict must precede delivery.

What guardrails inspect

Virtual-key authentication, permissions, budgets, and selected policies still apply. Content inspection depends on the request path:

Input path	Current inspection coverage
Chat/Responses text	Conversation text is evaluated under selected policies. Attachment text is extracted from the last message; earlier attachments are not generally re-extracted into policy text.
Inline PDF	Supported page/form text and font mappings are decoded within parser bounds. Font/image binary data is not treated as prose. PDF inspection-failure checks also visit attachments in earlier messages.
Inline image	Supported textual metadata participates in inspection; this extractor does not OCR image pixels.
Dedicated image/audio/video operations	Eligible prompts, input text, and transcript text are evaluated when server multimodal guardrails are enabled. This does not transcribe raw audio or inspect video frames.
File upload or file_id	Upload/management is not selected for file-content inspection by the LLM guardrail hook. The extractor does not resolve provider file IDs. A working upload lifecycle does not establish content inspection.
Remote image/file URL	The extractor does not fetch remote bytes for inspection. A reference alone does not make that content available to guardrails.

Dedicated media evaluation requires the administrator to enable GUARDRAILS_MULTIMODAL=true on the server. This is separate from SDK connection configuration and enables the implemented paths; it does not add OCR, speech-to-text, video analysis, or file-reference resolution.

Binary redaction and PDF inspection

When an enforcing policy requires redaction inside a binary attachment and a safe rewrite is unavailable, the gateway blocks the request before the provider. Supply a sanitized copy. Unsupported, encrypted, or over-limit PDF inspection follows the selected policy execution mode. An image-only PDF may be valid but contain no extractable text; this does not establish that its pixels are free of sensitive data.

For negative policy checks, use synthetic input that matches an active enforcing policy and require the structured guardrail_blocked result. A provider access denial, quota error, service failure, or successful inference is a different outcome. Native provider failures retain native exception types; see Error codes.