Log Exports

DeepIntShield Enterprise provides comprehensive log export capabilities, allowing you to automatically export request logs, traces, and telemetry data to various storage systems and data lakes on configurable schedules.

Log exports are managed entirely from the DeepIntShield Workspace under Analytics → Exports. Each export is created as a job with a destination, a data configuration, and an optional schedule.

Overview

The log export system enables:

• Scheduled Exports: Daily, weekly, or monthly automated exports
• Multiple Destinations: Object stores, data warehouses, and data lakes
• Format Flexibility: JSON or CSV, with optional gzip compression
• Filtering & Transformation: Export specific data subsets with custom transformations
• Compliance: Meet data retention and audit requirements

Creating an Export Job

From Analytics → Exports in the DeepIntShield dashboard:

1. Click New Export and give the job a descriptive Name.
2. Choose a Destination and fill in its connection fields (see Supported Export Destinations).
3. Configure the Data to export - the format, compression, which datasets to include, and any filters or transformations.
4. (Optional) Set a Schedule to run the export automatically, or leave it off to run on demand.
5. Save the job, then use Run now to export immediately. Completed exports appear in the Exports list, where you can download the produced artifacts.

The sections below describe the destination, data, schedule, and transformation options you configure on this page.

Supported Export Destinations

Each destination is configured with a type and a set of credentials and location fields entered in the export form.

Local download

Choose Local to keep the artifact on the gateway and download it from the Exports list - useful for one-off audits before wiring a remote destination.

Object Storage

Amazon S3

Field	Example
Bucket	deepintshield-logs
Region	us-west-2
Prefix	logs/{year}/{month}/{day}/
Access Key ID	${AWS_ACCESS_KEY_ID}
Secret Access Key	${AWS_SECRET_ACCESS_KEY}

Google Cloud Storage

Field	Example
Bucket	deepintshield-logs
Prefix	logs/{year}/{month}/{day}/
Service Account Key	${GCP_SERVICE_ACCOUNT_KEY}

Azure Blob Storage

Field	Example
Container	deepintshield-logs
Account Name	${AZURE_ACCOUNT_NAME}
Account Key	${AZURE_ACCOUNT_KEY}
Prefix	logs/{year}/{month}/{day}/

Data Warehouses & Lakes

Snowflake

Field	Example
Account	your-account.snowflakecomputing.com
Database	DEEPINTSHIELD_LOGS
Schema	PUBLIC
Table	request_logs
Warehouse	COMPUTE_WH
Username	${SNOWFLAKE_USERNAME}
Password	${SNOWFLAKE_PASSWORD}

Amazon Redshift

Field	Example
Host	deepintshield-cluster.xxxxxx.us-west-2.redshift.amazonaws.com:5439
Database	deepintshield_logs
Schema	public
Table	request_logs
Username	${REDSHIFT_USERNAME}
Password	${REDSHIFT_PASSWORD}

Google BigQuery

Field	Example
Project ID	your-project-id
Dataset	deepintshield_logs
Table	request_logs
Service Account Key	${GCP_SERVICE_ACCOUNT_KEY}

Export Schedules

Enable a schedule on the export to run it automatically. Set the frequency, time, timezone, and (for weekly/monthly) the day. Leave the schedule off to run the export on demand.

Frequency	Settings
Daily	Time (e.g. 02:00) and timezone (e.g. UTC).
Weekly	Day of week (e.g. sunday), time, and timezone.
Monthly	Day of month (e.g. 1), time, and timezone.

Data Formats

Choose JSON or CSV as the output format and, optionally, gzip compression. Each export can include request logs, response logs, error logs, traces, metrics, and MCP tool logs.

JSON Format

{
  "timestamp": "2024-01-15T10:30:00Z",
  "request_id": "req_123456789",
  "customer_id": "cust_abc123",
  "provider": "openai",
  "model": "gpt-4",
  "endpoint": "/v1/chat/completions",
  "method": "POST",
  "status_code": 200,
  "latency_ms": 1250,
  "input_tokens": 100,
  "output_tokens": 150,
  "cost_usd": 0.0045
}

CSV Format

timestamp,request_id,customer_id,provider,model,endpoint,method,status_code,latency_ms,input_tokens,output_tokens,cost_usd
2024-01-15T10:30:00Z,req_123456789,cust_abc123,openai,gpt-4,/v1/chat/completions,POST,200,1250,100,150,0.0045

Data Filtering & Transformation

Narrow the exported data and reshape it directly in the export’s Data configuration.

Filtering Options

Restrict the export to a subset of logs by date range, provider, model, status code, customer, latency bounds, or error state:

{
  "filters": {
    "date_range": {
      "start": "2024-01-01T00:00:00Z",
      "end": "2024-01-31T23:59:59Z"
    },
    "providers": ["openai", "anthropic", "azure"],
    "models": ["gpt-4", "claude-3-sonnet"],
    "status_codes": [200, 201, 400, 401, 403, 404, 500],
    "customers": ["cust_123", "cust_456"],
    "min_latency_ms": 100,
    "max_latency_ms": 10000,
    "has_errors": true
  }
}

Transformation Options

Aggregate, anonymize, or enrich records before they leave DeepIntShield:

{
  "transformations": [
    {
      "type": "aggregate",
      "group_by": ["provider", "model", "date"],
      "metrics": [
        "count",
        "avg_latency",
        "p95_latency",
        "total_tokens",
        "total_cost",
        "error_rate"
      ]
    },
    {
      "type": "anonymize",
      "fields": ["customer_id", "request_id"],
      "method": "hash"
    },
    {
      "type": "enrich",
      "add_fields": {
        "export_timestamp": "${EXPORT_TIMESTAMP}",
        "export_version": "${EXPORT_VERSION}"
      }
    }
  ]
}