Skip to main content

​
S3 Configuration

Depths v0.1.1 can seal each UTC day of data and ship it to S3 or any S3-compatible store. S3-backup is optional : local-only works fine. When S3 is set by environment variables, the server ships sealed days and verifies row counts. You can then read sealed days from S3. The querying experience via the query endpoints stays identical, allowing seamless analysis over local and s3-backed telemetry.

​
When to enable S3

  • Durable storage for historical days
  • Read sealed days from object storage instead of the ingest box
  • S3-compatible endpoints (MinIO, DigitalOcean Spaces, etc.)

​
Environment variables

Set these before you run depths init and depths start.

​
Required

VariableMeaning
S3_BUCKETBucket name
AWS_ACCESS_KEY_ID or S3_ACCESS_KEY_IDAccess key
AWS_SECRET_ACCESS_KEY or S3_SECRET_KEY or S3_SECRET_ACCESS_KEYSecret
AWS_REGION or S3_REGIONRegion (leave empty for some S3-compatible vendors)
AWS_ENDPOINT_URL or S3_URLEndpoint URL, e.g. https://s3.amazonaws.com or http://minio:9000

​
Optional

VariableMeaning
S3_PREFIXKey prefix, e.g. depths-prod
AWS_SESSION_TOKENSession token if using temporary creds
For S3-compatible endpoints using http://, reads use AWS_ALLOW_HTTP=true internally. Prefer TLS in production.

​
Set variables

export S3_BUCKET="my-bucket"
export AWS_ACCESS_KEY_ID="…"
export AWS_SECRET_ACCESS_KEY="…"
export AWS_REGION="ap-south-1"
export AWS_ENDPOINT_URL="https://s3.amazonaws.com"
# optional
export S3_PREFIX="depths-prod"
export AWS_SESSION_TOKEN=""

​
Start with shipping enabled

depths init
depths start
What happens:
  1. During the day, rows are appended to local Delta tables.
  2. On UTC day rollover, the server seals the day, uploads the tables to S3, verifies row counts, then cleans local copies when verification passes.

​
Bucket layout

Days are stored under: 
s3://<bucket>/<prefix>/<instance_id>/days/<YYYY-MM-DD>/
Each day contains six Delta tables under otel/: spans, span_events, span_links, logs, metrics_points, metrics_hist.

​
Verify shipping

Health: 
curl -s http://localhost:4318/healthz | jq
Index (recent lines): 
tail -n 20 ./depths_data/default/index/days.jsonl
You should see the phases per day: sealed → uploaded → verified → cleaned with row counts.

​
Read from S3

Use the CLI to read directly from S3, or let it auto-select. 
depths view -t spans -S s3
depths view -t spans -S auto
The HTTP query endpoints also accept a storage parameter (auto, local, s3) if you prefer API access.

​
Common pitfalls

  • Partial env : any missing required variable disables shipping; set env before depths start.
  • Endpoint : http:// works for compatible stores; prefer https:// for production.
  • Permissions : the key needs PutObject, List, and Get in the bucket and prefix.