S3 Configuration

Depths v0.1.1 can seal each UTC day of data and ship it to S3 or any S3-compatible store. S3-backup is optional : local-only works fine. When S3 is set by environment variables, the server ships sealed days and verifies row counts. You can then read sealed days from S3. The querying experience via the query endpoints stays identical, allowing seamless analysis over local and s3-backed telemetry.

When to enable S3

Durable storage for historical days
Read sealed days from object storage instead of the ingest box
S3-compatible endpoints (MinIO, DigitalOcean Spaces, etc.)

Environment variables

Set these before you run depths init and depths start.

Required

Variable	Meaning
`S3_BUCKET`	Bucket name
`AWS_ACCESS_KEY_ID` or `S3_ACCESS_KEY_ID`	Access key
`AWS_SECRET_ACCESS_KEY` or `S3_SECRET_KEY` or `S3_SECRET_ACCESS_KEY`	Secret
`AWS_REGION` or `S3_REGION`	Region (leave empty for some S3-compatible vendors)
`AWS_ENDPOINT_URL` or `S3_URL`	Endpoint URL, e.g. `https://s3.amazonaws.com` or `http://minio:9000`

Optional

Variable	Meaning
`S3_PREFIX`	Key prefix, e.g. `depths-prod`
`AWS_SESSION_TOKEN`	Session token if using temporary creds

For S3-compatible endpoints using http://, reads use AWS_ALLOW_HTTP=true internally. Prefer TLS in production.

Set variables

export S3_BUCKET="my-bucket"
export AWS_ACCESS_KEY_ID="…"
export AWS_SECRET_ACCESS_KEY="…"
export AWS_REGION="ap-south-1"
export AWS_ENDPOINT_URL="https://s3.amazonaws.com"
# optional
export S3_PREFIX="depths-prod"
export AWS_SESSION_TOKEN=""

$env:S3_BUCKET="my-bucket"
$env:AWS_ACCESS_KEY_ID="…"
$env:AWS_SECRET_ACCESS_KEY="…"
$env:AWS_REGION="ap-south-1"
$env:AWS_ENDPOINT_URL="https://s3.amazonaws.com"
# optional
$env:S3_PREFIX="depths-prod"
$env:AWS_SESSION_TOKEN=""

Start with shipping enabled

depths init
depths start

depths init
depths start

What happens:

During the day, rows are appended to local Delta tables.
On UTC day rollover, the server seals the day, uploads the tables to S3, verifies row counts, then cleans local copies when verification passes.

Bucket layout

Days are stored under:

s3://<bucket>/<prefix>/<instance_id>/days/<YYYY-MM-DD>/

Each day contains six Delta tables under otel/: spans, span_events, span_links, logs, metrics_points, metrics_hist.

Verify shipping

Health:

curl -s http://localhost:4318/healthz | jq

Index (recent lines):

tail -n 20 ./depths_data/default/index/days.jsonl

Get-Content .\depths_data\default\index\days.jsonl -Tail 20

You should see the phases per day: sealed → uploaded → verified → cleaned with row counts.

Read from S3

Use the CLI to read directly from S3, or let it auto-select.

depths view -t spans -S s3
depths view -t spans -S auto

depths view -t spans -S s3
depths view -t spans -S auto

The HTTP query endpoints also accept a storage parameter (auto, local, s3) if you prefer API access.

Common pitfalls

Partial env : any missing required variable disables shipping; set env before depths start.
Endpoint : http:// works for compatible stores; prefer https:// for production.
Permissions : the key needs PutObject, List, and Get in the bucket and prefix.

Get Started

Guides

S3 Configuration

S3 Configuration

When to enable S3

Environment variables

Required

Optional

Set variables

Start with shipping enabled

Bucket layout

Verify shipping

Read from S3

Common pitfalls

​S3 Configuration

​When to enable S3

​Environment variables

​Required

​Optional

​Set variables

​Start with shipping enabled

​Bucket layout

​Verify shipping

​Read from S3

​Common pitfalls

S3 Configuration

When to enable S3

Environment variables

Required

Optional

Set variables

Start with shipping enabled

Bucket layout

Verify shipping

Read from S3

Common pitfalls