# Architecture
Source: https://docs.depthsai.com/architecture

Bird’s-eye view of signal flow, components, and storage model.

Depths v0.1.1 runs a FastAPI service (`depths.cli.app`) that accepts OpenTelemetry over HTTP and persists normalized rows into six Delta Lake tables. The service is built around a single orchestrator, `depths.core.logger.DepthsLogger`, which wires ingestion, validation, batching, local storage, optional S3 shipping, and read APIs.

## Signals and endpoints

Depths listens on the standard OTLP HTTP paths:

* `POST /v1/traces`
* `POST /v1/logs`
* `POST /v1/metrics`
* `GET /healthz` for liveness and minimal diagnostics
* `GET /api/spans`, `GET /api/logs`, `GET /api/metrics/points`, `GET /api/metrics/hist` for quick reads

OTLP over HTTP uses JSON or protobuf payloads. Typical content types are `application/json` and `application/x-protobuf`. The default HTTP port is 4318 and most SDKs add `/v1/{signal}` to the base endpoint automatically.

## Core components

* **Ingestion surface**: `depths.cli.app` exposes the endpoints and owns process lifecycle.
* **Orchestrator**: `depths.core.logger.DepthsLogger` constructs and coordinates everything under a single instance root.
* **Mappers**: `depths.core.otlp_mapper` converts decoded OTLP payloads to row dicts shaped for our tables, stamping resource and scope context.
* **Producer**: `depths.core.producer.LogProducer` validates and normalizes events against an `EventSchema`.
* **Aggregator**: `depths.core.aggregator.LogAggregator` drains the producer buffer, batches rows into typed DataFrames, and appends to Delta tables.
* **Delta I/O**: `depths.core.delta` provides safe creates, appends, compaction, checkpoint, and vacuum helpers over `deltalake` + Polars.
* **S3 shipper**: `depths.core.shipper` can seal a UTC day, upload to S3, and verify row counts, using `depths.core.config.S3Config`.
* **Config**: `depths.core.config` centralizes options for logger, producer, aggregator, shipper, and S3.

## Data model and tables

Depths persists six OTel-aligned tables under a per-day directory:

* Spans
* Span events
* Span links
* Logs
* Metric points (Gauge or Sum)
* Metric histograms (Histogram, Exponential Histogram, Summary)

Tables are Delta Lake. Polars reads and writes Delta and supports lazy scans that push down filters before work is done, which helps the read APIs.

## Signal journey

1. **Receive**\
   FastAPI endpoint in `depths.cli.app` accepts OTLP JSON or protobuf. Payloads may be gzip-encoded. The app creates a process-wide `depths.core.logger.DepthsLogger` on first request.

2. **Map**\
   `depths.core.otlp_mapper` converts the OTLP message to table-specific rows. Resource and scope attributes are normalized. Correlation IDs are coerced to stable formats.

3. **Produce**\
   `depths.core.producer.LogProducer` applies the `depths.core.schema.EventSchema` contract: defaults, computed fields, type coercion, and required checks. Valid rows go into a bounded queue.

4. **Aggregate and persist**\
   `depths.core.aggregator.LogAggregator` batches rows into typed Polars DataFrames and appends them to the correct Delta table under the current UTC day. The aggregator tracks triggers like batch age and size.

5. **Seal and ship (optional)**\
   At day close, or on demand, `depths.core.shipper` seals the day: compact files, write a checkpoint, vacuum old debris, compute row counts and versions, then upload to S3 and verify. Optimize, checkpoint, and vacuum are standard Delta maintenance steps.

6. **Read**\
   Read endpoints and the `depths.core.logger.DepthsLogger.read_*` helpers construct Polars lazy scans over local paths or S3 URIs. Delta’s transaction log lets Polars read only what is needed.

## Instance layout and day boundaries

Each Depths instance has a root directory that contains configs, indexes, and a `staging/days/YYYY-MM-DD/otel/` tree with one Delta table per signal. Day boundaries are in UTC. Capturing the target table path at enqueue time keeps late batches on the correct day.

## Storage modes

Readers accept a `storage` selector:

* `auto` picks S3 if a sealed day exists remotely, otherwise local.
* `local` forces local Delta tables.
* `s3` forces S3 and uses `S3Config.to_delta_storage_options()` to construct reader options.

## Health, lifecycle, and reliability

* `GET /healthz` returns a lightweight JSON document with process and pipeline stats.
* On startup, the orchestrator prepares schemas and directories, installs exit hooks, and resumes any unshipped days.
* The producer buffer and the two aggregator threads create backpressure and clear durability points between network ingress and storage.

## Why Delta Lake and Polars

* **Delta** gives ACID tables on object storage and a transaction log. Operations like **OPTIMIZE**, **CHECKPOINT**, and **VACUUM** reduce file counts, make table state discovery fast, and clean stale files.
* **Polars** integrates with Delta via `scan_delta` for lazy reads and supports Delta writes, which keeps ingestion simple and reads efficient.

## What this means for you

* Point any OTLP HTTP exporter to `http://host:4318`. Most exporters append `/v1/{signal}` automatically and support JSON or protobuf.
* Depths handles the rest: map, validate, batch, write to Delta, optionally ship to S3, and let you query back with simple endpoints or the Python API.


# CLI options
Source: https://docs.depthsai.com/guides/cli-options

Deep dive into each depths CLI command with flags and OS-specific examples.

# CLI options

The CLI in `depths.cli.cli` manages a local OTLP/HTTP server (`depths.cli.app:app`) and your on-disk instance layout. This page explains every command, its flags, defaults, and short examples for macOS/Linux and Windows (PowerShell).

> Tip: See `depths <command> --help` any time for built-in usage.

## `depths` init

Initialize a new on-disk instance. Creates the directory layout and baseline config by instantiating `depths.core.logger.DepthsLogger` once.

### Synopsis

```bash
depths init [OPTIONS]
```

### Options

| Flag                  | Type   |         Default | Description                                              |
| --------------------- | ------ | --------------: | -------------------------------------------------------- |
| `--instance-id`, `-I` | string |       `default` | Logical instance id (used as folder name under `--dir`). |
| `--dir`, `-D`         | path   | `./depths_data` | Root directory where the instance lives.                 |

### Examples

<CodeGroup>
  ```bash macOS/Linux
  depths init
  depths init -I prod -D ./observability
  ```

  ```powershell Windows (PowerShell)
  depths init
  depths init -I prod -D .\observability
  ```
</CodeGroup>

***

## `depths` start

Start the OTLP/HTTP server (FastAPI + Uvicorn) for an instance.

### Synopsis

```bash
depths start [OPTIONS]
```

### Options

| Flag                  | Type   |         Default | Description                                          |
| --------------------- | ------ | --------------: | ---------------------------------------------------- |
| `--instance-id`, `-I` | string |       `default` | Instance to serve.                                   |
| `--dir`, `-D`         | path   | `./depths_data` | Root directory for the instance.                     |
| `--host`, `-H`        | string |       `0.0.0.0` | Bind host for OTLP/HTTP.                             |
| `--port`, `-P`        | int    |          `4318` | Bind port (OTLP/HTTP default).                       |
| `--reload`, `-R`      | bool   |         `false` | Code reload. **Only supported with `--foreground`.** |
| `--foreground`, `-F`  | bool   |         `false` | Run in the current terminal (not daemonized).        |

### Examples

<CodeGroup>
  ```bash macOS/Linux
  # background (daemonized)
  depths start -I prod -D ./observability -H 0.0.0.0 -P 4318

  # foreground with live logs
  depths start -I prod -D ./observability --foreground

  # hot reload (foreground only)
  depths start --foreground --reload
  ```

  ```powershell Windows (PowerShell)
  # background (daemonized)
  depths start -I prod -D .\observability -H 0.0.0.0 -P 4318

  # foreground with live logs
  depths start -I prod -D .\observability --foreground

  # hot reload (foreground only)
  depths start --foreground --reload
  ```
</CodeGroup>

***

## `depths` view

Pretty-print the latest rows from a persisted OTel table (from local or S3, depending on `--storage` and your environment).

### Synopsis

```bash
depths view [OPTIONS]
```

### Options

| Flag                  | Type       |         Default | Description                                                                                                            |         |       |
| --------------------- | ---------- | --------------: | ---------------------------------------------------------------------------------------------------------------------- | ------- | ----- |
| `--instance-id`, `-I` | string     |       `default` | Instance to read from.                                                                                                 |         |       |
| `--dir`, `-D`         | path       | `./depths_data` | Root directory for the instance.                                                                                       |         |       |
| `--storage`, `-S`     | enum       |          `auto` | Source: `auto`                                                                                                         | `local` | `s3`. |
| `--rows`, `-n`        | int        |            `10` | Show latest N rows by `event_ts`.                                                                                      |         |       |
| `--table`, `-t`       | enum       |               — | One of: `spans`, `span_events`, `span_links`, `logs`, `metrics_points`, `metrics_hist`. If omitted, a picker is shown. |         |       |
| `--select`, `-s`      | repeatable |               — | Project specific columns (repeat `-s` for multiple).                                                                   |         |       |
| `--date-from`         | YYYY-MM-DD |               — | Start day (UTC, inclusive).                                                                                            |         |       |
| `--date-to`           | YYYY-MM-DD |               — | End day (UTC, inclusive).                                                                                              |         |       |

### Examples

```bash
# latest 10 spans
depths view -t spans

# latest 50 logs with selected columns
depths view -t logs -n 50 -s trace_id -s span_id -s body

# force local or s3
depths view -t spans -S local
depths view -t spans -S s3
```

***

## `depths` status

Query the running server’s `/healthz` and render producer/aggregator stats.

### Synopsis

```bash
depths status [OPTIONS]
```

### Options

| Flag              | Type   |     Default | Description              |
| ----------------- | ------ | ----------: | ------------------------ |
| `--host`, `-H`    | string | `127.0.0.1` | Server host to probe.    |
| `--port`, `-P`    | int    |      `4318` | Server port to probe.    |
| `--timeout`, `-T` | float  |       `5.0` | HTTP timeout in seconds. |

### Examples

```bash
depths status
depths status -H 127.0.0.1 -P 4318 -T 5
```

***

## `depths` stop

Stop a background server for an instance. Reads the pidfile written by `start`, sends a graceful terminate, and escalates when needed.

### Synopsis

```bash
depths stop [OPTIONS]
```

### Options

| Flag                  | Type   |         Default | Description                                                                                                   |
| --------------------- | ------ | --------------: | ------------------------------------------------------------------------------------------------------------- |
| `--instance-id`, `-I` | string |       `default` | Instance to stop.                                                                                             |
| `--dir`, `-D`         | path   | `./depths_data` | Root directory for the instance.                                                                              |
| `--force`, `-F`       | bool   |         `false` | Force kill if graceful stop fails. If `SIGKILL` is not available on your OS, the CLI falls back to `SIGTERM`. |

### Examples

<CodeGroup>
  ```bash macOS/Linux
  depths stop
  depths stop -I prod -D ./observability
  depths stop --force
  ```

  ```powershell Windows (PowerShell)
  depths stop
  depths stop -I prod -D .\observability
  depths stop --force
  ```
</CodeGroup>

***


# Customizing Depths
Source: https://docs.depthsai.com/guides/customize-depths

Configure DepthsLoggerOptions with LogProducerConfig and LogAggregatorConfig, verify persistence, and reload.

Depths v0.1.1 exposes clear knobs for ingestion and batching. In this guide you will create a `depths.core.config.LogProducerConfig` and a `depths.core.config.LogAggregatorConfig`, attach them to `depths.core.config.DepthsLoggerOptions`, run a short ingest, verify that options persist to disk, and recreate the logger without passing options to confirm reload.

## What you will build

* A customized `depths.core.logger.DepthsLogger` with tuned producer and aggregator
* A small dataset ingested locally
* A check that `options.json` is written to the instance and reloaded on restart

## Prerequisites

* Python 3.12+
* `pip install depths`

## Imports and versions

The instance layout uses `INSTANCE_DIR/INSTANCE_ID` as the instance root. Options are stored under `<instance_root>/configs/options.json`.

```python
import os, json, time, datetime as dt
from pathlib import Path

from depths.core.config import (
    LogProducerConfig,
    LogAggregatorConfig,
    DepthsLoggerOptions,
)
from depths.core.logger import DepthsLogger
```

## Producer knobs — `depths.core.config.LogProducerConfig`

The producer handles validation, normalization, and queueing before rows hit the aggregator.
The key fields are queue size and drop policy. Validation is enabled by default.

The key aspect of producer is to control how much dynamic memory do you want to allocate for the in-memory queue.

But it is also a tradeoff between persistence and throughput: larger in-memory queue gives higher throughput but also
increases odds of data loss if the server crashes. Similarly, keeping the in-memory queue too small applies backpressure frequently,
causing incoming signals to be dropped even before they hit the aggregator

So, think of producer as the conduit between the tap (incoming signals) and the swimming pool (aggregator)

```python
producer = LogProducerConfig(
    max_queue_size=2048,
    drop_policy="block",
    validate_required=True,
    validate_types=True,
    normalize_service_name=True,
    default_service_name="unknown",
)
```

## Aggregator knobs — `depths.core.config.LogAggregatorConfig`

The aggregator batches rows into typed Polars frames and appends to a Delta table.
The core controls are batch age (how long does a batch of data stays in-memory)
and row thresholds. Strict frames keep types stable.

So, essentially, tuning Aggregator is tuning a tradeoff between throughput and persistence.
If you increase the batch age, you are doing fewer disk writes so potentially higher throughput, but
then more data is in-memory: potential loss if the program terminates

Similarly, keeping it too low blows up file counts on the disk and makes the write process slow amortized, hence lower
throughput but higher persistence guarantee.

```python
aggregator = LogAggregatorConfig(
    max_age_s=0.5,
    min_batch_rows=100,
    max_batch_rows=1000,
    strict_df=True,
)
```

## Assemble `depths.core.config.DepthsLoggerOptions`

Options orchestrate startup, signal hooks, and shipper toggles. Signal handlers essentially tell the `DepthsLogger` how to
handle program terminations with as much grace as possible (flush to disk wherever possible). The auto start toggle tells `DepthsLogger`
to not require an explicit `DepthsLogger.start()` before starting to ingest telemetry.

The `DepthsLogger.start()` behind the scenes essentially gears up the producer to start handling incoming signals.
The `auto_start` toggle hides this behind the scenes for you. There can be instances where you would exact control when the logging should
start in the code, and for those scenarios, you would opt for manually controlling this (so, `auto_start=False` and a manual `DepthsLogger.start()` when a start is desired)

```python
options = DepthsLoggerOptions(
    auto_start=True,
    install_signal_handlers=False,
    atexit_hook=True,
    producer_config=producer,
    aggregator_config=aggregator,
)
```

## Initialize `depths.core.logger.DepthsLogger` with options

The logger prepares the instance, merges and persists options, and starts the aggregators.

```python
INSTANCE_ID = "custom_opts_demo"
INSTANCE_DIR = os.path.abspath("./depths_custom_opts")
INSTANCE_ROOT = os.path.join(INSTANCE_DIR, INSTANCE_ID)

logger = DepthsLogger(
    instance_id=INSTANCE_ID,
    instance_dir=INSTANCE_DIR,
    options=options,
)
```

## Ingest a sample batch

A tiny dataset with varying severities makes filtering obvious.

```python
PROJECT_ID = "opts_project"
SERVICE_NAME = "opts_service"
N = 600

now_ns = lambda: int(time.time() * 1_000_000_000)

accepted = 0
for i in range(N):
    sev_num = 13 if (i % 5 == 0) else 9
    sev_txt = "WARN" if sev_num >= 13 else "INFO"
    ok, _ = logger.ingest_log(
        {
            "project_id": PROJECT_ID,
            "service_name": SERVICE_NAME,
            "time_unix_nano": now_ns(),
            "severity_number": sev_num,
            "severity_text": sev_txt,
            "body": f"opts row {i}",
        }
    )
    if ok:
        accepted += 1
```

## Stop and flush

With an explicit `stop()`, we gracefully tell the `DepthsLogger` instance
to flush the remaining in-memory data to memory. The `auto` lets the age expiry gracefully come through.
You can also do `flush="none"` to immediately flush without waiting for batch age to expire.

The key difference is the `flush="none"` doesn't drain the non-aggregated signals in the Producer's queue. It simply wraps up the
queued disk write tasks and shutdowns.

```python
logger.stop(flush="auto")
```

## Verify persisted rows

Use the named read helper to pull a small projection.

```python
today = dt.datetime.utcnow().strftime("%Y-%m-%d")

rows = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["event_ts", "severity_text", "body", "service_name"],
    max_rows=5,
)

print(len(rows))
for r in rows:
    print(r)
```

## Verify options persistence (`options.json`)

Options are serialized in the instance configs. Load the file and check selected fields. The exact path is `<instance_root>/configs/options.json`.

```python
opts_path = Path(INSTANCE_ROOT, "configs", "options.json")
on_disk = json.loads(opts_path.read_text())

print(on_disk["producer_config"]["max_queue_size"])
print(on_disk["producer_config"]["drop_policy"])
print(on_disk["aggregator_config"]["max_age_s"])
print(on_disk["aggregator_config"]["max_batch_rows"])
```

## Recreate logger without options and verify reload

Construct a new logger pointing at the same instance. The saved options are merged in automatically.

```python
logger2 = DepthsLogger(
    instance_id=INSTANCE_ID,
    instance_dir=INSTANCE_DIR,
)

rows2 = logger2.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["event_ts", "severity_text", "body", "service_name"],
    max_rows=3,
)

print(len(rows2))
for r in rows2:
    print(r)
```

## Quick peek query (WARN+ with substring)

Filter helpers are pushed down and collected only at the end.

```python
q = logger2.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    severity_ge=13,
    body_like="opts",
    select=["event_ts", "severity_text", "body", "service_name"],
    max_rows=5,
    return_as="lazy",
)

print(q.collect())
```

## What just happened

* The producer enforced queueing and validation using `LogProducerConfig`
* The aggregator flushed on age or row thresholds using `LogAggregatorConfig`
* `DepthsLoggerOptions` persisted to `<instance_root>/configs/options.json` and were reloaded on the next run
* `read_logs` applied equality, substring, severity, and time-range filters with projection and row limits

## Wrap-up and next steps

* Keep these knobs small at first, then widen `max_batch_rows` or `max_age_s` to trade latency for throughput
* Move on to **Querying possibilities with Depths** to explore grouped and lazy reads
* When you are ready for object storage, try **S3 backups from scratch**


# Depths from scratch
Source: https://docs.depthsai.com/guides/depths-from-scratch

Create a DepthsLogger, ingest a small dataset, and run a basic query end to end.

Depths v0.1.1 can be used directly as a Python library. In this guide you will create a telemetry ingestor
using `DepthsLogger()` class, ingest a small set of log rows (which automatically get persisted on disk),
and run a basic query with filters and projection.

## What you will build

* A local instance directory for persisting telemetry signals on disk
* A `DepthsLogger()` object that automatically persists received telemetry signals on disk
* A tiny dataset of log rows with a few varying fields
* A verification query that returns either dicts or a lazy frame

## Prerequisites

* Python 3.12+
* `pip install depths`

## Imports and setup

We keep the instance id and directory explicit. Day boundaries are UTC. The logger will create directories and configs on first use.

```python
import os, time, datetime as dt
from depths.core.logger import DepthsLogger

INSTANCE_ID = "demo_from_scratch"
INSTANCE_DIR = os.path.abspath("./depths_from_scratch")
PROJECT_ID = "scratch_project"
SERVICE_NAME = "scratch_service"
N = 800
```

## Create the logger

`depths.core.logger.DepthsLogger` prepares today’s day folder, installs defaults, and starts aggregators by default.

```python
logger = DepthsLogger(instance_id=INSTANCE_ID, instance_dir=INSTANCE_DIR)
```

## Build a minimal dataset

Depths expects OTel-shaped rows for each table. For logs, the required fields are satisfied by providing `project_id` and a timestamp in nanoseconds; the rest is normalized by the producer. We vary severity and body to make filtering obvious.

```python
now_ns = lambda: int(time.time() * 1_000_000_000)

def make_row(i: int) -> dict:
    sev_num = 13 if (i % 7 == 0) else 9
    sev_txt = "WARN" if sev_num >= 13 else "INFO"
    return {
        "project_id": PROJECT_ID,
        "service_name": SERVICE_NAME,
        "time_unix_nano": now_ns(),
        "severity_number": sev_num,
        "severity_text": sev_txt,
        "body": f"hello depths {i}"
    }

rows = [make_row(i) for i in range(N)]
```

## Ingest synchronously

The logger validates and enqueues each row. Aggregators batch and persist to Delta in the background.

```python
accepted = 0
for r in rows:
    ok, reason = logger.ingest_log(r)
    if ok:
        accepted += 1
```

## Stop and flush

`stop(flush="auto")` performs a bounded, quick flush of pending batches to the local disk.

```python
logger.stop(flush="auto")
```

## Basic verification as dicts

`depths.core.logger.DepthsLogger.read_logs` composes a lazy plan with pushdown filters. By default it materializes to a list of dicts. We select a few columns and limit to five latest rows.

```python
today = dt.datetime.now(dt.UTC).strftime("%Y-%m-%d")

rows = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["event_ts", "severity_text", "body", "service_name"],
    max_rows=5
)

print(len(rows), "rows")
for r in rows:
    print(r)
```

## The same query as a lazy frame

Set `return_as="lazy"` to keep everything lazy for downstream operations. Collect only when you are ready.

```python
q = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    severity_ge=13,
    body_like="hello",
    select=["event_ts", "severity_text", "body", "service_name"],
    max_rows=5,
    return_as="lazy"
)

print(q.collect())
```

## What just happened

* `depths.core.logger.DepthsLogger` created our telemetry ingestor, with all the config and empty storage tables on the disk.
* Each call to `ingest_log` validated the row against `depths.core.schema.LOG_SCHEMA` via `depths.core.producer.LogProducer`,
  ensuring that only correct OpenTelemtry compatible rows get stored.
* Behind the scenes, `depths.core.aggregator.LogAggregator` batched rows into typed Polars frames and appended to the logs Delta table under today’s UTC day,
  giving us the disk persistence.
* `read_logs` built a lazy plan with equality, substring, and time-range predicates for efficient querying.
  We can then view these result as either a list of dictionaries, a polars dataframe or further perform ops on lazyframe (useful for operations like counts and other statistical options).
  We recommend reading about the Polars package and LazyFrame concept. Since we don't expect every Depths user to be familiar with Polars,
  the default query function response is a list of dictionary objects.

## Where to go next

* Tune behavior with `depths.core.config.DepthsLoggerOptions` and producer+aggregator configs in **Customizing Depths**
* Explore more query patterns (projection, limits, groups) in **Querying possibilities with Depths**
* Add S3 shipping and read sealed days from object storage in **S3 backups from scratch**


# Querying possibilities with Depths
Source: https://docs.depthsai.com/guides/depths-querying

Explore read_logs as LazyFrame or dicts, apply predicates, project columns, and group results.

# Querying possibilities with Depths

Depths v0.1.1 lets you query persisted data through helpers on `depths.core.logger.DepthsLogger`.
This guide focuses on logs and shows four patterns: return rows as Python dicts, keep results lazy for Polars transforms, apply named predicates, and group results.
This rich flexibility allows you to perform efficient queries on your data, regardless of whether it is stored on disk or on S3.

## What you will build

* A local dataset written via `DepthsLogger`
* A “rows as dicts” read for quick printing
* A LazyFrame read for chained transforms and a final `collect`
* Examples of named predicates and projection
* A grouped severity summary

## Prerequisites

* Python 3.12+
* `pip install depths`

## Imports and setup

We’ll set up a fresh instance and generate a small log dataset with varying severities and bodies. We capture today’s date as a **timezone-aware** UTC string so day filters are precise.

```python
import os, time, datetime as dt
import polars as pl
from depths.core.logger import DepthsLogger

INSTANCE_ID = "querying_demo"
INSTANCE_DIR = os.path.abspath("./depths_querying_demo")
PROJECT_ID = "q_project"
SERVICE_NAME = "q_service"
N = 900
```

## Create the logger and write a sample dataset

As shown in previous guides, we construct a logger, create minimal OTel-shaped log rows, ingest them,
and then stop with an automatic flush to persist batches.

```python
logger = DepthsLogger(instance_id=INSTANCE_ID, instance_dir=INSTANCE_DIR)

now_ns = lambda: int(time.time() * 1_000_000_000)

def make_row(i: int) -> dict:
    sev_num = 17 if (i % 10 == 0) else (13 if (i % 4 == 0) else 9)
    sev_txt = "ERROR" if sev_num >= 17 else ("WARN" if sev_num >= 13 else "INFO")
    return {
        "project_id": PROJECT_ID,
        "service_name": SERVICE_NAME,
        "time_unix_nano": now_ns(),
        "severity_number": sev_num,
        "severity_text": sev_txt,
        "body": f"searchable message {i}"
    }

accepted = 0
for i in range(N):
    ok, _ = logger.ingest_log(make_row(i))
    if ok:
        accepted += 1

logger.stop(flush="auto")
```

## A basic read as dicts

We query today’s data using an aware UTC day string, select a few columns, and cap results with max\_rows.
Returning dicts is ergonomic for quick inspection and printing.

```python
today = dt.datetime.now(dt.UTC).strftime("%Y-%m-%d")

rows = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["event_ts", "severity_text", "body"],
    max_rows=5
)

print(len(rows))
for r in rows:
    print(r)
```

## Keep it lazy for downstream transforms

Ask Depths for a LazyFrame to build a transform pipeline.
Here we add a string-length column using str.len\_chars() on the body column and
then collect() to realize results. with\_columns adds or replaces columns in a lazy plan.

LazyFrame option allows us to prune the data down to most relevant segments, minimizing disk/S3 reads.

```python
lf = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["event_ts", "severity_text", "body"],
    return_as="lazy"
)

out = (
    lf
    .with_columns(pl.col("body").str.len_chars().alias("body_len"))
    .limit(5)
    .collect()
)

print(out)
```

## Named predicates: time, equality, severity, substring

Depths read helpers expose common predicates that push down work.
Here we constrain by day, project, and service; filter by minimum severity;
and search for a substring in body via the `body_like` predicate.
We also project a small set of columns and limit rows.

```python
subset = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    severity_ge=13,
    body_like="searchable",
    select=["event_ts", "severity_text", "body"],
    max_rows=10
)

for r in subset:
    print(r["severity_text"], r["body"])
```

## Group-by summary on a lazy plan

We compute a quick severity distribution.
In Polars, call group\_by then agg with expression and sort the result.
We collect only once at the end.

```python
lf2 = logger.read_logs(
    date_from=today,
    date_to=today,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["severity_text"],
    return_as="lazy"
)

summary = (
    lf2
    .group_by("severity_text")
    .agg(pl.len().alias("count"))
    .sort("count", descending=True)
    .collect()
)

print(summary)
```

## Storage selector overview

`read_logs` accepts a `storage` selector. Use `local` to force local Delta tables, `s3` to force object storage when S3 is configured, or `auto` to let Depths choose.
This guide used local storage only. The default value is `auto` so you don't have to worry about explicitly selecting `s3`
when you start backing up the telemetry on an S3 bucket, the queries would run seamlessly.

## What you learned

* Returning rows as dicts fits simple scripts and prints
* Returning a lazy frame lets you add Polars transforms and collect once
* Named predicates cut input size and speed up reads
* Group-by on a lazy plan gives quick summaries without building a separate ETL


# S3 backups from scratch
Source: https://docs.depthsai.com/guides/s3-backups

Load S3 config from env, ingest to a past day, seal and ship, and query back from S3.

# S3 backups from scratch

Depths v0.1.1 can seal each UTC day of data and ship it to S3 or an S3-compatible endpoint.
In this guide you will configure S3 via environment, ingest logs into a past day for determinism, run a synchronous ship,
and then read back from S3.

Please note that this process happens automatically behind the scenes, we are
manually doing it here for demonstrative purposes.

## What you will build

* A local instance with a small dataset written into “yesterday”
* A one-shot ship that seals, uploads, verifies row counts, and cleans local copies
* A verification query that reads sealed data from S3

## Prerequisites — S3 environment variables

Set the environment before starting Python.

**Required**

* `S3_BUCKET`
* `AWS_ACCESS_KEY_ID` or `S3_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY` or `S3_SECRET_KEY` or `S3_SECRET_ACCESS_KEY`
* `AWS_REGION` or `S3_REGION`
* `AWS_ENDPOINT_URL` or `S3_URL`

**Optional**

* `S3_PREFIX`
* `AWS_SESSION_TOKEN`

## Imports and setup

We keep the instance explicit and generate a small dataset with stable timestamps in the target day.

```python
import os, time, datetime as dt
from pathlib import Path
import polars as pl
from depths.core.logger import DepthsLogger
from depths.core.config import S3Config, DepthsLoggerOptions

from dotenv import load_dotenv

load_dotenv()

INSTANCE_ID = "s3_from_scratch"
INSTANCE_DIR = os.path.abspath("./depths_s3_from_scratch")
PROJECT_ID = "s3_project"
SERVICE_NAME = "s3_service"
N = 600
```

For convenience, we loaded the environment from .env file using the python-dotenv package.
The `S3Config.from_env()` reads the loaded environment, validates configuration and provides reader and upload options internally.

Note that `from_env()` will also pick the environment variables set via terminal, just that
load\_dotenv provides convenience of loading from a .env file

## Load S3 config and initialize the logger

We ensure S3 is present, then start a logger. The logger prepares directories, schemas, and background services.

Note that since we are manually controlling S3 backup here, we need to toggle
a setting called `shipping_enabled` as false, to ensure that we manually control the S3 shipping behavior.

```python
s3 = S3Config.from_env()
logger = DepthsLogger(
    instance_id=INSTANCE_ID,
    instance_dir=INSTANCE_DIR,
    s3=s3,
    options=DepthsLoggerOptions(
        shipper_enabled=False  # we’re driving shipping explicitly
    ),
)
```

If the environment is incomplete, shipping is disabled but an attempt to ship to S3 will throw errors.
Therefore, before proceeding forward, you can quickly do a `print(s3)` to check if config was correctly loaded.

Reads can still use local storage.

## Target a past day for deterministic shipping

We ingest into “yesterday” by two steps: compute a base timestamp at UTC midnight yesterday, and retarget the logs table to yesterday’s local path so all batches land under the same day.
Retargeting the aggregator ensures files are created under the desired day directory even if the process runs today.

We are doing this manually purely for demonstrative purposes. In direct usage, the `DepthsLogger` automatically does such
rollovers.

```python
day = (dt.datetime.now(dt.UTC).date() - dt.timedelta(days=1)).strftime("%Y-%m-%d")
midnight_y = dt.datetime.combine(dt.datetime.now(dt.UTC).date() - dt.timedelta(days=1), dt.time(0, 0, 0), tzinfo=dt.UTC)
base_ns = int(midnight_y.timestamp() * 1_000_000_000)

instance_root = os.path.join(INSTANCE_DIR, INSTANCE_ID)

# Toggling internals to manually create and handle `yesterday`'s data
day_root = DepthsLogger._local_day_path(Path(instance_root), day)
Path(day_root).mkdir(parents=True, exist_ok=True)
logs_path = DepthsLogger._otel_table_path(day_root, "logs")
logger._aggs["logs"].retarget_table_path(logs_path, initialize=True)
```

## Ingest a batch into yesterday

We vary severity and body so the S3 read is easy to spot. Timestamps advance by 1 ms to stay in order.

```python
accepted = 0
for i in range(N):
    sev_num = 13 if (i % 6 == 0) else 9
    sev_txt = "WARN" if sev_num >= 13 else "INFO"
    ok, _ = logger.ingest_log(
        {
            "project_id": PROJECT_ID,
            "service_name": SERVICE_NAME,
            "schema_version": 1,
            "scope_name": "s3-guide",
            "scope_version": "v0.1.1",
            "time_unix_nano": base_ns + (i * 1_000_000),
            "observed_time_unix_nano": base_ns + (i * 1_000_000),
            "severity_text": sev_txt,
            "severity_number": sev_num,
            "body": f"s3 demo row {i}",
        }
    )
    if ok:
        accepted += 1

logger.stop(flush="auto")
```

## Seal and ship now

We call a synchronous ship. It seals yesterday, uploads to S3, verifies remote row counts, and cleans local copies if verification passes.
The return value is a compact status summary of the process.

Once again, this shipping happens automatically behind the scenes and is generally not to be controlled manually.

```python
result = logger.ship_now(day)
print(result)
```

To prevent data corruption, current day is not shippable in `DepthsLogger`.
Past days are allowed. The summary includes per-table counts and an overall status.

## Read back from S3

We run two reads: a small dict sample and a LazyFrame for a quick severity rollup. The `storage="s3"` selector forces object storage.

```python
rows = logger.read_logs(
    date_from=day,
    date_to=day,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["event_ts", "severity_text", "body"],
    max_rows=5,
    storage="s3"
)

print(len(rows))
for r in rows:
    print(r)

lf = logger.read_logs(
    date_from=day,
    date_to=day,
    project_id=PROJECT_ID,
    service_name=SERVICE_NAME,
    select=["severity_text"],
    storage="s3",
    return_as="lazy"
)

summary = (
    lf
    .group_by("severity_text")
    .agg(pl.len().alias("count"))
    .sort("count", descending=True)
    .collect()
)

print(summary)
```

The querying experience is identical to what we have seen so far.
The design choice ensures that your querying DX stays agnostic of whether
the logs reside locally or on S3.

## What you learned

* How to configure S3 for Depths using environment variables
* How to write into a past day by retargeting the logs table and using UTC timestamps
* How to run a one-shot ship and inspect its summary
* How to read sealed data from S3 as dicts or as a LazyFrame for quick summaries


# Quickstart - Experience depths in 100 seconds
Source: https://docs.depthsai.com/quickstart

Start an OTLP/HTTP server with the CLI and send your first signal.

Depths v0.1.1 ships a CLI that boots a local OTLP/HTTP server (FastAPI) ready to receive traces, logs, and metrics at the standard `/v1/*` endpoints.

## Install (OS-agnostic)

```bash
# Base install, server accepts OTLP JSON bodies
pip install depths

# If you want to accept OTLP protobuf bodies as well
pip install "depths[proto]"
```

## Start the server (OS-agnostic)

```bash
# 1) Create an instance with default scaffold (configs, tables, etc.)
depths init 

# 2) Start the uvicorn server for the default instance (binds 0.0.0.0:4318 by default)
depths start
```

Depths is by design multi-tenant at a project level. We define an instance as a particularly configured
logger. Under an instance, you can store telemetry for multiple projects, just that each of them will share the same
configuration for the ingestion server.

After `start`, the server exposes:

* `GET /healthz` (liveness + minimal metrics)
* `POST /v1/traces`  (OTLP/HTTP JSON or protobuf)
* `POST /v1/logs`
* `POST /v1/metrics`
* `GET /api/spans`, `GET /api/logs`, `GET /api/metrics/points`, `GET /api/metrics/hist` (simple read APIs)

## Send your first event

### Option A: point an OTel SDK or Collector at Depths

Set your exporter endpoint to the server:

To update that command for Mintlify's tabbed code blocks, you need to use the platform-specific syntax for setting an environment variable.

Here is the correct Mintlify **`<CodeGroup>`** component:

<CodeGroup>
  ```bash macOS/Linux (Bash/Zsh)
  export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
  # or use the signal-specific endpoints if you prefer:
  # http://localhost:4318/v1/traces
  # http://localhost:4318/v1/logs
  # http://localhost:4318/v1/metrics
  ```

  ```powershell Windows (PowerShell)
  $env:OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
  # or use the signal-specific endpoints if you prefer:
  # http://localhost:4318/v1/traces
  # http://localhost:4318/v1/logs
  # http://localhost:4318/v1/metrics
  ```
</CodeGroup>

***

### Option B: send a minimal trace with cURL

Save this as `span.json` in the directory where you ran `depths init`. This is a toy payload:

```json
{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "test-with-curl" } }
        ]
      },
      "scopeSpans": [
        {
          "scope": { "name": "manual-test" },
          "spans": [
            {
              "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
              "spanId": "3c191d03fa8be065",
              "name": "spanitron",
              "kind": 2,
              "droppedAttributesCount": 0,
              "events": [],
              "droppedEventsCount": 0,
              "status": { "code": 1 }
            }
          ]
        }
      ]
    }
  ]
}
```

Send it to the Depths server:

<CodeGroup>
  ```bash macOS/Linux
  curl -i http://localhost:4318/v1/traces \
    -X POST \
    -H "Content-Type: application/json" \
    -d @span.json
  ```

  ```powershell Windows (PowerShell)
  # Use curl.exe in a single line (no backslash for continuation)
  curl.exe -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d "@span.json"
  ```
</CodeGroup>

## Verify ingestion

First, we can check the health of the server using `status` command.
This command provides a comprehensive single-pane snapshot of the telemetry ingestion server.

```bash
depths status
```

We can also take a peak at the most recent rows in our telemetry tables, persisted locally.

```bash
# Quick terminal peek at recent rows
depths view --table spans --rows 5
```

## Stop the server

```bash
# Clean shutdown
depths stop
```

With that, you have experienced a purely pythonic OTel native telemetry ingestion server.


# S3 Configuration
Source: https://docs.depthsai.com/s3-configuration

Configure S3 with environment variables, start the server, and query sealed days from S3 using the CLI.

# S3 Configuration

Depths v0.1.1 can seal each UTC day of data and ship it to S3 or any S3-compatible store. S3-backup is optional : local-only works fine. When S3 is set by environment variables, the server ships sealed days and verifies row counts. You can then read sealed days from S3. The querying experience
via the query endpoints stays identical, allowing seamless analysis over local and s3-backed telemetry.

## When to enable S3

* Durable storage for historical days
* Read sealed days from object storage instead of the ingest box
* S3-compatible endpoints (MinIO, DigitalOcean Spaces, etc.)

## Environment variables

Set these before you run `depths init` and `depths start`.

### Required

| Variable                                                             | Meaning                                                              |
| -------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `S3_BUCKET`                                                          | Bucket name                                                          |
| `AWS_ACCESS_KEY_ID` or `S3_ACCESS_KEY_ID`                            | Access key                                                           |
| `AWS_SECRET_ACCESS_KEY` or `S3_SECRET_KEY` or `S3_SECRET_ACCESS_KEY` | Secret                                                               |
| `AWS_REGION` or `S3_REGION`                                          | Region (leave empty for some S3-compatible vendors)                  |
| `AWS_ENDPOINT_URL` or `S3_URL`                                       | Endpoint URL, e.g. `https://s3.amazonaws.com` or `http://minio:9000` |

### Optional

| Variable            | Meaning                                |
| ------------------- | -------------------------------------- |
| `S3_PREFIX`         | Key prefix, e.g. `depths-prod`         |
| `AWS_SESSION_TOKEN` | Session token if using temporary creds |

> For S3-compatible endpoints using `http://`, reads use `AWS_ALLOW_HTTP=true` internally. Prefer TLS in production.

## Set variables

<CodeGroup>
  ```bash macOS/Linux
  export S3_BUCKET="my-bucket"
  export AWS_ACCESS_KEY_ID="…"
  export AWS_SECRET_ACCESS_KEY="…"
  export AWS_REGION="ap-south-1"
  export AWS_ENDPOINT_URL="https://s3.amazonaws.com"
  # optional
  export S3_PREFIX="depths-prod"
  export AWS_SESSION_TOKEN=""
  ```

  ```powershell Windows
  $env:S3_BUCKET="my-bucket"
  $env:AWS_ACCESS_KEY_ID="…"
  $env:AWS_SECRET_ACCESS_KEY="…"
  $env:AWS_REGION="ap-south-1"
  $env:AWS_ENDPOINT_URL="https://s3.amazonaws.com"
  # optional
  $env:S3_PREFIX="depths-prod"
  $env:AWS_SESSION_TOKEN=""
  ```
</CodeGroup>

## Start with shipping enabled

<CodeGroup>
  ```bash macOS/Linux
  depths init
  depths start
  ```

  ```powershell Windows
  depths init
  depths start
  ```
</CodeGroup>

What happens:

1. During the day, rows are appended to local Delta tables.
2. On UTC day rollover, the server seals the day, uploads the tables to S3, verifies row counts, then cleans local copies when verification passes.

## Bucket layout

Days are stored under:

```
s3://<bucket>/<prefix>/<instance_id>/days/<YYYY-MM-DD>/
```

Each day contains six Delta tables under `otel/`:
`spans`, `span_events`, `span_links`, `logs`, `metrics_points`, `metrics_hist`.

## Verify shipping

Health:

```bash
curl -s http://localhost:4318/healthz | jq
```

Index (recent lines):

<CodeGroup>
  ```bash macOS/Linux
  tail -n 20 ./depths_data/default/index/days.jsonl
  ```

  ```powershell Windows
  Get-Content .\depths_data\default\index\days.jsonl -Tail 20
  ```
</CodeGroup>

You should see the phases per day: `sealed` → `uploaded` → `verified` → `cleaned` with row counts.

## Read from S3

Use the CLI to read directly from S3, or let it auto-select.

<CodeGroup>
  ```bash macOS/Linux
  depths view -t spans -S s3
  depths view -t spans -S auto
  ```

  ```powershell Windows
  depths view -t spans -S s3
  depths view -t spans -S auto
  ```
</CodeGroup>

The HTTP query endpoints also accept a `storage` parameter (`auto`, `local`, `s3`) if you prefer API access.

## Common pitfalls

* **Partial env** : any missing required variable disables shipping; set env before `depths start`.
* **Endpoint** : `http://` works for compatible stores; prefer `https://` for production.
* **Permissions** : the key needs `PutObject`, `List`, and `Get` in the bucket and prefix.