Health and Metrics

Asset Core daemons expose health endpoints for monitoring and Prometheus metrics for observability.

Overview

Both write and read daemons provide:

Health endpoint: JSON status for load balancers and monitoring
Metrics endpoint: Prometheus-format metrics for dashboards and alerting

Structure

Health Endpoints

Daemon	Endpoint	Description
Write	`GET /v1/health`	Commit log status and lag
Read	`GET /v1/health`	Projection freshness

Metrics Endpoints

Daemon	Endpoint	Format
Write	`GET /v1/metrics`	Prometheus text
Read	`GET /v1/metrics`	Prometheus text

Fields

Write Daemon Health

{
  "status": "healthy",
  "commit_log": {
    "end_seq": 1000,
    "checkpoint_seq": 1000,
    "lag": 0,
    "driver": {
      "name": "file",
      "path": "/var/lib/assetcore/commit_log.log"
    }
  }
}

Field	Description
`status`	`healthy` or `degraded`
`commit_log.end_seq`	Latest committed sequence
`commit_log.checkpoint_seq`	Latest checkpointed sequence
`commit_log.lag`	Difference (should be 0 or near 0)

Read Daemon Health

{
  "status": "healthy",
  "freshness": {
    "checkpoint_seq": 998,
    "commit_log_seq": 1000,
    "lag": 2
  }
}

Field	Description
`freshness.checkpoint_seq`	Sequences applied to projection
`freshness.commit_log_seq`	Latest in commit log
`freshness.lag`	How far behind the read daemon is

Key Metrics

Write Daemon

Metric	Type	Description
`assetcore_write_requests_total`	Counter	Total requests by route and status
`assetcore_write_request_duration_seconds`	Histogram	Request latency
`assetcore_write_commit_duration_seconds`	Histogram	Commit-specific latency
`assetcore_write_ingress_queue_depth`	Gauge	Items waiting in queue
`assetcore_write_ingress_inflight`	Gauge	Requests being processed
`assetcore_write_commit_log_end_seq`	Gauge	Latest committed sequence
`assetcore_write_commit_log_lag`	Gauge	Checkpoint lag

Read Daemon

Metric	Type	Description
`assetcore_read_requests_total`	Counter	Total requests by route
`assetcore_read_request_duration_seconds`	Histogram	Request latency
`assetcore_read_checkpoint_seq`	Gauge	Applied sequence
`assetcore_read_freshness_lag`	Gauge	Events behind commit log
`assetcore_read_snapshot_publish_duration_seconds`	Histogram	Snapshot publish time

Examples

Check Write Health

curl http://localhost:8080/v1/health | jq .

Check Read Freshness

curl http://localhost:8081/v1/read/freshness | jq .

Scrape Metrics

Add to your Prometheus configuration:

scrape_configs:
  - job_name: 'assetcore-write'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/v1/metrics'

  - job_name: 'assetcore-read'
    static_configs:
      - targets: ['localhost:8081']
    metrics_path: '/v1/metrics'

Example Prometheus Queries

Request rate:

rate(assetcore_write_requests_total[5m])

P99 latency:

histogram_quantile(0.99, rate(assetcore_write_request_duration_seconds_bucket[5m]))

Read daemon lag:

assetcore_read_freshness_lag

Error rate:

rate(assetcore_write_requests_total{status!="200"}[5m])

Example Alerts

groups:
  - name: assetcore
    rules:
      - alert: HighReadLag
        expr: assetcore_read_freshness_lag > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Read daemon is behind commit log"

      - alert: WriteQueueFull
        expr: assetcore_write_ingress_queue_depth > 900
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Write queue approaching capacity"

Deployment Basics - Running daemons
Freshness and Replay - Understanding lag
HTTP API - Endpoint reference