Observability

Dispatch provides two layers of observability:

System-wide metrics via observability.MetricsExtension — counters for every lifecycle event.
Per-job tracing and metrics via middleware.Tracing() and middleware.Metrics() — spans and histograms per execution.

System-wide metrics (extension)

import "github.com/xraph/dispatch/observability"

metricsExt := observability.NewMetricsExtension(
    observability.WithTracerProvider(tp),
    observability.WithMetricFactory(factory),
)

eng := engine.Build(d,
    engine.WithExtension(metricsExt),
)

Metrics recorded

Event	Metric	Labels
Job enqueued	`dispatch_jobs_enqueued_total`	`name`, `queue`
Job completed	`dispatch_jobs_completed_total`	`name`, `queue`
Job failed	`dispatch_jobs_failed_total`	`name`, `queue`
Job retrying	`dispatch_jobs_retried_total`	`name`, `queue`
Job DLQ	`dispatch_jobs_dlq_total`	`name`, `queue`
Workflow started	`dispatch_workflows_started_total`	`name`
Workflow completed	`dispatch_workflows_completed_total`	`name`
Workflow failed	`dispatch_workflows_failed_total`	`name`
Cron fired	`dispatch_crons_fired_total`	`name`

Per-job tracing (middleware)

Add middleware.Tracing() to the engine:

eng := engine.Build(d,
    engine.WithMiddleware(middleware.Tracing()),
    engine.WithTracerProvider(tp),
)

Each job execution creates an OpenTelemetry span with attributes:

dispatch.job.name
dispatch.job.queue
dispatch.job.id
dispatch.job.retry_count
error (if the job failed)

Per-job metrics (middleware)

Add middleware.Metrics() to the engine:

eng := engine.Build(d,
    engine.WithMiddleware(middleware.Metrics()),
    engine.WithMetricFactory(factory),
)

Records a histogram dispatch_job_duration_seconds labelled by name, queue, and status (completed / failed).

Combining both layers

For full observability, use both:

eng := engine.Build(d,
    engine.WithExtension(observability.NewMetricsExtension(
        observability.WithTracerProvider(tp),
    )),
    engine.WithMiddleware(middleware.Tracing()),
    engine.WithMiddleware(middleware.Metrics()),
    engine.WithTracerProvider(tp),
)