Last9 Go Agent

Last9 Go Agent provides distributed traces, runtime metrics, and log-trace correlation. One call to agent.Start() replaces the OpenTelemetry SDK setup boilerplate, and each framework integration is a drop-in replacement for the standard constructor.

This is the SDK path: works anywhere Go runs — VMs, bare metal, Lambda, local development. If you're on Kubernetes and want zero-code instrumentation, the eBPF operator is the right tool. Both can coexist: eBPF for base HTTP and DB coverage, this SDK for custom business spans.

Table of Contents

• Quick Start
• Framework Support
• Database Support
• ORM Support (GORM)
• MongoDB
• Redis
• Kafka
• HTTP Client
• Log-Trace Correlation
• Metrics
• Route Exclusion
• HTTP Body Capture
• Code Call-Site Attributes
• Configuration
• Testing

Quick Start

go get github.com/last9/go-agent

Set your environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="<your last9 otlp endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your last9 token>"
export OTEL_SERVICE_NAME="my-service"

Add two lines to main.go:

func main() {
    agent.Start()
    defer agent.Shutdown()

    // your application code, unchanged
}

Traces and metrics now export to your configured endpoint.

Framework Support

Every web framework integration is a drop-in replacement for the standard constructor. You change the import and the instantiation call — nothing else in your application changes.

net/http

import nethttpagent "github.com/last9/go-agent/instrumentation/nethttp"

mux := nethttpagent.NewServeMux()
mux.HandleFunc("/users", usersHandler)
http.ListenAndServe(":8080", mux)

// Or wrap an existing handler
http.ListenAndServe(":8080", nethttpagent.WrapHandler(existingMux))

// Or wrap individual handlers
http.Handle("/ping", nethttpagent.Handler(pingHandler, "/ping"))

// Or use the drop-in ListenAndServe
nethttpagent.ListenAndServe(":8080", mux)

Gin

import ginagent "github.com/last9/go-agent/instrumentation/gin"

r := ginagent.Default()   // includes logging & recovery
r := ginagent.New()       // minimal

// Or add to an existing router
r := gin.New()
r.Use(ginagent.Middleware())

Chi

import chiagent "github.com/last9/go-agent/instrumentation/chi"

r := chiagent.New()

// Or instrument an existing router — add AFTER defining routes
// so the middleware can capture the matched route pattern
r := chi.NewRouter()
r.Get("/users/{id}", handler)
chiagent.Use(r)

Echo

import echoagent "github.com/last9/go-agent/instrumentation/echo"

e := echoagent.New()

Gorilla Mux

import gorillaagent "github.com/last9/go-agent/instrumentation/gorilla"

r := gorillaagent.NewRouter()
r.HandleFunc("/ping", handler).Methods("GET")

gRPC

import grpcagent "github.com/last9/go-agent/instrumentation/grpc"

// Server
lis, _ := net.Listen("tcp", ":50051")
s := grpcagent.NewServer()
pb.RegisterGreeterServer(s, &server{})
s.Serve(lis)

// Client
conn, _ := grpc.NewClient("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpcagent.NewClientDialOption(),
)

gRPC-Gateway

import (
    "github.com/last9/go-agent"
    "github.com/last9/go-agent/instrumentation/grpcgateway"
)

grpcServer := grpcgateway.NewGrpcServer()
pb.RegisterYourServiceServer(grpcServer, &server{})

gwMux := grpcgateway.NewGatewayMux()

conn, _ := grpc.NewClient("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpcgateway.NewDialOption(),
)

httpMux := http.NewServeMux()
httpMux.Handle("/", gwMux)
http.ListenAndServe(":8080", grpcgateway.WrapHTTPMux(httpMux, "my-gateway"))

The gRPC-Gateway HTTP layer automatically excludes high-volume infrastructure paths from tracing. Default exclusions: /health, /healthz, /readyz, /livez, /metrics (exact match), and prefix matches for /actuator/ and /eureka/apps/. User-configured LAST9_EXCLUDED_PATHS and related variables apply on top of these defaults.

fasthttp

import fasthttpagent "github.com/last9/go-agent/instrumentation/fasthttp"

handler := func(ctx *fasthttp.RequestCtx) {
    ctx.WriteString("hello")
}
fasthttp.ListenAndServe(":8080", fasthttpagent.Middleware(handler))

// Access the active span context inside a handler
func myHandler(ctx *fasthttp.RequestCtx) {
    otelCtx := fasthttpagent.ContextFromRequest(ctx)
    _, span := otel.Tracer("my-service").Start(otelCtx, "my-op")
    defer span.End()
}

Iris

import irisagent "github.com/last9/go-agent/instrumentation/iris"

app := irisagent.New()   // drop-in for iris.New()
app.Get("/ping", func(ctx iris.Context) {
    ctx.WriteString("pong")
})
app.Listen(":8080")

// Or add to an existing application
app := iris.New()
app.Use(irisagent.Middleware())

Beego

import beegoagent "github.com/last9/go-agent/instrumentation/beego"

app := beegoagent.New()
app.Get("/ping", func(ctx *context.Context) {
    ctx.Output.Body([]byte("pong"))
})
app.Run()

// Or add to an existing server
app := web.NewHttpSever()
app.InsertFilterChain("/*", beegoagent.Middleware())

Database Support

SQL tracing uses database.Open() instead of sql.Open(). Every query gets a span. Connection pool metrics are collected automatically. The agent extracts host, port, user, and database name from your DSN and stamps them onto spans as OTel semantic convention attributes.

import "github.com/last9/go-agent/integrations/database"

db, err := database.Open(database.Config{
    DriverName:   "postgres",
    DSN:          "postgres://user:pass@localhost/mydb",
    DatabaseName: "mydb",
})
defer db.Close()

// Use db normally — all queries are automatically traced
rows, err := db.Query("SELECT * FROM users")

// Panic on error variant for quick initialization
db := database.MustOpen(database.Config{
    DriverName:   "postgres",
    DSN:          os.Getenv("DATABASE_URL"),
    DatabaseName: "mydb",
})

Supported drivers: postgres, pgx, mysql, sqlite, sqlite3.

Manual Wrapper Spans

When you create spans around repository methods, they won't inherit the connection attributes auto-generated by the SQL layer. Use ParseDSNAttributes to stamp them yourself:

import "github.com/last9/go-agent/integrations/database"

func (r *UserRepo) FindByID(ctx context.Context, id int) (*User, error) {
    ctx, span := tracer.Start(ctx, "FindByID")
    defer span.End()

    span.SetAttributes(database.ParseDSNAttributes(r.dsn, "mysql")...)

    // ... run query
}

ORM Support (GORM)

For GORM v2, use the official gorm.io/plugin/opentelemetry tracing plugin directly. It is maintained by the GORM team, ships current OpenTelemetry semantic conventions, and emits connection-pool metrics. go-agent does not wrap or replace it.

Pair the upstream plugin with database.Open from this package and you get two-layer tracing per query: the GORM plugin emits an OTel span for the ORM operation, and integrations/database emits a child span for the wire-level SQL.

import (
    agent "github.com/last9/go-agent"
    "github.com/last9/go-agent/integrations/database"
    _ "github.com/lib/pq"
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "gorm.io/plugin/opentelemetry/tracing"
)

func main() {
    agent.Start()
    defer agent.Shutdown()

    sqlDB, err := database.Open(database.Config{
        DriverName: "postgres",
        DSN:        os.Getenv("DATABASE_URL"),
    })
    if err != nil {
        log.Fatal(err)
    }
    defer sqlDB.Close()

    db, err := gorm.Open(postgres.New(postgres.Config{Conn: sqlDB}), &gorm.Config{})
    if err != nil {
        log.Fatal(err)
    }

    if err := db.Use(tracing.NewPlugin()); err != nil {
        log.Fatal(err)
    }

    // Use db normally — pass request contexts via db.WithContext(ctx).
}

Trace shape:

Gin / HTTP server span
  └─ select users  (gorm.io/plugin/opentelemetry — db.system.name, db.query.text, db.collection.name)
        └─ SELECT  (integrations/database — wire-level SQL, rows affected)

A complete docker-compose example with Postgres lives at opentelemetry-examples/go/gorm.

MongoDB

import mongoagent "github.com/last9/go-agent/integrations/mongodb"

client, err := mongoagent.NewClient(mongoagent.Config{
    URI: "mongodb://localhost:27017/mydb",
})
defer client.Disconnect(context.Background())

col := client.Database("mydb").Collection("users")
col.InsertOne(ctx, bson.M{"name": "Alice"})

Or instrument an existing options struct:

opts := options.Client().ApplyURI(os.Getenv("MONGO_URI"))
client, err := mongoagent.Instrument(opts)

All CRUD operations, aggregation pipelines, and index operations are traced. Connection housekeeping (hello, ping, isMaster) and auth handshakes are silently skipped.

Redis

import redisagent "github.com/last9/go-agent/integrations/redis"

rdb := redisagent.NewClient(&redis.Options{
    Addr: "localhost:6379",
})

// All commands are automatically traced
err := rdb.Set(ctx, "key", "value", 0).Err()
val, err := rdb.Get(ctx, "key").Result()

// Cluster support
rdb := redisagent.NewClusterClient(&redis.ClusterOptions{
    Addrs: []string{":7000", ":7001", ":7002"},
})

Kafka

Producer

import kafkaagent "github.com/last9/go-agent/integrations/kafka"

producer, err := kafkaagent.NewSyncProducer(kafkaagent.ProducerConfig{
    Brokers: []string{"localhost:9092"},
})
defer producer.Close()

partition, offset, err := producer.SendMessage(ctx, &sarama.ProducerMessage{
    Topic: "my-topic",
    Value: sarama.StringEncoder("Hello Kafka"),
})

Consumer Group

consumer, err := kafkaagent.NewConsumerGroup(kafkaagent.ConsumerConfig{
    Brokers: []string{"localhost:9092"},
    GroupID: "my-consumer-group",
})
defer consumer.Close()

handler := kafkaagent.WrapConsumerGroupHandler(&MyHandler{})
consumer.Consume(ctx, []string{"my-topic"}, handler)

Trace context is propagated from producer to consumer automatically. When you receive a message, its context already carries the producer's span as parent.

HTTP Client

import (
    "net/http/httptrace"
    httpagent "github.com/last9/go-agent/integrations/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace"
)

client := httpagent.NewClient(&http.Client{
    Timeout: 10 * time.Second,
})

ctx = httptrace.WithClientTrace(ctx, otelhttptrace.NewClientTrace(ctx))
req, _ := http.NewRequestWithContext(ctx, "GET", "https://api.example.com/data", nil)
resp, err := client.Do(req)

Log-Trace Correlation

The agent injects trace_id and span_id into your log entries so you can jump from a log line directly to its trace. Works with both log/slog and Uber's zap.

slog

import slogagent "github.com/last9/go-agent/instrumentation/slog"

// One-line global setup
slogagent.SetDefault(os.Stdout, nil, nil)

// All *Context calls now include trace_id and span_id
slog.InfoContext(ctx, "processing request", "user_id", 42)
// Output: {"level":"INFO","msg":"processing request","user_id":42,"trace_id":"abc123...","span_id":"def456..."}

Or wrap an existing handler:

base := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
handler := slogagent.NewHandler(base, nil)
logger := slog.New(handler)

Custom attribute keys:

handler := slogagent.NewJSONHandler(os.Stdout, nil, &slogagent.Options{
    TraceKey: "dd.trace_id",
    SpanKey:  "dd.span_id",
})

Trace fields are only injected when you use *Context methods (InfoContext, ErrorContext, etc.) with a context that holds an active span. Calls without context pass through unchanged.

zap

import zapagent "github.com/last9/go-agent/instrumentation/zap"

// Spread trace fields inline — no wrapper needed
logger.Info("request handled",
    zap.String("path", r.URL.Path),
    zapagent.TraceFields(ctx)...,
)

Or use the logger wrapper for context-aware methods:

base, _ := zap.NewProduction()
logger := zapagent.New(base, nil)

logger.InfoContext(ctx, "user created", zap.String("user_id", "42"))
logger.ErrorContext(ctx, "payment failed", zap.Error(err))

Metrics

Runtime, HTTP, gRPC, database, Kafka, and Redis metrics are collected automatically — no configuration required. For business metrics, the metrics package provides helpers for the four standard instrument types.

Custom Metrics

import "github.com/last9/go-agent/metrics"

// Counter — monotonically increasing
requestCounter := metrics.NewCounter(
    "app.requests.total",
    "Total number of requests processed",
    "{request}",
)
requestCounter.Inc(ctx, attribute.String("endpoint", "/api/users"))

// Histogram — distribution of values
latencyHistogram := metrics.NewHistogram(
    "app.processing.duration",
    "Processing duration in milliseconds",
    "ms",
)
latencyHistogram.Record(ctx, duration, attribute.String("operation", "compute"))

// Gauge — current value via async callback
workerGauge := metrics.NewGauge(
    "app.workers.active",
    "Number of active worker goroutines",
    "{worker}",
    func(ctx context.Context) int64 {
        return atomic.LoadInt64(&activeWorkers)
    },
)

// UpDownCounter — value that increases and decreases
queueSize := metrics.NewUpDownCounter(
    "app.queue.size",
    "Number of items in processing queue",
    "{item}",
)
queueSize.Add(ctx, 10, attribute.String("queue", "high-priority"))
queueSize.Add(ctx, -5, attribute.String("queue", "high-priority"))

Use standard UCUM units: ms/s for time, By for bytes, {item}/{request} for counts, % for percentages.

Automatic Metrics Reference

Source	Metrics
Runtime	heap alloc, goroutines, GC count, GC pause — Go 1.24+ gets the full OTel runtime suite (15+ metrics)
HTTP/gRPC	request duration, request/response size, active requests, RPC latency
Database	connection pool usage, idle, max, wait/use/idle times
MongoDB	operation count, error count, operation duration
Kafka	messages sent/received, errors, send/process latency, message size
Redis	pool usage, command duration, connection timeouts

Route Exclusion

Health checks and infrastructure endpoints are excluded from tracing by default. This works across all supported frameworks.

Default excluded paths: /health, /healthz, /metrics, /ready, /live, /ping, and glob variants like /*/health. The gRPC-Gateway integration uses the same defaults plus /readyz, /livez, /actuator/**, and /eureka/apps/**.

Configure via environment variables:

# Exact paths
export LAST9_EXCLUDED_PATHS="/health,/healthz,/status,/version"

# Prefix exclusions
export LAST9_EXCLUDED_PATH_PREFIXES="/internal/,/debug/"

# Glob patterns
export LAST9_EXCLUDED_PATH_PATTERNS="/*/health,/*/metrics"

# Trace everything — disable all defaults
export LAST9_EXCLUDED_PATHS=""
export LAST9_EXCLUDED_PATH_PATTERNS=""

Matching runs in order: exact path (O(1) map lookup) → prefix → glob. First match wins.

HTTP Body Capture

The httpcapture middleware records HTTP request and response bodies onto the active OTel span as http.request.body and http.response.body attributes. It is framework-agnostic — a single net/http middleware that wraps any handler or router.

Body capture is opt-in and disabled by default. Enable it only after confirming your payloads do not contain PII or credentials. For production, prefer redacting sensitive fields at the collector layer using a transform processor rather than in the application.

go get github.com/last9/go-agent/instrumentation/httpcapture

Place httpcapture.Middleware inside your OTel tracing middleware so the span is already active in context:

net/http

import (
    nethttpagent "github.com/last9/go-agent/instrumentation/nethttp"
    "github.com/last9/go-agent/instrumentation/httpcapture"
)

mux := http.NewServeMux()
mux.HandleFunc("/api", myHandler)

// httpcapture wraps the mux; nethttp wraps httpcapture — span exists before httpcapture runs
http.ListenAndServe(":8080", nethttpagent.WrapHandler(httpcapture.Middleware(mux)))

Gin

import (
    ginagent "github.com/last9/go-agent/instrumentation/gin"
    "github.com/last9/go-agent/instrumentation/httpcapture"
)

r := ginagent.New()
r.POST("/api", myHandler)

http.ListenAndServe(":8080", httpcapture.Middleware(r))

Echo

import (
    echoagent "github.com/last9/go-agent/instrumentation/echo"
    "github.com/last9/go-agent/instrumentation/httpcapture"
)

e := echoagent.New()
e.POST("/api", myHandler)

http.ListenAndServe(":8080", httpcapture.Middleware(e))

Configuration

Enable and tune body capture via environment variables:

# Enable body capture (default: false)
export LAST9_BODY_CAPTURE_ENABLED=true

# Maximum bytes captured per body (default: 8192)
export LAST9_BODY_CAPTURE_MAX_BYTES=4096

# Only capture bodies on error responses — status >= 400 (default: false)
# No allocation overhead on successful requests when enabled.
export LAST9_BODY_CAPTURE_ON_ERROR_ONLY=true

# Comma-separated Content-Type prefixes to capture
# (default: application/json,application/xml,text/plain)
export LAST9_BODY_CAPTURE_CONTENT_TYPES="application/json,text/plain"

Setting LAST9_BODY_CAPTURE_CONTENT_TYPES="" captures all content types. Setting LAST9_BODY_CAPTURE_MAX_BYTES=0 records no bytes but the middleware overhead still applies — use LAST9_BODY_CAPTURE_ENABLED=false to disable entirely.

Span Attributes

Attribute	Description
http.request.body	Captured request body, truncated to LAST9_BODY_CAPTURE_MAX_BYTES
http.response.body	Captured response body, truncated to LAST9_BODY_CAPTURE_MAX_BYTES

These attributes are not part of OTel semantic conventions; they follow the convention established by last9/dotnet-otel-body-capture.

Code Call-Site Attributes

The agent automatically stamps the source location of each span onto its attributes — no configuration required. agent.Start() registers a span processor that walks the call stack when a span begins and records where in your code it originated.

This applies to Client, Producer, and Consumer spans — the outbound calls (HTTP requests, DB queries, message publishes, queue consumes) where knowing the call site speeds up debugging. Server and Internal spans are skipped to keep overhead off the hot path. The processor reuses its stack-frame buffer via sync.Pool and adds no per-span heap allocation.

Span Attributes

Attribute	Description
code.function	Function that created the span
code.filepath	Source file path
code.lineno	Line number

Attribute keys follow OTel semantic conventions (semconv v1.25.0). Stack frames inside the standard library, the OTel SDK, the agent itself, and instrumented drivers are skipped so the recorded location points at your application code.

Configuration

Variable	Required	Description
OTEL_EXPORTER_OTLP_ENDPOINT	Yes	Last9 OTLP endpoint
OTEL_EXPORTER_OTLP_HEADERS	Yes	Authorization header
OTEL_SERVICE_NAME	No	Service name (default: unknown-service)
OTEL_SERVICE_VERSION	No	Service version, e.g. git commit SHA
OTEL_RESOURCE_ATTRIBUTES	No	Additional attributes as key=value pairs
OTEL_TRACES_SAMPLER	No	Sampling strategy (default: always_on)
LAST9_TRACE_SAMPLE_RATE	No	Probabilistic sample rate, e.g. 0.1 for 10%
LAST9_EXCLUDED_PATHS	No	Exact paths excluded from tracing
LAST9_EXCLUDED_PATH_PREFIXES	No	Path prefixes excluded from tracing
LAST9_EXCLUDED_PATH_PATTERNS	No	Glob patterns excluded from tracing
LAST9_BODY_CAPTURE_ENABLED	No	Enable HTTP body capture (default: false)
LAST9_BODY_CAPTURE_MAX_BYTES	No	Max bytes captured per body (default: 8192)
LAST9_BODY_CAPTURE_ON_ERROR_ONLY	No	Capture only on status >= 400 (default: false)
LAST9_BODY_CAPTURE_CONTENT_TYPES	No	Content-Type prefixes to capture (default: application/json,application/xml,text/plain)

The agent automatically detects and records host info, OS, architecture, container ID, and process details as resource attributes. It also stamps telemetry.distro.name=last9-go-agent and telemetry.distro.version so telemetry from this agent is identifiable on the backend.

Requirements

• Go 1.22 or later (1.24+ recommended — full OTel runtime instrumentation)
• OpenTelemetry Tracing/Metrics API 1.39.0
• Semantic Conventions v1.26.0

Testing

# Unit tests — no Docker required
make test-unit

# Integration tests — requires Docker
make docker-up
make test-integration
make docker-down

Integration tests require Docker for Postgres, MySQL, Redis, and Kafka. Proto files for gRPC tests are generated via buf.

Contributing

Open an issue first, then fork, branch, and submit a pull request. Run golangci-lint run --timeout=5m ./... before pushing.

License

Apache License 2.0. See LICENSE.

---

Built on OpenTelemetry Go and opentelemetry-go-contrib.