Let AI set up TraceKit for you

AI assistants can guide you through the entire setup process

Try AI Setup

Go Integration Guide

Learn how to instrument your Go applications to send distributed traces to TraceKit.

Using Gin?

See our dedicated Gin observability guide for framework-specific setup, pain points, and best practices.

Go Distributed Tracing Guide

Go deeper with our Go distributed tracing guide -- covering common pain points, production patterns, and code examples.

New: Frontend Observability

Looking for browser-side error tracking? Check out the new Browser SDK with React, Vue, Angular, Next.js, and Nuxt integrations via Framework Wrappers.

Advanced: Manual OpenTelemetry Setup (For More Control)

Manual Setup: The sections below show manual OpenTelemetry instrumentation for users who need fine-grained control. For most use cases, the TraceKit SDK above is recommended.

90% Automatic Tracing!

With the right libraries, most of your application will be traced automatically with minimal setup. No need to manually instrument every function.

Prerequisites

What Gets Traced Automatically?

With proper setup, these operations are traced automatically with zero manual instrumentation:

ComponentSpan TypeCaptured AttributesAuto-Traced?
HTTP EndpointsSERVERmethod, route, status_code, duration, client_ipYes
Database QueriesDBdb.system, db.statement, db.name, durationYes
HTTP Client CallsCLIENTmethod, url, status_code, duration, peer.serviceYes
Redis OperationsDBdb.system (redis), db.statement, durationYes
gRPC CallsSERVER/CLIENTrpc.system, rpc.service, rpc.method, status_codeYes
MongoDB QueriesDBdb.system (mongodb), db.operation, db.name, durationYes
LLM (OpenAI/Anthropic)CLIENTgen_ai.system, model, tokens, cost, finish_reason, latencyYes
Custom Business LogicCustomuser-defined attributesManual

Installation

Install the required OpenTelemetry packages:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

Basic Setup

Create a tracing initialization function in your application:

1. Create tracing.go

package tracing

import (
    "context"
    "crypto/tls"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func InitTracer(serviceName, endpoint, apiKey string, useSSL bool) (func(), error) {
    ctx := context.Background()

    // Configure OTLP exporter options
    var opts []otlptracehttp.Option
    opts = append(opts,
        otlptracehttp.WithEndpoint(endpoint),
        otlptracehttp.WithURLPath("/v1/traces"),
        otlptracehttp.WithHeaders(map[string]string{
            "X-API-Key": apiKey,
        }),
    )

    // Configure TLS
    if useSSL {
        opts = append(opts, otlptracehttp.WithTLSClientConfig(&tls.Config{}))
    } else {
        opts = append(opts, otlptracehttp.WithInsecure())
    }

    // Create OTLP exporter
    exporter, err := otlptracehttp.New(ctx, opts...)
    if err != nil {
        return nil, err
    }

    // Create trace provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
            semconv.ServiceVersionKey.String("1.0.0"),
        )),
    )

    // Set global trace provider
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})

    // Return cleanup function
    return func() {
        _ = tp.Shutdown(ctx)
    }, nil
}

2. Initialize in main.go

package main

import (
    "log"
    "os"

    "yourapp/tracing"
)

func main() {
    // Initialize OpenTelemetry tracing
    cleanup, err := tracing.InitTracer(
        "my-service",              // Service name
        "{ appURL }",          // TraceKit endpoint
        os.Getenv("TRACEKIT_API_KEY"), // API key from environment
        false,                     // Use SSL (false for local development)
    )
    if err != nil {
        log.Fatalf("Failed to initialize tracer: %v", err)
    }
    defer cleanup()

    // Your application code here...
}

Verify It Works

Start your application and make a few requests. Then open the Traces page in your TraceKit dashboard. You should see:

  • • SERVER spans for each incoming HTTP request with route and status
  • • CLIENT spans for outgoing HTTP calls (if using wrapped http.Client)
  • • DB spans for database queries (if using GORM plugin or otelsql wrapper)

Traces typically appear within 30 seconds. If you don't see them, check the Troubleshooting section.

Framework Integration

TraceKit works seamlessly with popular Go web frameworks through OpenTelemetry instrumentation.

Gin Framework

For Gin, use the otelgin middleware:

package main

import (
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

func main() {
    // Initialize tracing (as shown above)
    cleanup, _ := tracing.InitTracer(...)
    defer cleanup()

    // Create Gin router
    r := gin.Default()

    // Add OpenTelemetry middleware
    r.Use(otelgin.Middleware("my-service"))

    // Define routes - they're automatically traced!
    r.GET("/api/users", func(c *gin.Context) {
        c.JSON(200, gin.H{"users": []string{"alice", "bob"}})
    })

    r.Run(":8080")
}

net/http (Standard Library)

For standard library HTTP servers:

package main

import (
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    // Initialize tracing (as shown above)
    cleanup, _ := tracing.InitTracer(...)
    defer cleanup()

    // Wrap your handlers with otelhttp
    mux := http.NewServeMux()

    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })

    mux.Handle("/", otelhttp.NewHandler(handler, "root"))

    http.ListenAndServe(":8080", mux)
}

Echo Framework

For Echo, use the otelecho middleware:

package main

import (
    "github.com/labstack/echo/v4"
    "go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho"
)

func main() {
    // Initialize tracing (as shown above)
    cleanup, _ := tracing.InitTracer(...)
    defer cleanup()

    e := echo.New()

    // Add OpenTelemetry middleware
    e.Use(otelecho.Middleware("my-service"))

    e.GET("/", func(c echo.Context) error {
        return c.String(200, "Hello, World!")
    })

    e.Start(":8080")
}

Gin Request Context Helper

Extract the request context from a Gin handler for use with SDK tracing methods:

// GetRequestContext extracts context.Context from *gin.Context
// Use this to pass trace context to SDK methods inside Gin handlers
func handleOrder(c *gin.Context) {
    ctx := sdk.GetRequestContext(c)
    ctx, span := sdk.StartSpan(ctx, "handle-order")
    defer span.End()
    // ...
}

Signature: SDK.GetRequestContext(c *gin.Context) context.Context — Returns the context.Context from the Gin request, preserving trace propagation.

Automatic Instrumentation Libraries

These libraries automatically create child spans for common operations. Set them up once, and every call is traced automatically.

Database Queries

Automatically trace all database operations:

GORM (PostgreSQL, MySQL, SQLite)

import (
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "go.opentelemetry.io/contrib/instrumentation/gorm.io/otelgorm"
)

func initDB() (*gorm.DB, error) {
    db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
    if err != nil {
        return nil, err
    }

    // Add OpenTelemetry plugin
    if err := db.Use(otelgorm.NewPlugin()); err != nil {
        return nil, err
    }

    return db, nil
}

Now every db.Find(), db.Create(), db.Update() call creates a span automatically!

database/sql with pgx

import (
    "database/sql"

    "github.com/jackc/pgx/v5/stdlib"
    "github.com/XSAM/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func initDB() (*sql.DB, error) {
    // Register with OpenTelemetry instrumentation
    driverName, err := otelsql.Register("pgx",
        otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    )
    if err != nil {
        return nil, err
    }

    db, err := sql.Open(driverName, dsn)
    if err != nil {
        return nil, err
    }

    return db, nil
}

HTTP Client Calls

Automatically trace all outgoing HTTP requests:

import (
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

// Create an HTTP client with OpenTelemetry transport (one-time setup)
var httpClient = &http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
}

// Now use it for all HTTP calls - automatically traced!
func callExternalAPI(ctx context.Context) (*Response, error) {
    req, _ := http.NewRequestWithContext(ctx, "GET", "https://api.example.com/users", nil)

    // This HTTP call is automatically traced as a child span
    resp, err := httpClient.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    // Parse response...
    return parseResponse(resp)
}

Every HTTP call made with this client is automatically traced, including request/response details. The spans will appear as children of your current operation.

Redis Operations

Trace Redis commands automatically with go-redis:

import (
    "github.com/redis/go-redis/v9"
    "github.com/redis/go-redis/extra/redisotel/v9"
)

func initRedis() *redis.Client {
    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    // Enable OpenTelemetry instrumentation
    if err := redisotel.InstrumentTracing(rdb); err != nil {
        panic(err)
    }

    return rdb
}

// Usage - all operations are automatically traced!
func cacheUser(ctx context.Context, rdb *redis.Client, userID string, data string) error {
    // This Redis SET is automatically traced
    return rdb.Set(ctx, "user:"+userID, data, time.Hour).Err()
}

func getUser(ctx context.Context, rdb *redis.Client, userID string) (string, error) {
    // This Redis GET is automatically traced
    return rdb.Get(ctx, "user:"+userID).Result()
}

All Redis operations (GET, SET, HGET, etc.) are now traced automatically.

gRPC Calls

Automatically trace gRPC server and client calls:

Server Side

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

func main() {
    // Initialize tracing first...
    cleanup, _ := tracing.InitTracer(...)
    defer cleanup()

    // Create gRPC server with OpenTelemetry interceptor
    s := grpc.NewServer(
        grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
        grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor()),
    )

    // Register your services
    pb.RegisterYourServiceServer(s, &yourService{})

    // All RPC calls are now automatically traced!
    s.Serve(lis)
}

Client Side

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

func createGRPCClient() (*grpc.ClientConn, error) {
    // Create gRPC client with OpenTelemetry interceptor
    conn, err := grpc.Dial(
        "localhost:50051",
        grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
        grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
    )
    if err != nil {
        return nil, err
    }

    // All outgoing RPC calls are automatically traced!
    return conn, nil
}

MongoDB Queries

Automatically trace MongoDB operations:

import (
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
    "go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo"
)

func initMongo(ctx context.Context) (*mongo.Client, error) {
    // Create client options with OpenTelemetry monitor
    opts := options.Client().
        ApplyURI("mongodb://localhost:27017").
        SetMonitor(otelmongo.NewMonitor())

    client, err := mongo.Connect(ctx, opts)
    if err != nil {
        return nil, err
    }

    return client, nil
}

// Usage - all operations are automatically traced!
func findUsers(ctx context.Context, client *mongo.Client) ([]User, error) {
    coll := client.Database("mydb").Collection("users")

    // This MongoDB query is automatically traced
    cursor, err := coll.Find(ctx, bson.M{"status": "active"})
    if err != nil {
        return nil, err
    }

    var users []User
    cursor.All(ctx, &users)
    return users, nil
}

All MongoDB queries, inserts, updates, and deletes are automatically traced.

Message Queues

Kafka (Sarama)

import (
    "github.com/IBM/sarama"
    "go.opentelemetry.io/contrib/instrumentation/github.com/IBM/sarama/otelsarama"
)

func createKafkaProducer() (sarama.SyncProducer, error) {
    config := sarama.NewConfig()
    config.Producer.Return.Successes = true

    producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, config)
    if err != nil {
        return nil, err
    }

    // Wrap with OpenTelemetry instrumentation
    producer = otelsarama.WrapSyncProducer(config, producer)

    return producer, nil
}

// Usage - all messages are automatically traced!
func publishMessage(ctx context.Context, producer sarama.SyncProducer, msg string) error {
    message := &sarama.ProducerMessage{
        Topic: "user-events",
        Value: sarama.StringEncoder(msg),
    }

    // This publish is automatically traced
    _, _, err := producer.SendMessage(message)
    return err
}

RabbitMQ (amqp091-go)

import (
    amqp "github.com/rabbitmq/amqp091-go"
    "go.opentelemetry.io/contrib/instrumentation/github.com/rabbitmq/amqp091-go/otelmqp"
)

func publishToQueue(ctx context.Context, ch *amqp.Channel, msg string) error {
    // Wrap channel with OpenTelemetry instrumentation
    otlChannel := otelmqp.NewChannel(ch)

    // Publish message - automatically traced!
    err := otlChannel.PublishWithContext(
        ctx,
        "",          // exchange
        "my-queue",  // routing key
        false,       // mandatory
        false,       // immediate
        amqp.Publishing{
            ContentType: "text/plain",
            Body:        []byte(msg),
        },
    )
    return err
}

// Consumer side
func consumeFromQueue(ctx context.Context, ch *amqp.Channel) error {
    otlChannel := otelmqp.NewChannel(ch)

    // Consume messages - automatically traced!
    msgs, err := otlChannel.Consume(
        "my-queue", // queue
        "",         // consumer
        true,       // auto-ack
        false,      // exclusive
        false,      // no-local
        false,      // no-wait
        nil,        // args
    )
    if err != nil {
        return err
    }

    for msg := range msgs {
        // Process message with tracing context
        processMessage(msg)
    }
    return nil
}

HTTP API Reference

Complete reference for all HTTP instrumentation methods in the Go SDK:

Method Parameters Returns Description
SDK.HTTPHandler(handler, op) handler http.Handler, op string http.Handler Wraps an http.Handler with automatic span creation. Use for individual route handlers with net/http.
SDK.HTTPMiddleware(op) op string func(http.Handler) http.Handler Returns middleware that traces all requests. Compatible with any http.Handler-based router (Chi, Gorilla, etc.).
SDK.HTTPClient(client) client *http.Client *http.Client Returns a traced HTTP client that creates CLIENT spans for all outgoing requests with context propagation.
SDK.WrapRoundTripper(rt) rt http.RoundTripper http.RoundTripper Wraps an http.RoundTripper with tracing. Use when you need custom transport configuration.
ExtractClientIP(r) r *http.Request string Extracts the client IP address from the request, checking X-Forwarded-For, X-Real-IP, and RemoteAddr headers.

Database API Reference

Complete reference for all database instrumentation methods in the Go SDK:

Method Parameters Returns Description
SDK.WrapDB(db, dbSystem) db *sql.DB, dbSystem string *TracedDB Wraps a *sql.DB with automatic span creation for every query. Pass the database system name (e.g. "postgresql", "mysql", "sqlite").
TracedDB — Wraps *sql.DB with automatic span creation. Implements all standard sql.DB methods:
QueryContext(ctx, query, args...)Query(query, args...)QueryRowContext(ctx, query, args...)QueryRow(query, args...)ExecContext(ctx, query, args...)Exec(query, args...)PrepareContext(ctx, query)Prepare(query)BeginTx(ctx, opts)Begin()PingContext(ctx)Ping()Close()DB()SetMaxOpenConns(n)SetMaxIdleConns(n)Stats()Driver()
SDK.GormPlugin() gorm.Plugin Returns a GORM plugin that traces all database operations. Use with db.Use(sdk.GormPlugin()).
SDK.WithGormTracing(config) config *gorm.Config *gorm.Config Returns a GORM config with tracing plugin pre-registered. Alternative to GormPlugin().
SDK.WrapMongoClient(client) client *mongo.Client *mongo.Client Wraps a MongoDB client with automatic span creation for all database operations.
SDK.WrapRedisCluster(client) client *redis.ClusterClient *redis.ClusterClient Wraps a Redis cluster client with automatic span creation. For single-node Redis, use SDK.WrapRedis().

TracedDB Usage Example

// Wrap a database/sql connection with tracing
sqlDB, _ := sql.Open("postgres", "postgres://localhost/mydb?sslmode=disable")
db := sdk.WrapDB(sqlDB, "postgresql")

// All queries are now automatically traced
rows, err := db.QueryContext(ctx, "SELECT id, name FROM users WHERE active = $1", true)
if err != nil {
    log.Fatal(err)
}
defer rows.Close()

for rows.Next() {
    var id int
    var name string
    rows.Scan(&id, &name)
    // process row...
}

// Exec also creates spans automatically
_, err = db.ExecContext(ctx, "UPDATE users SET last_login = $1 WHERE id = $2", time.Now(), userID)

// Transactions are traced too
tx, _ := db.BeginTx(ctx, nil)
tx.ExecContext(ctx, "INSERT INTO orders (user_id, total) VALUES ($1, $2)", userID, total)
tx.Commit()

Code Monitoring

Code monitoring lets you capture variable snapshots from running production code without stopping your application. Set non-breaking breakpoints from the dashboard and inspect variable state when they trigger. v6.0 adds production safety features including automatic PII scrubbing, crash isolation, circuit breakers, and a remote kill switch — all enabled by default.

Code Monitoring API

Method Parameters Returns Description
Primary Methods
SDK.CheckAndCapture(file, line, vars) file string, line int, vars map[string]interface{} Checks if a breakpoint is set at the given file:line and captures a snapshot of the provided variables if so. Primary method for code monitoring.
SDK.CheckAndCaptureWithContext(ctx, label, vars) ctx context.Context, label string, vars map[string]interface{} Context-aware version of CheckAndCapture. Includes trace correlation so snapshots link to the active span.
SDK.SnapshotClient() *SnapshotClient Returns the underlying snapshot client for advanced use cases.
Advanced / Internal Types
SnapshotClient Used internally by the SDK for breakpoint polling and snapshot transmission. Most users should use SDK.CheckAndCapture() instead.
NewSnapshotClient(apiKey, url, serviceName) apiKey string, url string, serviceName string *SnapshotClient Creates a new snapshot client. Advanced — prefer using the SDK's built-in client via EnableCodeMonitoring config.
BreakpointConfig Configuration for a breakpoint including target file, line number, and optional condition expression.
SecurityFlag Flags for sensitive data detection in snapshots. Used to redact PII and credentials before transmission.
Snapshot Captured variable state at a breakpoint. Contains the variable map, timestamp, trace context, and file location.
v6.0 Production Safety Types
CaptureConfig Configuration for snapshot capture behavior including PII scrubbing toggles, custom redaction patterns, and capture depth limits.
CircuitBreakerConfig Circuit breaker settings: failure threshold (default 3), window duration (default 60s), and cooldown period (default 5m). Controls automatic disabling on repeated failures.
PIIPattern Defines a sensitive data detection pattern with a name, compiled regex, and redaction marker type. 13 built-in patterns are provided; custom patterns can be added via CaptureConfig.

Usage Example

// Enable code monitoring in config
sdk, _ := tracekit.NewSDK(&tracekit.Config{
    APIKey:               os.Getenv("TRACEKIT_API_KEY"),
    ServiceName:          "order-service",
    EnableCodeMonitoring: true,
})
defer sdk.Shutdown(context.Background())

// Capture variables at a specific point in your code
func processOrder(ctx context.Context, orderID string, total float64) error {
    user := getCurrentUser(ctx)

    // CheckAndCapture only captures when a breakpoint is active at this location
    sdk.CheckAndCapture("handlers/order.go", 42, map[string]interface{}{
        "orderID": orderID,
        "total":   total,
        "user":    user,
    })

    // Context-aware version links snapshots to the current trace
    sdk.CheckAndCaptureWithContext(ctx, "order-processing", map[string]interface{}{
        "orderID": orderID,
        "total":   total,
        "status":  "validated",
    })

    // Business logic continues...
    return nil
}

Production Safety

All production safety features are enabled by default — no configuration required.

PII Scrubbing

13 built-in patterns detect sensitive data (passwords, tokens, SSNs, credit cards). Enabled by default with typed [REDACTED:type] markers.

Crash Isolation

All entry points use defer/recover to catch panics. Snapshot failures never crash your application.

Circuit Breaker

After 3 failures within 60 seconds, the circuit opens for a 5-minute cooldown. Automatically recovers when the backend is healthy again.

Remote Kill Switch

Toggle code monitoring on or off from the dashboard. Changes propagate via SSE in real-time — no restart required.

Real-Time Updates: The Go SDK uses SSE (Server-Sent Events) for auto-discovery of breakpoint changes and kill switch state. This replaces the previous 30-second polling approach, giving you instant control over production instrumentation.

Note: Code monitoring requires EnableCodeMonitoring: true in the SDK config. Breakpoints are managed from the Code Monitoring dashboard.

End-to-End Workflow

1

Enable Code Monitoring

Code monitoring defaults to disabled in the Go SDK. You must explicitly enable it in your config.

sdk, _ := tracekit.NewSDK(&tracekit.Config{
    APIKey:               os.Getenv("TRACEKIT_API_KEY"),
    ServiceName:          "my-service",
    EnableCodeMonitoring: true,  // default: false
})
defer sdk.Shutdown(context.Background())
2

Add Capture Points

Place CheckAndCaptureWithContext calls at code paths you want to debug. The context-aware version links snapshots to the current trace.

func processOrder(ctx context.Context, orderID string, total float64) error {
    sdk.CheckAndCaptureWithContext(ctx, "order-processing", map[string]interface{}{
        "orderID": orderID,
        "total":   total,
        "status":  "validated",
    })
    // Business logic continues...
    return nil
}
3

Deploy and Verify Traces

Deploy your application and confirm traces are flowing. Check the TraceKit dashboard at /traces to see incoming requests.

# Build and deploy
go build -o myservice ./cmd/server
./myservice

# Verify traces appear in dashboard at /traces
4

Navigate to Code Monitoring

Go to /snapshots and click the "Browse Code" tab. You'll see auto-discovered files and functions from your traces.

5

Set Breakpoints

Breakpoints are auto-registered on the first CheckAndCaptureWithContext call. You can also manually add breakpoints via the UI "Set Breakpoint" button.

6

Trigger a Capture

Make a request that hits a code path with CheckAndCaptureWithContext. The SDK checks for active breakpoints, captures the variable map, stack trace, and request context.

7

View Snapshot

Go to /snapshots and click the captured snapshot. View the captured variables, call stack, request context, security flags, and trace link.

Production Safety — Enabled by Default

No extra configuration needed. PII scrubbing redacts sensitive values before transmission, crash isolation prevents panics from affecting your app, the circuit breaker disables capture after repeated failures, and the remote kill switch lets you toggle monitoring from the dashboard in real-time via SSE.

LLM Instrumentation

TraceKit provides an HTTP transport wrapper that automatically instruments OpenAI and Anthropic API calls. LLM calls appear as spans with OTel GenAI semantic convention attributes.

Setup

Create an LLM-instrumented HTTP transport and pass it to your OpenAI or Anthropic client:

Go
// Create LLM-instrumented transport (one line)
transport := sdk.NewLLMTransport(nil)
httpClient := &http.Client{Transport: transport}

// Use with OpenAI
openaiClient := openai.NewClient(option.WithHTTPClient(httpClient))

// Use with Anthropic
anthropicClient, _ := anthropic.NewClient(
    anthropic.WithHTTPClient(httpClient),
)

Captured Attributes

AttributeDescription
gen_ai.systemProvider name (openai, anthropic)
gen_ai.request.modelModel name (gpt-4o, claude-sonnet-4-20250514, etc.)
gen_ai.usage.input_tokensPrompt token count
gen_ai.usage.output_tokensCompletion token count
gen_ai.response.finish_reasonsFinish reason (stop, end_turn, tool_calls)

Configuration

Go
sdk, err := tracekit.NewSDK(&tracekit.Config{
    APIKey:      os.Getenv("TRACEKIT_API_KEY"),
    ServiceName: "my-service",

    // LLM configuration (optional — defaults shown)
    InstrumentLLM: &tracekit.LLMConfig{
        Enabled:        true,  // Master toggle
        OpenAI:         true,  // Detect OpenAI API calls
        Anthropic:      true,  // Detect Anthropic API calls
        CaptureContent: false, // Capture prompts/completions
    },
})

Environment Variable Override

Use TRACEKIT_LLM_CAPTURE_CONTENT=true to enable prompt/completion capture without code changes. Useful for enabling in staging but not production.

Streaming Support

Streaming responses produce a single span covering the entire stream. The transport wraps the response body to parse SSE events and accumulate token counts. The span finalizes when the stream is closed.

LLM Dashboard

View LLM cost, token usage, and latency metrics on the dedicated LLM Observability dashboard at /ai/llm in your TraceKit instance.

Local UI (Development Mode)

Debug your Go application locally without creating an account. TraceKit Local UI runs on your machine at http://localhost:9999 and automatically receives traces when you run your app in development mode.

Automatic Detection

The Go SDK automatically detects when Local UI is running on port 9999 and sends traces to both Local UI and cloud (if you have an API key configured).

Quick Start

1. Install the Local UI:

npm install -g @tracekit/local-ui

2. Start the Local UI:

tracekit-local

3. Run your app in development mode:

ENV=development go run main.go

4. Open your browser:

http://localhost:9999

Features

Auto-Detection

SDK checks for Local UI at localhost:9999 on startup

Real-Time Updates

See traces instantly with WebSocket live updates

Development Only

Only activates when ENV=development

Works Offline

No internet connection required - everything runs locally

Benefits

  • See your first trace in under 60 seconds
  • Debug locally without switching to the cloud dashboard
  • Stay in your flow - everything runs on your machine
  • Works completely offline
  • Perfect for development and demos

Troubleshooting

If traces aren't appearing in Local UI, check:

  • Local UI is running (curl http://localhost:9999/api/health)
  • ENV=development is set
  • SDK version is v0.3.2 or higher
  • Check console for " Local UI detected" message

Service Discovery

TraceKit automatically instruments outgoing HTTP calls to create service dependency graphs. When your service makes an HTTP request to another service, TraceKit creates CLIENT spans and injects trace context headers.

Custom Service Name Mappings

For local development or when service names can't be inferred from hostnames, configure service name mappings:

// Initialize SDK with service name mappings
sdk, _ := tracekit.NewSDK(&tracekit.Config{
    APIKey:      os.Getenv("TRACEKIT_API_KEY"),
    ServiceName: "my-service",
    // Map localhost URLs to actual service names
    ServiceNameMappings: map[string]string{
        "localhost:8082": "payment-service",
        "localhost:8083": "user-service",
        "localhost:8084": "inventory-service",
    },
})

// Now requests to localhost:8082 will show as "payment-service"
client := sdk.HTTPClient(&http.Client{})
resp, _ := client.Get("http://localhost:8082/charge")
// -> Creates CLIENT span with peer.service = "payment-service"

Service Name Detection

TraceKit intelligently extracts service names from URLs:

URLExtracted Service Name
http://payment-service:3000payment-service
http://payment.internalpayment
http://payment.svc.cluster.localpayment
https://api.example.comapi.example.com

Viewing Service Dependencies

Visit your TraceKit dashboard to see the Service Map - a visual graph showing which services call which, with health metrics and latency data.

Manual Instrumentation

The Go SDK provides convenience wrappers around OpenTelemetry for common tracing operations. Use these methods to add custom spans, attributes, and error handling to your business logic.

SDK Tracing API

Method Parameters Returns Description
Core Tracing
SDK.StartSpan(ctx, name, opts...) ctx context.Context, name string, opts ...trace.SpanStartOption (context.Context, trace.Span) Starts a new span as a child of the current span in ctx. Wraps tracer.Start().
Attributes
SDK.AddAttribute(span, key, value) span trace.Span, key string, value string Adds a string attribute to the span.
SDK.AddAttributes(span, attrs...) span trace.Span, attrs ...attribute.KeyValue Adds multiple typed attributes to the span in one call.
SDK.AddIntAttribute(span, key, value) span trace.Span, key string, value int64 Adds an integer attribute to the span.
SDK.AddFloatAttribute(span, key, value) span trace.Span, key string, value float64 Adds a float attribute to the span.
SDK.AddBoolAttribute(span, key, value) span trace.Span, key string, value bool Adds a boolean attribute to the span.
Events & Errors
SDK.AddEvent(span, name, attrs...) span trace.Span, name string, attrs ...attribute.KeyValue Records a timestamped event on the span with optional attributes.
SDK.RecordError(span, err) span trace.Span, err error Records an error on the span with automatic stack trace capture.
SDK.RecordErrorWithMessage(span, err, message) span trace.Span, err error, message string Records an error with a custom message for additional context.
Status
SDK.SetSuccess(span) span trace.Span Marks the span status as OK.
SDK.SetSuccessWithMessage(span, message) span trace.Span, message string Marks the span status as OK with a descriptive message.
SDK.SetError(span, message) span trace.Span, message string Marks the span status as Error with a message.
Semantic Helpers
SDK.AddHTTPAttributes(span, method, url, statusCode) span trace.Span, method string, url string, statusCode int Adds standard HTTP semantic attributes (http.method, http.url, http.status_code).
SDK.AddDatabaseAttributes(span, dbSystem, dbName, operation, table) span trace.Span, dbSystem string, dbName string, operation string, table string Adds standard database semantic attributes (db.system, db.name, db.operation, db.sql.table).
SDK.AddUserAttributes(span, userID, email) span trace.Span, userID string, email string Adds user identification attributes (enduser.id, enduser.email).
SDK.AddBusinessAttributes(span, attrs) span trace.Span, attrs map[string]interface{} Adds arbitrary business-specific attributes from a map.
Utilities
SDK.TraceFunction(ctx, name, fn) ctx context.Context, name string, fn func(context.Context, trace.Span) error error Wraps a function with automatic span creation, error recording, and status setting.

Usage Example

// Using SDK wrapper methods (recommended)
ctx, span := sdk.StartSpan(ctx, "process-order")
defer span.End()

// Add typed attributes
sdk.AddAttribute(span, "order.id", orderID)
sdk.AddIntAttribute(span, "order.total", int64(total))

// Add semantic attributes for HTTP context
sdk.AddHTTPAttributes(span, "POST", "/api/orders", 201)

// Record business attributes
sdk.AddBusinessAttributes(span, map[string]interface{}{
    "order.priority": "high",
    "order.region":   region,
})

// Handle errors
if err != nil {
    sdk.RecordError(span, err)
    return err
}

sdk.SetSuccess(span)

SDK Wrappers vs Raw OpenTelemetry

Recommended

SDK Wrappers

// Start a span
ctx, span := sdk.StartSpan(ctx, "process-order")
defer span.End()

// Add attributes
sdk.AddAttribute(span, "order.id", id)
sdk.AddIntAttribute(span, "total", int64(100))

// Record error
sdk.RecordError(span, err)

// Set status
sdk.SetSuccess(span)
Raw OTel Equivalent
// Start a span
tracer := otel.Tracer("my-service")
ctx, span := tracer.Start(ctx, "process-order")
defer span.End()

// Add attributes
span.SetAttributes(attribute.String("order.id", id))
span.SetAttributes(attribute.Int64("total", 100))

// Record error
span.RecordError(err)

// Set status
span.SetStatus(codes.Ok, "")

The SDK wrappers reduce boilerplate and automatically handle stack trace capture on errors. Both approaches produce identical trace output.

Function Tracing

Use TraceFunction to wrap an entire function call with automatic span lifecycle management:

// TraceFunction wraps an entire function call with automatic span management
err := sdk.TraceFunction(ctx, "validate-order", func(ctx context.Context, span trace.Span) error {
    sdk.AddAttribute(span, "order.id", orderID)

    // Your business logic here
    if !isValid(orderID) {
        return fmt.Errorf("invalid order: %s", orderID)
    }

    return nil
})
// If the function returns an error, TraceFunction automatically:
// 1. Records the error on the span
// 2. Sets span status to Error
// If it returns nil, span status is set to OK

Lifecycle Methods

Method Parameters Returns Description
SDK.Tracer() trace.Tracer Returns the underlying OpenTelemetry Tracer instance for advanced use cases.
SDK.Shutdown(ctx) ctx context.Context error Flushes all pending spans and shuts down the trace provider. Call in defer from main().

Raw OpenTelemetry

For users who need direct OpenTelemetry API access, here is the equivalent raw approach. The SDK wrappers above are recommended for most use cases.

package main

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
)

func processOrder(ctx context.Context, orderID string) error {
    // Get tracer
    tracer := otel.Tracer("my-service")

    // Start a parent span
    ctx, span := tracer.Start(ctx, "processOrder")
    defer span.End()

    // Add attributes to the span
    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("order.status", "processing"),
    )

    // Create nested child spans - they automatically inherit from parent via ctx
    if err := validateOrder(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }

    if err := chargePayment(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }

    // Mark span as successful
    span.SetStatus(codes.Ok, "Order processed successfully")
    return nil
}

func validateOrder(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("my-service")

    // Child span - automatically linked to parent via ctx
    ctx, span := tracer.Start(ctx, "validateOrder")
    defer span.End()

    span.SetAttributes(attribute.String("order.id", orderID))

    // Validation logic here...
    span.SetStatus(codes.Ok, "Validation successful")
    return nil
}

func chargePayment(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("my-service")

    // Another child span - also linked to parent via ctx
    ctx, span := tracer.Start(ctx, "chargePayment")
    defer span.End()

    span.SetAttributes(attribute.String("order.id", orderID))

    // Payment processing logic here...
    span.SetStatus(codes.Ok, "Payment successful")
    return nil
}

Environment Variables

Best practice: Store sensitive configuration in environment variables:

# .env
TRACEKIT_API_KEY=tk_your_generated_api_key_here
TRACEKIT_ENDPOINT=app.tracekit.dev
TRACEKIT_SERVICE_NAME=my-backend-service

Load in your application:

import "os"

sdk, err := tracekit.NewSDK(&tracekit.Config{
    APIKey:      os.Getenv("TRACEKIT_API_KEY"),
    ServiceName: os.Getenv("TRACEKIT_SERVICE_NAME"),
    Endpoint:    os.Getenv("TRACEKIT_ENDPOINT"),
})

All configuration options available for the Go SDK:

Option Type Default Env Variable Description
APIKey string required TRACEKIT_API_KEY Your TraceKit API key for authentication
ServiceName string required TRACEKIT_SERVICE_NAME Name of your service in the trace dashboard
Endpoint string "app.tracekit.dev" TRACEKIT_ENDPOINT TraceKit collector endpoint URL
TracesPath string "/v1/traces" TRACEKIT_TRACES_PATH HTTP path for trace data export
MetricsPath string "/v1/metrics" TRACEKIT_METRICS_PATH HTTP path for metrics data export
UseSSL bool false TRACEKIT_USE_SSL Enable TLS for the exporter connection
ServiceVersion string "1.0.0" TRACEKIT_SERVICE_VERSION Version of your service (shown in traces)
Environment string "" TRACEKIT_ENVIRONMENT Deployment environment (e.g. production, staging)
EnableCodeMonitoring bool false TRACEKIT_CODE_MONITORING_ENABLED Enable live code debugging / snapshot capture
CodeMonitoringPollInterval time.Duration 30s TRACEKIT_CODE_MONITORING_POLL_INTERVAL How often to poll for active breakpoints
SamplingRate float64 1.0 TRACEKIT_SAMPLE_RATE Trace sampling rate (0.0 to 1.0, where 1.0 = 100%)
BatchTimeout time.Duration 5s TRACEKIT_BATCH_TIMEOUT Max time before flushing a batch of spans
ServiceNameMappings map[string]string nil - Map host:port to service names for service discovery
ResourceAttributes map[string]string nil - Additional OTel resource attributes added to all spans

Note: The Go SDK does not auto-read environment variables. You must load them with os.Getenv() and pass values to the Config struct.

Production Configuration

Production Checklist

  • • Use HTTPS/TLS for the OTLP endpoint
  • • Store API keys in a secrets manager (AWS Secrets Manager, HashiCorp Vault)
  • • Set appropriate service names and versions
  • • Configure resource attributes (deployment.environment, host.name, etc.)
  • • Adjust sampling rates if needed for high-traffic services

Troubleshooting

Traces not appearing?

Cause: The SDK is not sending traces to TraceKit, usually due to misconfiguration.

Fix:

  • Verify API key is correct and not revoked
  • Check endpoint URL is app.tracekit.dev (no http:// prefix when using WithEndpoint)
  • Look for OpenTelemetry export errors in application logs
  • Confirm SamplingRate is not set to 0

Connection refused errors?

Cause: The TraceKit endpoint is unreachable from your application.

Fix:

  • Test connectivity with curl -X POST https://app.tracekit.dev/v1/traces (should return 401, not connection refused)
  • Check firewall/DNS rules
  • For local development, set UseSSL: false in Config or use http:// prefix in Endpoint

Code monitoring not working?

Cause: Snapshots require explicit enablement and a captureSnapshot call in your code path.

Fix:

  • Set EnableCodeMonitoring: true in Config struct
  • Verify captureSnapshot() is called in the code path you want to inspect
  • Reduce CodeMonitoringPollInterval to 5s for testing
  • Check that your API key has code monitoring permissions in the dashboard

Authentication errors (401/403)?

Cause: The API key is invalid, revoked, or has whitespace from copy-paste.

Fix:

  • Regenerate API key in dashboard settings
  • Trim whitespace: strings.TrimSpace(os.Getenv("TRACEKIT_API_KEY"))
  • Ensure you are using the correct key for your environment (production vs staging)

Partial or missing spans?

Cause: Some instrumentation is not registered or context propagation is broken across goroutines.

Fix:

  • Verify all middleware is registered (Gin, database, HTTP client)
  • Pass context.Context through goroutine boundaries -- use go func(ctx context.Context) {...}(ctx) not go func() {...}()
  • Check that ServiceNameMappings is set if external service spans show IP addresses instead of names

Complete Example

Here's a complete working example with Gin:

package main

import (
    "log"
    "os"

    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

    "yourapp/tracing"
)

func main() {
    // Initialize tracing
    cleanup, err := tracing.InitTracer(
        "backend-api",
        "{ appURL }",
        os.Getenv("TRACEKIT_API_KEY"),
        false,
    )
    if err != nil {
        log.Fatalf("Failed to initialize tracer: %v", err)
    }
    defer cleanup()

    // Create Gin router
    r := gin.Default()
    r.Use(otelgin.Middleware("backend-api"))

    // Routes
    r.GET("/api/users", getUsers)
    r.POST("/api/users", createUser)

    // Start server
    log.Println("Server starting on :8080")
    r.Run(":8080")
}

func getUsers(c *gin.Context) {
    // This endpoint is automatically traced!
    c.JSON(200, gin.H{
        "users": []string{"alice", "bob", "charlie"},
    })
}

func createUser(c *gin.Context) {
    // This endpoint is automatically traced too!
    c.JSON(201, gin.H{
        "message": "User created successfully",
    })
}

You're all set!

Your Go application is now sending traces to TraceKit. Visit the Dashboard to see your traces.

Custom Metrics

The TraceKit Go SDK includes a lightweight metrics API for tracking counters, gauges, and histograms.

Counter

Track monotonically increasing values (requests, events):

// Create a counter with optional tags
counter := sdk.Counter("http.requests.total", map[string]string{
    "service": "api",
    "method":  "GET",
})

// Increment by 1
counter.Inc()

// Add a specific value
counter.Add(5)

Gauge

Track values that can go up or down (queue size, connections):

// Create a gauge
gauge := sdk.Gauge("http.connections.active", nil)

// Set to specific value
gauge.Set(42)

// Increment/decrement
gauge.Inc()
gauge.Dec()

Histogram

Track value distributions (latencies, sizes):

// Create a histogram with tags
histogram := sdk.Histogram("http.request.duration", map[string]string{
    "unit": "ms",
})

// Record values
histogram.Record(45.2)
histogram.Record(123.5)

Note: Metrics are automatically buffered and exported via OTLP. View them in the Metrics Explorer.

🔄 Migrating from OpenTelemetry

TraceKit wraps OpenTelemetry internally, so you get the same standards-based trace data with significantly less setup code. Here's how to migrate from a raw OpenTelemetry setup to TraceKit.

Before vs After

Before: Raw OpenTelemetry ~60 lines
// 7 imports required
import (
    "context"
    "crypto/tls"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func initTracer() (func(), error) {
    ctx := context.Background()
    res, _ := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceNameKey.String("my-service"),
        ),
    )
    tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12}
    exporter, _ := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("api.tracekit.io"),
        otlptracehttp.WithTLSClientConfig(tlsCfg),
        otlptracehttp.WithHeaders(map[string]string{
            "Authorization": "Bearer " + apiKey,
        }),
    )
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{},
            propagation.Baggage{},
        ),
    )
    return func() { tp.Shutdown(ctx) }, nil
}
// + middleware setup, manual span creation...
After: TraceKit SDK 3 lines
import "github.com/tracekit-dev/go-sdk/tracekit"

sdk, _ := tracekit.NewSDK(&tracekit.Config{
    APIKey:      os.Getenv("TRACEKIT_API_KEY"),
    ServiceName: "my-service",
})
defer sdk.Shutdown(context.Background())

// Middleware: one line
r.Use(sdk.GinMiddleware())

Migration Steps

1

Install the SDK: go get github.com/tracekit-dev/go-sdk

2

Replace init code: Remove your initTracer() function and replace with tracekit.NewSDK()

3

Replace middleware: Swap otelgin.Middleware() with sdk.GinMiddleware()

4

Remove OTel dependencies: Run go mod tidy to clean up unused OpenTelemetry packages

5

Verify: Start your app and check the Traces page for incoming data

Key Migration Benefits

  • 60 lines to 3 lines — no more boilerplate for exporters, TLS, propagators
  • No OTel dependency management — TraceKit handles version pinning internally
  • Built-in code monitoring — not available with raw OpenTelemetry
  • Built-in security scanning — automatic variable redaction on snapshots
  • One-line middleware — replaces manual instrumentation registration

Best Practices

DO: Initialize tracing before loading routes/handlers

Call tracekit.Init() at the top of main() before registering any HTTP handlers or routes so that auto-instrumentation hooks are in place.

DO: Use environment variables for API keys

Store your API key in TRACEKIT_API_KEY rather than hardcoding it. The SDK reads this automatically.

DO: Use defer sdk.Shutdown(ctx) immediately after init

Go's defer ensures all pending spans are flushed when main() exits, even on panic recovery.

sdk, err := tracekit.Init(ctx)
if err != nil {
    log.Fatal(err)
}
defer sdk.Shutdown(context.Background())

DO: Use context.Context propagation for manual spans

Always pass context.Context through your call chain so that child spans correctly attach to the parent trace.

DO: Enable code monitoring in staging first

Test breakpoint capture and snapshot behavior in a staging environment before rolling out to production.

DO: Use sampling in high-traffic services

Set TRACEKIT_SAMPLING_RATE to a value below 1.0 for services handling thousands of requests per second to reduce overhead without losing visibility.

DO: Set meaningful service names

Use TRACEKIT_SERVICE_NAME to give your service a descriptive name that makes it easy to identify in the trace viewer.

DON'T: Create spans for every function

Trace boundaries like HTTP handlers, database calls, and external service calls. Instrumenting internal helper functions adds noise and overhead without useful insight.

DON'T: Add high-cardinality attributes

Avoid using user IDs, request IDs, or session tokens as span attributes. These create excessive unique time series and degrade query performance. Use structured logging for per-request identifiers.

DON'T: Disable TLS in production

The TRACEKIT_INSECURE flag is for local development only. Always use TLS when sending traces to TraceKit in production.

DON'T: Forget to pass context between goroutines

When spawning goroutines, pass the parent context.Context so child spans maintain trace correlation. Without context propagation, spans appear as disconnected root traces.

Performance Overhead

TraceKit is built on OpenTelemetry's efficient batch processing pipeline. The SDK adds minimal overhead to your Go application.

Negligible

Request Tracing

< 1ms per request

Spans are batched and exported asynchronously.

Negligible

Code Monitoring (Idle)

Zero overhead

No performance impact when no active breakpoints.

Low

Code Monitoring (Capture)

< 5ms per snapshot

Includes variable serialization and security scanning.

Negligible

Memory Footprint

~5-10 MB

SDK runtime and span buffer.

Negligible

SDK Initialization

< 100ms one-time

One-time cost at application startup. Does not affect request latency.

Note: Performance characteristics are typical for production workloads and may vary with application complexity, request volume, and number of instrumented libraries. Use sampling configuration to reduce overhead in high-traffic services.

Next Steps

  • Add auto-instrumentation libraries for components you use (Redis, gRPC, MongoDB, etc.)
  • Explore your traces on the Traces page to identify performance bottlenecks
  • Optionally add custom spans for specific business logic you want to measure
  • Configure sampling for high-traffic services to reduce overhead
  • Set up alert rules to get notified when issues occur

Pro Tip

Start with framework middleware + database instrumentation. This usually gives you 80% coverage with just 2-3 lines of setup code. Add more auto-instrumentation libraries as needed.

Copilot

No conversations yet

TraceKit Copilot

Ask about services, traces, alerts, snapshots, and LLM usage.

Copilot uses AI to analyze your APM data. Responses may not always be accurate.

Try TraceKit in 10 seconds

Explore real traces, dashboards, and live code monitoring. No signup needed.

Sign Up Free
No credit card No signup Real data