Jones Charles

Posted on Mar 5

Building Observable Microservices: Distributed Tracing with Jaeger and GoFrame

#go #microservices #jaeger #tracing

Ever found yourself lost in a maze of microservices, wondering where that request disappeared to? 🤔 You're not alone! In this guide, I'll show you how to implement distributed tracing in your Go applications using Jaeger and GoFrame. By the end, you'll be able to track requests across your entire system like a pro! 🚀

What We'll Cover 📋

Setting up Jaeger with Docker
Integrating Jaeger with GoFrame
Creating and managing traces
Handling errors gracefully
Visualizing and analyzing traces

Prerequisites

Basic knowledge of Go and microservices
Docker installed on your machine
A GoFrame project (or willingness to start one!)

Getting Started with Jaeger 🐋

First things first, let's get Jaeger up and running. The easiest way is using Docker:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.21

This command launches a complete Jaeger setup in one container. Pretty neat, right? 👌

Setting Up the Tracer in GoFrame 🔧

Let's dive into a complete setup example. Here's how to configure Jaeger with different sampling strategies and options:

First, grab the Jaeger client library:

go get github.com/uber/jaeger-client-go

Now, let's set up our tracer. Here's a simple initialization:

package main

import (
    "github.com/opentracing/opentracing-go"
    "github.com/uber/jaeger-client-go"
    "github.com/uber/jaeger-client-go/config"
)

func main() {
    // Get config from environment
    cfg, _ := config.FromEnv()

    // Create the tracer
    tracer, closer, _ := cfg.NewTracer(config.Logger(jaeger.StdLogger))
    defer closer.Close()

    // Set as global tracer
    opentracing.SetGlobalTracer(tracer)

    // Start your server...
}

// A more detailed configuration example
func initJaeger(service string) (opentracing.Tracer, io.Closer, error) {
    cfg := &config.Configuration{
        ServiceName: service,
        Sampler: &config.SamplerConfig{
            Type:  "const",
            Param: 1,
        },
        Reporter: &config.ReporterConfig{
            LogSpans:           true,
            LocalAgentHostPort: "localhost:6831",
            BufferFlushInterval: 1 * time.Second,
            QueueSize:          1000,
        },
        Tags: []opentracing.Tag{
            {Key: "environment", Value: "development"},
            {Key: "version", Value: "1.0.0"},
        },
    }

    tracer, closer, err := cfg.NewTracer(
        config.Logger(jaeger.StdLogger),
        config.ZipkinSharedRPCSpan(true),
    )

    if err != nil {
        return nil, nil, err
    }

    return tracer, closer, nil
}

Complete Tracing Setup Example 🎯

Let's look at a complete example of how to set up tracing in your application:

package main

import (
    "github.com/gogf/gf/v2/frame/g"
    "github.com/gogf/gf/v2/net/ghttp"
    "github.com/opentracing/opentracing-go"
)

type App struct {
    Server *ghttp.Server
    Tracer opentracing.Tracer
}

func NewApp() (*App, error) {
    // Initialize Jaeger
    tracer, closer, err := initJaeger("my-service")
    if err != nil {
        return nil, err
    }
    defer closer.Close()

    // Create server
    server := g.Server()

    app := &App{
        Server: server,
        Tracer: tracer,
    }

    // Register middleware and routes
    server.Use(app.TracingMiddleware)
    server.Group("/api", func(group *ghttp.RouterGroup) {
        group.POST("/orders", app.HandleOrder)
        group.GET("/orders/:id", app.GetOrder)
    })

    return app, nil
}

// Complete example of a traced HTTP client
func (app *App) makeTracedRequest(parentSpan opentracing.Span, url string) error {
    // Create a child span
    span := app.Tracer.StartSpan(
        "http_request",
        opentracing.ChildOf(parentSpan.Context()),
    )
    defer span.Finish()

    // Create request
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        span.SetTag("error", true)
        span.LogKV("event", "error", "message", err.Error())
        return err
    }

    // Inject tracing headers
    carrier := opentracing.HTTPHeadersCarrier(req.Header)
    err = app.Tracer.Inject(span.Context(), opentracing.HTTPHeaders, carrier)
    if err != nil {
        span.SetTag("error", true)
        span.LogKV("event", "error", "message", "failed to inject tracing headers")
        return err
    }

    // Make the request
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        span.SetTag("error", true)
        span.LogKV("event", "error", "message", err.Error())
        return err
    }
    defer resp.Body.Close()

    // Add response info to span
    span.SetTag("http.status_code", resp.StatusCode)

    return nil
}

Creating Your First Trace 📝

Let's create a middleware to trace all incoming requests:

func TracingMiddleware(r *ghttp.Request) {
    // Extract any existing trace from headers
    spanCtx, _ := opentracing.GlobalTracer().Extract(
        opentracing.HTTPHeaders,
        opentracing.HTTPHeadersCarrier(r.Request.Header),
    )

    // Start a new span
    span := opentracing.GlobalTracer().StartSpan(
        r.URL.Path,
        opentracing.ChildOf(spanCtx),
    )
    defer span.Finish()

    // Pass the span through headers
    opentracing.GlobalTracer().Inject(
        span.Context(),
        opentracing.HTTPHeaders,
        opentracing.HTTPHeadersCarrier(r.Request.Header),
    )

    // Add to request context
    r.SetCtx(opentracing.ContextWithSpan(r.Context(), span))

    r.Middleware.Next()
}

Adding Business Logic Traces 💼

Now for the fun part - tracing your actual business logic:

func ProcessOrder(r *ghttp.Request) {
    // Get the current span
    span := opentracing.SpanFromContext(r.Context())

    // Add some business context
    span.SetTag("order_id", "12345")

    // Log important events
    span.LogKV("event", "order_received")

    // Your business logic here...
    processPayment()
    updateInventory()
    sendConfirmation()

    span.LogKV("event", "order_completed")
}

Error Handling Like a Pro 🛠️

Let's make our error handling more traceable:

// Define custom errors
type MyError struct {
    Code    int
    Message string
}

func (e *MyError) Error() string {
    return fmt.Sprintf("error code: %d, message: %s", e.Code, e.Message)
}

// Error handling in your handlers
func HandleOrder(r *ghttp.Request) {
    span := opentracing.SpanFromContext(r.Context())

    err := processOrder()
    if err != nil {
        // Mark the span as failed
        span.SetTag("error", true)
        span.SetTag("error.code", err.(*MyError).Code)

        // Log detailed error info
        span.LogKV(
            "event", "error",
            "message", err.Error(),
            "stack", string(debug.Stack()),
        )

        // Handle the error appropriately
        r.Response.WriteJson(g.Map{
            "error": err.Error(),
        })
        return
    }
}

Viewing Your Traces 👀

Once everything is set up, you can view your traces at http://localhost:16686. The Jaeger UI lets you:

Search for traces across services
View detailed timing information
Analyze error patterns
Export traces for further analysis

Pro Tips 💡

Use Meaningful Span Names: Instead of generic names like "process", use descriptive names like "order_processing" or "payment_validation".
Add Relevant Tags: Tags help filter and analyze traces. Add tags for things like:
- User IDs
- Request IDs
- Environment information
- Business-specific identifiers
Log Key Events: Use LogKV to mark important points in your process:

   span.LogKV("event", "cache_miss", "key", "user:123")

Common Pitfalls to Avoid ⚠️

Memory Leaks: Always remember to call span.Finish()
Over-instrumentation: Don't trace everything; focus on important operations
Missing Context: Always propagate context through your service calls

Advanced Troubleshooting Guide 🔍

1. Common Issues and Solutions

Missing Traces

// Problem: Traces not showing up in Jaeger UI
// Solution: Check sampling configuration
cfg := &config.Configuration{
    Sampler: &config.SamplerConfig{
        Type:  "const",    // Try different sampling strategies
        Param: 1,          // 1 = sample all requests
    },
}

// Verify spans are being created
span := opentracing.SpanFromContext(ctx)
if span == nil {
    // No span in context - check your middleware
    log.Println("No span found in context")
}

Context Propagation Issues

// Problem: Broken trace chains
// Solution: Properly propagate context through your application

// Wrong ❌
func (s *Service) ProcessOrder(orderID string) error {
    // Starting new trace chain
    span := tracer.StartSpan("process_order")
    defer span.Finish()

    // ... processing
}

// Correct ✅
func (s *Service) ProcessOrder(ctx context.Context, orderID string) error {
    span, ctx := opentracing.StartSpanFromContext(ctx, "process_order")
    defer span.Finish()

    // Pass ctx to other functions
    return s.updateInventory(ctx, orderID)
}

Performance Issues

// Problem: Too many spans affecting performance
// Solution: Use batch processing for spans

cfg := &config.Configuration{
    Reporter: &config.ReporterConfig{
        QueueSize:          1000,    // Buffer size
        BufferFlushInterval: 1 * time.Second,
        LogSpans:           true,    // Set to false in production
    },
}

2. Debugging Tools

// Debug span creation
func debugSpan(span opentracing.Span) {
    // Get span context
    spanContext, ok := span.Context().(jaeger.SpanContext)
    if !ok {
        log.Println("Not a Jaeger span")
        return
    }

    // Print span details
    log.Printf("Trace ID: %s", spanContext.TraceID())
    log.Printf("Span ID: %s", spanContext.SpanID())
    log.Printf("Parent ID: %s", spanContext.ParentID())
}

// Monitor span metrics
type SpanMetrics struct {
    TotalSpans     int64
    ErrorSpans     int64
    AverageLatency time.Duration
}

func collectSpanMetrics(span opentracing.Span) *SpanMetrics {
    metrics := &SpanMetrics{}

    // Add your metric collection logic
    if span.BaggageItem("error") != "" {
        atomic.AddInt64(&metrics.ErrorSpans, 1)
    }

    return metrics
}

3. Best Practices for Problem Resolution

1. Validate Configuration

func validateJaegerConfig(cfg *config.Configuration) error {
    if cfg.ServiceName == "" {
        return errors.New("service name is required")
    }

    if cfg.Reporter.LocalAgentHostPort == "" {
        return errors.New("reporter host:port is required")
    }

    return nil
}

2. Implement Health Checks

func jaegerHealthCheck() error {
    span := opentracing.GlobalTracer().StartSpan("health_check")
    defer span.Finish()

    carrier := opentracing.HTTPHeadersCarrier{}
    err := opentracing.GlobalTracer().Inject(
        span.Context(),
        opentracing.HTTPHeaders,
        carrier,
    )

    if err != nil {
        return fmt.Errorf("jaeger injection failed: %v", err)
    }

    return nil
}

3. Monitor Trace Quality

type TraceQuality struct {
    MissingParentSpans int
    BrokenChains       int
    HighLatencyTraces  int
}

func monitorTraceQuality(span opentracing.Span) *TraceQuality {
    quality := &TraceQuality{}

    // Check for parent span
    if span.BaggageItem("parent_id") == "" {
        quality.MissingParentSpans++
    }

    // Check latency
    if duration, ok := span.BaggageItem("duration"); ok {
        if d, err := time.ParseDuration(duration); err == nil {
            if d > 1*time.Second {
                quality.HighLatencyTraces++
            }
        }
    }

    return quality
}

Wrapping Up 🎉

Distributed tracing with Jaeger and GoFrame gives you x-ray vision into your microservices. You can:

Track requests across services
Identify performance bottlenecks
Debug issues faster
Understand system behavior

What's Next?

Explore Jaeger sampling strategies
Add metrics and logging
Implement trace-based alerts

Found this helpful? Follow me for more Go tips and tricks! And don't forget to drop a comment if you have questions or suggestions! 🚀

Resources:

DEV Community