OpenTelemetry integration with Dynatrace

Credit Decision Application Flow (Highest Priority)

• Track complete decision flow from application submission to final decision
• Measure processing time at each decision checkpoint
• Monitor credit bureau API response times
• Track decision rules execution time
• Key metrics to collect:
• Time from submission to final decision
• Processing time per decision rule
• Number of applications in each state
• Failure rates at each checkpoint

Cross-Service Transaction Monitoring

• Since your services are synchronous, track the entire request chain
• Monitor service dependencies and their health
• Identify service communication bottlenecks
• Key areas to instrument:
• Inter-service REST calls
• Service response times
• Error rates between services
• Request payload sizes

MongoDB Performance Optimization

• Track query patterns across all services
• Monitor document access patterns
• Key metrics to collect:
• Query execution times
• Frequently accessed collections
• Slow queries (> 100ms)
• Index usage statistics
• Connection pool utilization

Span: CreditApplication
|- Attributes:
   |- application_id
   |- customer_id
   |- current_stage
   |- decision_outcome
|- Child Spans:
   |- DocumentValidation
   |- CreditCheck
   |- DecisionEngine
   |- CardIssuance

<properties>
    <opentelemetry.version>1.32.0</opentelemetry.version>
</properties>

<dependencies>
    <!-- OpenTelemetry API -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>${opentelemetry.version}</version>
    </dependency>

    <!-- OpenTelemetry SDK -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk</artifactId>
        <version>${opentelemetry.version}</version>
    </dependency>

    <!-- OpenTelemetry OTLP Exporter -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
        <version>${opentelemetry.version}</dependency>

    <!-- Spring Boot Starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
</dependencies>

@Configuration
public class OpenTelemetryConfig {

    @Value("${dynatrace.endpoint}")
    private String dynatraceEndpoint;

    @Bean
    public OpenTelemetry openTelemetry() {
        Resource resource = Resource.getDefault()
            .merge(Resource.create(Attributes.of(
                ResourceAttributes.SERVICE_NAME, "credit-check-service",
                ResourceAttributes.SERVICE_VERSION, "1.0"
            )));

        SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
            .setResource(resource)
            .addSpanProcessor(BatchSpanProcessor.builder(
                OtlpGrpcSpanExporter.builder()
                    .setEndpoint(dynatraceEndpoint)
                    .build())
                .build())
            .build();

        return OpenTelemetrySdk.builder()
            .setTracerProvider(sdkTracerProvider)
            .buildAndRegisterGlobal();
    }
}

@Service
@Slf4j
public class CreditCheckService {

    private final Tracer tracer;
    private final CreditBureauClient creditBureauClient;

    public CreditCheckService(OpenTelemetry openTelemetry, CreditBureauClient creditBureauClient) {
        this.tracer = openTelemetry.getTracer("credit-check-service");
        this.creditBureauClient = creditBureauClient;
    }

    public CreditCheckResult performCreditCheck(String applicationId, String customerId) {
        Span span = tracer.spanBuilder("credit.check")
            .setAttribute("application.id", applicationId)
            .setAttribute("customer.id", customerId)
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            long startTime = System.currentTimeMillis();

            // Create child span for bureau call
            Span bureauCallSpan = tracer.spanBuilder("credit.bureau.call")
                .setAttribute("bureau.name", "ExampleBureau")
                .startSpan();

            CreditBureauResponse bureauResponse;
            try {
                bureauResponse = creditBureauClient.getCreditScore(customerId);
                bureauCallSpan.setAttribute("credit.score", bureauResponse.getCreditScore());
                bureauCallSpan.setStatus(StatusCode.OK);
            } catch (Exception e) {
                bureauCallSpan.setStatus(StatusCode.ERROR);
                bureauCallSpan.recordException(e);
                throw e;
            } finally {
                bureauCallSpan.end();
            }

            // Add important metrics to main span
            span.setAttribute("credit.score", bureauResponse.getCreditScore());
            span.setAttribute("processing.time_ms", System.currentTimeMillis() - startTime);
            span.setAttribute("check.status", "COMPLETED");

            return new CreditCheckResult(bureauResponse);
        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR, e.getMessage());
            span.recordException(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

dynatrace.endpoint=https://your-environment-id.live.dynatrace.com/api/v2/otlp/v1/traces

Parent Span (Application Process):

• Basic Info: application.id, customer.id, product.type
• Application Details: application.type, channel, priority
• Customer Info: customer.segment
• Timing: timestamp.received, process.start.time, process.duration_ms
• Outcome: application.status, decision.outcome
• Error Tracking: error.occurred, error.reason, error.code

Child Span (Credit Check):

• Basic Info: application.id, customer.id, check.type
• Bureau Info: bureau.name, bureau.request.id
• Timing: check.start.time, check.duration_ms
• Results: credit.score, credit.score.band
• Decision: decision.outcome, decision.factors.count
• Performance: check.duration_ms
• Additional Info: additional.checks.required

Nested Child Span (Bureau Call):

• Request Info: bureau.request.id, bureau.request.type
• Response: credit.score, bureau.response.status
• Performance: bureau.response.time_ms
• Error Handling: error.type, error.message

These attributes provide:

• Complete traceability of the application process
• Performance metrics at each stage
• Business metrics (credit scores, decisions)
• Error tracking and debugging information
• Operational metrics for monitoring and alerting