APP-WIDE-EVALS.md

Enhancing Telemetry Across the Entire Application Lifecycle

Overview

In the context of integrating Langfuse with your legacy Stripes Framework Java application, you can extend telemetry beyond AI-specific pipelines to capture data flow across the entire lifecycle of a request or process. This includes non-AI components (e.g., database queries, business logic, user input processing) intertwined with AI elements (e.g., LLM calls, retrieval).

The key enabler is OpenTelemetry (OTel), which Langfuse uses for tracing. OTel allows you to instrument any function or method—AI-related or not—with spans (observations in Langfuse). This creates a unified trace that visualizes the end-to-end flow: inputs, outputs, timings, errors, and custom metadata.

Benefits:

• Full Visibility: Track data in/out, bottlenecks, and dependencies across non-AI → AI → non-AI sequences.
• No Major Refactoring: Use annotations or manual spans on existing methods.
• Correlation: All spans link to a parent trace, showing the complete lifecycle (e.g., from HTTP request to response).
• Analytics in Langfuse: View traces with timings, attributes (e.g., input params, output values), and errors for debugging and optimization.

This approach works for any Java flow, such as a Stripes ActionBean handling a user request that involves validation (non-AI), AI generation, and post-processing (non-AI).

Prerequisites

• Completed the basic setup from the previous guide: Langfuse SDK and OTel dependencies in pom.xml.
• OTel Java Agent configured and running (for auto-instrumentation and exporter to Langfuse).
• Familiarity with OTel concepts: Traces (overall flow), Spans (individual operations), Attributes (key-value metadata).

If not set up, refer to Step 5 in the previous guide for OTel agent configuration.

Step 1: Instrumenting Non-AI Functions with Annotations

Use OTel's @WithSpan annotation on any method to automatically create a span. This captures execution time, exceptions, and allows adding custom attributes for data in/out.

Example: Instrumenting a Non-AI Utility Method

Suppose you have a UserValidationService that processes input before an AI call—non-AI logic like checking user data.

import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.api.trace.Span;

public class UserValidationService {

    @WithSpan("user_validation")  // Span name for Langfuse observation
    public boolean validateUserInput(String userInput, int userId) {
        Span span = Span.current();  // Access the current span
        
        // Add input data as attributes (visible in Langfuse)
        span.setAttribute("input.user_input_length", userInput.length());
        span.setAttribute("input.user_id", userId);
        
        // Your non-AI logic (e.g., regex check, DB query)
        boolean isValid = userInput.matches("^[a-zA-Z0-9]+$");  // Example validation
        
        // Add output data
        span.setAttribute("output.is_valid", isValid);
        
        if (!isValid) {
            span.recordException(new IllegalArgumentException("Invalid input"));  // Log errors
        }
        
        return isValid;
    }
}

Integration in ActionBean:

Call this from your AI-powered ActionBean to include it in the trace.

@DefaultHandler
public Resolution handleChat() {
    // Non-AI pre-processing
    boolean valid = userValidationService.validateUserInput(getUserInput(), getUserId());
    if (!valid) {
        return new ErrorResolution(400, "Invalid input");
    }
    
    // AI call (already traced)
    String aiResult = aiService.callLLM(promptTemplate, getContext());
    
    // Non-AI post-processing (instrument similarly if needed)
    String processedResult = postProcess(aiResult);
    
    return new ForwardResolution("/chat.jsp");
}

What Happens:

• When validateUserInput runs, a child span is created under the parent trace (e.g., the HTTP request trace if auto-instrumented).
• In Langfuse: See the full flow with spans like "user_validation" → "knowledge_retrieval" → "llm_call", including data in/out.

Step 2: Manual Spans for Fine-Grained Control

For code without methods (e.g., inline logic) or more control, use OTel's manual API to start/end spans.

Example: Manual Span in a Complex Flow

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.Span;

public class DataProcessingService {
    private static final Tracer tracer = GlobalOpenTelemetry.getTracer("my-app-tracer");

    public String processData(String rawData) {
        // Start a manual span
        Span span = tracer.spanBuilder("data_processing").startSpan();
        try {
            span.setAttribute("input.raw_data", rawData);  // Log input
            
            // Non-AI processing logic
            String cleanedData = rawData.trim().toUpperCase();
            
            span.setAttribute("output.cleaned_data", cleanedData);  // Log output
            
            return cleanedData;
        } catch (Exception e) {
            span.recordException(e);
            throw e;
        } finally {
            span.end();  // Always end the span
        }
    }
}

Use Case: Wrap legacy code blocks or third-party calls that aren't annotatable.

Step 3: Auto-Instrumenting the Entire Lifecycle

Leverage the OTel Java Agent for zero-code instrumentation of common libraries and frameworks:

• HTTP/Stripes Requests: Automatically traces incoming requests (e.g., ActionBean executions).
• Databases/JDBC: Traces queries (e.g., SQL statements, params).
• Other Libs: Supports OkHttp (for API calls), Spring, etc.—add if your app uses them.

Configuration Tips:

• In JVM args: -Dotel.instrumentation.jdbc.enabled=true for DB tracing.
• Export all to Langfuse: Ensure the exporter is set (as in previous guide).
• Sampling: Use -Dotel.traces.sampler=always_on for full traces during dev; adjust for prod.

This captures non-AI parts like servlet handling → validation → AI → response rendering without annotations.

Step 4: Capturing Data In/Out and Custom Metrics

• Attributes: Always add key data via span.setAttribute("key", value). Supports strings, numbers, booleans, arrays.
  • Inputs: Params, user data.
  • Outputs: Results, transformed data.
• Events: Log intermediate steps with span.addEvent("milestone", Attributes.of("detail", "value")).
• Errors: Use span.recordException(e) to capture stack traces.
• Linking Traces: For distributed flows (e.g., microservices), propagate context via OTel's Context.

In Langfuse, filter traces by attributes for insights (e.g., slow validations).

Step 5: Monitoring and Analysis in Langfuse

• Dashboard: View end-to-end traces with all spans (AI and non-AI).
• Queries: Search by attributes (e.g., "user_id=123").
• Scores: Annotate non-AI spans too (e.g., score validation accuracy).
• Alerts: Set up for high-latency spans or errors.

Best Practices

• Granularity: Instrument key methods; avoid over-tracing minor utils to prevent noise.
• Privacy: Anonymize sensitive data in attributes (e.g., hash user IDs).
• Testing: Run sample flows and verify traces in Langfuse.
• Performance: Spans add <1% overhead; monitor in prod.
• Extensions: For metrics (e.g., counters), add OTel Metrics API; Langfuse supports it partially.