Observe and Troubleshoot Apps on Azure

1. The Three Pillars of Observability

1. Distributed Traces — full path of a request through all services with timing. Answers: "Where is the latency?" Primary AI-200 focus.
2. Metrics — aggregate numbers over time (request rate, error rate, p95 latency). Answers: "Is something trending wrong?"
3. Logs — timestamped discrete events. Answers: "Why did this specific operation fail?"

Together: Metrics tell you something changed, Traces tell you where, Logs tell you why.

2. Traces and Spans Anatomy

A trace = the complete record of one request across all services. A span = one named, timed unit of work within that trace.

1. Trace ID — 128-bit hex, shared by ALL spans in one request journey
2. Span ID — 64-bit hex, unique to this specific operation
3. Parent Span ID — links this span to its caller (forms a tree/waterfall)
4. Name — operation name: HTTP GET /api/chat, vector_search, llm_inference
5. Start/End timestamps — exact duration in milliseconds
6. Attributes — key-value metadata: HTTP method, model name, token count
7. Status — OK or ERROR

3. W3C traceparent — How Spans Connect Across Services

When Service A calls Service B via HTTP, it passes the trace context in a header so B's spans link to A's span in the same trace:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ version  ^^ 32-char trace ID (shared!)    ^^ 16-char span ID  ^^ flags

Without this header: each service creates a disconnected trace. With it: all services' spans form one unified waterfall.

4. OpenTelemetry → Application Insights Term Mapping

1. Install and Initialize

pip install azure-monitor-opentelemetry

from azure.monitor.opentelemetry import configure_azure_monitor

# One call — sets up traces, metrics, and logs
# Reads APPLICATIONINSIGHTS_CONNECTION_STRING from environment variable
configure_azure_monitor()

This single call initializes trace, metric, and log providers — all exporting to Application Insights. No separate configuration needed for each.

2. Connection String

# Recommended: environment variable
export APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=...;IngestionEndpoint=..."

# Code-based (avoid in production)
configure_azure_monitor(connection_string="InstrumentationKey=...")

3. What the Distro Auto-Instruments (No Code Changes)

1. requests / urllib3 — outgoing HTTP calls to OpenAI, vector DBs → appear as dependency spans
2. Flask / Django / FastAPI — incoming HTTP requests → appear as server spans (requests table)
3. psycopg2 — PostgreSQL queries → dependencies table
4. Azure SDK — Key Vault, Storage, Service Bus calls → dependencies table
5. Python logging module — standard log calls flow to traces table automatically

4. Cloud Role Name — Multi-Service Identification

Without unique cloud role names, all services appear as a single node on the Application Map. You lose visibility into inter-service latency.

from opentelemetry.sdk.resources import Resource

configure_azure_monitor(
    resource=Resource.create({
        "service.name": "embedding-service",
        "service.namespace": "rag-pipeline"
    })
)
# App Map shows node: "rag-pipeline.embedding-service"

# Or via env var (no code change):
export OTEL_SERVICE_NAME=embedding-service
export OTEL_RESOURCE_ATTRIBUTES=service.namespace=rag-pipeline

1. Custom Span for Model Inference

from opentelemetry import trace

tracer = trace.get_tracer("rag-pipeline")

def run_inference(prompt: str) -> dict:
    with tracer.start_as_current_span("llm_inference") as span:
        span.set_attribute("model", "gpt-4o")
        span.set_attribute("prompt_tokens", len(prompt.split()))

        result = call_openai_api(prompt)

        span.set_attribute("completion_tokens", result.usage.completion_tokens)
        span.set_attribute("total_tokens", result.usage.total_tokens)
        return result

2. Record Errors in Spans

from opentelemetry.trace import StatusCode

with tracer.start_as_current_span("vector_search") as span:
    try:
        results = vector_db.search(embedding, top_k=5)
        span.set_attribute("results_count", len(results))
    except Exception as e:
        span.set_status(StatusCode.ERROR, str(e))
        span.record_exception(e)   # Adds stack trace to span
        raise

3. Span Kinds and Their Mappings

1. SpanKind.SERVER — incoming request handler → requests table in App Insights
2. SpanKind.CLIENT — outgoing call to external service → dependencies table
3. SpanKind.INTERNAL — internal processing (chunking, ranking) → dependencies table

1. Application Map

Visualizes the topology of your AI pipeline. Shows each service as a node, with call volume and failure rates on edges. Requires each service to have a unique cloud role name. Immediately reveals which service is the bottleneck or source of errors.

2. KQL Queries for AI Pipeline Analysis

// 1. Find slowest operations in last hour
requests
| where timestamp > ago(1h)
| summarize avg(duration), percentile(duration, 95) by name
| order by percentile_duration_95 desc

// 2. Find slow OpenAI calls
dependencies
| where name contains "openai"
| where duration > 5000    // over 5 seconds
| project timestamp, name, duration, customDimensions

// 3. Query by custom span attribute (model name)
requests
| where customDimensions.model == "gpt-4o"
| summarize count(), avg(duration) by bin(timestamp, 5m)

// 4. Error rate by service (cloud_RoleName)
requests
| where timestamp > ago(1h)
| summarize total=count(), errors=countif(success==false) by cloud_RoleName
| extend error_rate = round(100.0 * errors / total, 2)

Essential Query Patterns

// Failed requests in last hour
requests
| where timestamp > ago(1h)
| where success == false
| project timestamp, name, resultCode, duration

// P95 latency by operation (bin = time bucket)
requests
| where timestamp > ago(24h)
| summarize p95=percentile(duration, 95) by
    bin(timestamp, 1h), name
| render timechart

// Join requests with exceptions
requests
| where timestamp > ago(1h) and success == false
| join kind=leftouter exceptions
    on operation_Id
| project timestamp, name, type, outerMessage

// Track OpenAI call latency via dependencies
dependencies
| where timestamp > ago(6h)
| where target contains "openai"
| summarize avg_ms=avg(duration), calls=count()
    by bin(timestamp, 30m)
| render timechart

Alert Rule → Action Group → Notification

# Create action group (email notification)
az monitor action-group create \
  --name ai-alerts-ag \
  --resource-group rg \
  --short-name aialerts \
  --email-receiver name=oncall [email protected]

# Create log alert — fires when error rate exceeds threshold
az monitor scheduled-query create \
  --name high-error-rate \
  --resource-group rg \
  --scopes $APP_INSIGHTS_ID \
  --condition-query "requests | where success==false
    | summarize count() by bin(timestamp,5m)
    | where count_ > 10" \
  --condition-threshold 0 \
  --condition-operator GreaterThan \
  --evaluation-frequency 5m \
  --window-size 5m \
  --action-groups $ACTION_GROUP_ID

Track AI-Specific Metrics

from applicationinsights import TelemetryClient

tc = TelemetryClient("YOUR_INSTRUMENTATION_KEY")

# Track token usage as custom metric
tc.track_metric("openai_tokens_used", total_tokens,
    properties={"model": model, "operation": "embed"})

# Track cache hit/miss ratio
tc.track_metric("semantic_cache_hit", 1 if cache_hit else 0)
tc.flush()

# Query in KQL:
# customMetrics
# | where name == "openai_tokens_used"
# | summarize sum(value) by bin(timestamp, 1h)