Distributed Tracing and Application Map

What Is Distributed Tracing?

Distributed tracing tracks requests as they flow through multiple services in a distributed system. It provides visibility into the complete request lifecycle — from the initial API call through APIM, Logic Apps, backend services, and databases.

Microsoft Reference: Distributed tracing in Azure

W3C Trace Context

The W3C Trace Context standard defines HTTP headers for propagating trace context:

Header	Description	Example
traceparent	Trace ID, span ID, and flags	00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate	Vendor-specific trace data	azure=eyJhbGciOiJIUzI1NiJ9

Traceparent Format

{version}-{trace-id}-{parent-id}-{trace-flags}
  00     -  32 hex  -  16 hex  -   2 hex

Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Field	Description
version	Always 00
trace-id	32-character hex — unique per end-to-end transaction
parent-id	16-character hex — unique per span (service hop)
trace-flags	01 = sampled, 00 = not sampled

Enabling Distributed Tracing

API Management

resource apimDiagnostic 'Microsoft.ApiManagement/service/diagnostics@2023-05-01-preview' = {
  parent: apim
  name: 'applicationinsights'
  properties: {
    loggerId: apimLogger.id
    httpCorrelationProtocol: 'W3C'  // Enable W3C Trace Context
    alwaysLog: 'allErrors'
    sampling: {
      samplingType: 'fixed'
      percentage: 100
    }
    frontend: {
      request: {
        headers: ['traceparent', 'tracestate', 'X-Correlation-Id']
      }
    }
  }
}

Logic Apps (Standard)

Standard Logic Apps automatically participate in distributed tracing when Application Insights is configured:

// host.json
{
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Exception"
      }
    }
  }
}

Logic Apps (Consumption)

For Consumption Logic Apps, enable diagnostic settings and use tracked properties for correlation:

{
  "trackedProperties": {
    "correlationId": "@triggerOutputs()?['headers']?['traceparent']",
    "operationId": "@workflow()?['run']?['name']"
  }
}

Azure Functions

// Functions automatically propagate W3C Trace Context when
// Application Insights is configured via APPLICATIONINSIGHTS_CONNECTION_STRING

Trace Flow Example

1. Client → APIM
   traceparent: 00-abc123...-span1-01

2. APIM → Logic App (HTTP trigger)
   traceparent: 00-abc123...-span2-01
   (APIM creates new span, preserves trace-id)

3. Logic App → SQL Database
   traceparent: 00-abc123...-span3-01
   (Logic App creates new span)

4. Logic App → External API
   traceparent: 00-abc123...-span4-01

5. Logic App → Service Bus
   traceparent: 00-abc123...-span5-01

All spans share the same trace-id (abc123...)

Application Map

Application Map provides a visual topology of your distributed application, showing:

• Service components and their dependencies
• Request volumes and error rates per component
• Average latency between components
• Health indicators (green/yellow/red)

Microsoft Reference: Application Map

Accessing Application Map

1. Open Application Insights in the Azure Portal
2. Select Application Map from the left menu
3. View the auto-discovered topology
4. Click any component to drill into details
5. Click connections to see dependency metrics

What Application Map Shows

┌─────────────┐    500 req/min    ┌─────────────────┐
│   Client     │ ──────────────→  │  API Management  │
│  (browser)   │    avg 150ms     │   (gateway)      │
└─────────────┘                   └────────┬────────┘
                                           │ 200 req/min
                                           ↓ avg 80ms
                                  ┌─────────────────┐
                                  │   Logic App      │
                                  │  (Standard)      │
                                  └───┬──────┬───────┘
                             120/min  │      │ 200/min
                             avg 30ms │      │ avg 15ms
                                      ↓      ↓
                              ┌────────┐  ┌──────────┐
                              │ SQL DB │  │ Service  │
                              │        │  │  Bus     │
                              └────────┘  └──────────┘

Customising Application Map

Cloud Role Names

Set cloud role names to identify services clearly:

// For Logic Apps / Functions (host.json)
{
  "logging": {
    "applicationInsights": {
      "samplingSettings": { "isEnabled": true }
    }
  }
}

// Set via app settings
{
  name: 'WEBSITE_CLOUD_ROLENAME'
  value: 'Order Processing Logic App'
}

Transaction Search

Find specific transactions across all services:

Search by Operation ID

// Find all telemetry for a specific operation
union requests, dependencies, exceptions, traces, customEvents
| where operation_Id == "abc123xyz"
| project timestamp, itemType, name, duration, success, resultCode, message
| order by timestamp asc

Search by Custom Property

// Find by order ID across all services
union requests, dependencies, traces
| where customDimensions.orderId == "ORD-001234"
| project timestamp, cloud_RoleName, itemType, name, duration
| order by timestamp asc

Microsoft Reference: Transaction search

End-to-End Transaction Details

The end-to-end transaction view shows:

1. Timeline — Waterfall chart of all operations
2. Duration — Time spent in each service
3. Dependencies — Outgoing calls from each service
4. Exceptions — Errors at any point in the chain
5. Properties — Custom dimensions and context

Accessing Transaction Details

1. Open Application Insights → Transaction search
2. Filter by time range and telemetry type
3. Click any request to see the full transaction
4. View the timeline showing all spans
5. Click any span for detailed properties

Correlation in Logic Apps Workflows

Passing Correlation IDs

Ensure correlation IDs flow through all workflow steps:

{
  "actions": {
    "Initialize_CorrelationId": {
      "type": "InitializeVariable",
      "inputs": {
        "variables": [
          {
            "name": "correlationId",
            "type": "string",
            "value": "@coalesce(triggerOutputs()?['headers']?['X-Correlation-Id'], triggerOutputs()?['headers']?['traceparent'], guid())"
          }
        ]
      }
    },
    "Call_Backend": {
      "type": "Http",
      "inputs": {
        "method": "POST",
        "uri": "https://api.example.com/process",
        "headers": {
          "X-Correlation-Id": "@variables('correlationId')",
          "traceparent": "@variables('correlationId')"
        }
      }
    }
  }
}

Multi-Workflow Correlation

When a Logic App calls another Logic App, maintain the trace context:

{
  "Call_child_workflow": {
    "type": "Http",
    "inputs": {
      "method": "POST",
      "uri": "@{listCallbackUrl()}",
      "headers": {
        "X-Correlation-Id": "@variables('correlationId')",
        "traceparent": "@triggerOutputs()?['headers']?['traceparent']"
      },
      "body": "@triggerBody()"
    }
  }
}

Querying Distributed Traces

End-to-End Latency by Service

// Latency breakdown across services
requests
| where timestamp > ago(1h)
| join kind=inner (
    dependencies
    | where timestamp > ago(1h)
  ) on operation_Id
| summarize
    TotalDuration = max(requests_duration),
    BackendDuration = sum(dependencies_duration)
  by operation_Id, requests_name
| extend OverheadDuration = TotalDuration - BackendDuration
| summarize
    AvgTotal = avg(TotalDuration),
    AvgBackend = avg(BackendDuration),
    AvgOverhead = avg(OverheadDuration)
  by requests_name

Service Dependency Health

dependencies
| where timestamp > ago(1h)
| summarize
    Calls = count(),
    FailRate = round(100.0 * countif(success == false) / count(), 2),
    AvgDuration = round(avg(duration), 0),
    P95Duration = round(percentile(duration, 95), 0)
  by target, type, cloud_RoleName
| order by FailRate desc

Slowest End-to-End Transactions

requests
| where timestamp > ago(1h)
| where duration > 5000  // > 5 seconds
| project timestamp, operation_Id, name, duration, success,
          cloud_RoleName, resultCode
| order by duration desc
| take 20

Cross-Service Error Correlation

// Find upstream requests that caused downstream failures
let failedDeps = dependencies
| where timestamp > ago(1h)
| where success == false
| project operation_Id, target, resultCode, depTimestamp = timestamp;

requests
| where timestamp > ago(1h)
| join kind=inner failedDeps on operation_Id
| project timestamp, requestName = name, depTarget = target,
          depResultCode = resultCode, requestDuration = duration
| order by timestamp desc

Live Metrics Stream

Real-time monitoring dashboard showing:

• Incoming requests — Rate and duration
• Outgoing requests — Dependency calls
• Overall health — Success/failure rates
• Server metrics — CPU, memory, request rate
• Sample telemetry — Live request and failure details

Accessing Live Metrics

1. Open Application Insights
2. Select Live Metrics from the left menu
3. View real-time data streaming (no delay)

Microsoft Reference: Live Metrics Stream

Performance Profiler

For Standard Logic Apps and Azure Functions, use the Application Insights Profiler:

• .NET Profiler — Code-level performance analysis
• Snapshot Debugger — Capture debug snapshots on exceptions
• Code-level insights — Identify hot paths and bottlenecks

Microsoft Reference: Application Insights Profiler

Best Practices

1. Enable W3C Trace Context on all services for proper correlation
2. Use a shared Application Insights instance for related services to see the full Application Map
3. Set meaningful cloud role names so services are identifiable in the map
4. Propagate correlation headers in Logic App HTTP actions
5. Exclude Exception from sampling to ensure all errors are captured
6. Use custom dimensions to add business context (order ID, customer ID)
7. Monitor Live Metrics during deployments and incident response
8. Set up availability tests for critical endpoints
9. Review Application Map regularly to understand dependency health
10. Archive trace data for compliance and post-incident analysis

Official Microsoft Resources

• Distributed tracing in Azure
• Application Map
• Transaction search
• Live Metrics Stream
• W3C Trace Context standard
• Application Insights Profiler
• Telemetry correlation