Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

Dash0 Lambda Extension

An extension for capturing observability data from AWS Lambda invocations and shipping to Dash0.

Table of Contents

This extension has five main functionalities:

  1. Enable auto-instrumentation for supported runtimes, which currently include Python, Node, Java.
  2. Receive traces from auto/manual instrumentations, enrich with data acquired in the extension, and send to Dash0.
  3. Detect runtime errors such as timeout or out of memory and create synthetic traces for them
  4. Collect all logs and send to Dash0, correlated with the trace id of the invocation.
  5. Create metrics for invocation duration, cold start duration, billed duration, and memory used.

Layer ARNs

See the release page for the latest ARNs of the extension layers for each runtime.

Supported Regions

The extension layers are published to the following AWS regions:

Region Location
us-east-1 US East (N. Virginia)
us-east-2 US East (Ohio)
us-west-1 US West (N. California)
us-west-2 US West (Oregon)
ap-south-1 Asia Pacific (Mumbai)
ap-northeast-1 Asia Pacific (Tokyo)
ap-northeast-2 Asia Pacific (Seoul)
ap-northeast-3 Asia Pacific (Osaka)
ap-southeast-1 Asia Pacific (Singapore)
ap-southeast-2 Asia Pacific (Sydney)
ca-central-1 Canada (Central)
eu-central-1 Europe (Frankfurt)
eu-west-1 Europe (Ireland)
eu-west-2 Europe (London)
eu-west-3 Europe (Paris)
eu-north-1 Europe (Stockholm)
il-central-1 Israel (Tel Aviv)
sa-east-1 South America (São Paulo)

Configuration

Required

  • AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper - This environment variable must be set in order to enable tracing. If this environment variable will not be set, only logs will be collected.

  • DASH0_ENDPOINT - The integration endpoint for you organization in Dash0, i.e. https://ingress.eu-west-1.aws.dash0.com:4318.

  • Either DASH0_TOKEN or DASH0_TOKEN_SECRET_ARN must be set (see below).

Optional

  • DASH0_TOKEN - The API token for your Dash0 project. Either this or DASH0_TOKEN_SECRET_ARN must be set.

  • DASH0_TOKEN_SECRET_ARN - The ARN of an AWS Secrets Manager secret containing the Dash0 API token. Either this or DASH0_TOKEN must be set. If both are set, DASH0_TOKEN takes precedence.

  • DASH0_TOKEN_SECRET_KEY - The JSON key within the secret to extract the token from. Required when the secret stored in DASH0_TOKEN_SECRET_ARN is a JSON object. If not set, the entire secret value is used as the token.

  • DASH0_DISABLE_AUTO_INSTRUMENTATION - Auto-instrumentation can be turned off by this environment variable, which will result in creating synthetic traces by the extension for all invocations.

  • DASH0_SEND_ON_INVOCATION_END - The extension has two modes of sending to the backend, either on invocation end or on the next invocation. The default is true. Sending on invocation end will increase the billed duration of the lambda, but not the response time. Sending on next invocation will decrease the billed duration since the sending will take place in parallel of the regular execution, but might delay the sending up to 7 minutes in case of last invocation in the container.

  • DASH0_EXTENSION_LOG_LEVEL - Log level for the extension itself. Valid values: trace, debug, info, warn, error. Default: warn.

  • DASH0_DISTRO_DEBUG - When set to true, additional logs related to tracing and auto-instrumentation will be emitted. Default: false.

  • DASH0_REQUEST_TIMEOUT - Timeout in milliseconds for HTTP requests to the backend. Default: 2000.

  • DASH0_CREATE_PAYLOAD_LOG_RECORDS - When set to true (the default), the extension creates log records containing the request and response payloads for the lambda invocation and each client call. Set to false to disable. Default: true.

  • DASH0_DISABLE_TELEMETRY_LOG_COLLECTION - When set to true, disables collecting logs from the Lambda Telemetry API. Default: false.

  • DASH0_DISABLE_TELEMETRY_METRICS - When set to true, disables emission of the supplementary FaaS metrics (faas.invoke_duration, faas.init_duration, dash0.faas.billed_duration, faas.mem_usage). Default: false.

  • DASH0_DISABLE_TELEMETRY_TRACES - When set to true, disables both auto-instrumentation and any synthetic spans created by the extension (including error-path synthetic traces). Stronger than DASH0_DISABLE_AUTO_INSTRUMENTATION, which only disables auto-instrumentation while still emitting synthetic spans. Default: false.

  • DASH0_DATASET - When set, the extension adds a Dash0-Dataset header to all OTLP export requests, routing telemetry to the specified dataset in the Dash0 backend.

  • DASH0_XRAY_TRACES_ENABLED - When set to true, the extension preserves the original X-Ray trace context instead of creating supplementary spans. Use this when AWS X-Ray active tracing is enabled on the Lambda function. Default: false.

The four API_GATEWAY variables below also apply to Application Load Balancer-triggered invocations; the names are kept for backwards compatibility.

  • DASH0_ENABLE_API_GATEWAY_SPAN_NAME - When set to true, renames the handler span for an HTTP-triggered invocation (API Gateway REST API v1, HTTP API v2 proxy integration, or ALB) to <method> <route>, e.g. GET /pets/{id}. ALB reports no route, so those spans are named after the bare method (e.g. GET). Off by default, so existing span names don't change under you. Default: false.

  • DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE - Comma-separated, case-insensitive list of request header names to capture as http.request.header.<name> span attributes for API Gateway- and ALB-triggered invocations. Empty by default: headers may carry PII, auth tokens, or cookies, so none are captured unless named here.

    Example: DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE=content-type,x-request-id

  • DASH0_API_GATEWAY_RESPONSE_HEADERS_TO_CAPTURE - Same as above, but for the handler's response headers, captured as http.response.header.<name>.

  • DASH0_CAPTURE_API_GATEWAY_QUERY_STRING - When set to true, captures the request query string as url.query for API Gateway- and ALB-triggered invocations. Off by default: query strings can carry signed-URL tokens or other sensitive values. Default: false.

  • DASH0_DISABLE_PYTHON_DEPENDENCY_CHECK - Python only. On startup, the Python distribution checks whether its own dependencies conflict with the versions installed in the function, and skips loading the auto-instrumentation if they do. When set to true, that check is skipped and the distribution loads regardless. Use this if the check reports a false positive; note that a real conflict may cause the function to fail at runtime. Default: false.

  • DASH0_JAVA_SDK_SELF_METRICS_ENABLED - Java only. When set to true, exports the OpenTelemetry SDK's self-metrics (otlp.exporter.seen, otlp.exporter.exported, queueSize, processedSpans, processedLogs). Default: false.

  • DASH0_JAVA_RUNTIME_METRICS_ENABLED - Java only. When set to true, exports the JVM runtime metrics (jvm.*). Default: false.

Secret Masking

The extension masks sensitive data in the payloads it captures. Any JSON key matching one of these patterns (case-insensitive) has its value replaced with ****:

  • .*pass.*
  • .*key.*
  • .*secret.*
  • .*credential.*
  • .*passphrase.*
  • .*token.*
  • .*signature.*
  • .*authorization.*

Secret masking by example shows what each option below does to the captured span attributes, with the payload before and after masking.

Masking applies to Lambda event and response payloads and to environment variables captured in traces, in every runtime. Masking of the payloads captured by the auto-instrumentation depends on the runtime:

Node.js Python Java
Lambda event and response payloads ✅ ✅ ✅
Environment variables (DASH0_MASK_ENV_VARS) ✅ ✅ ✅
Payloads captured by the auto-instrumentation (DASH0_MASK_RULES) ✅ ⚠️ ¹ ❌
DASH0_MASK_RULES=all ✅ ❌ ❌
DASH0_MASK_REQUEST_BODY ✅ ⚠️ ¹ ❌
DASH0_MASK_RESPONSE_BODY ✅ ⚠️ ¹ ❌
DASH0_MASK_REQUEST_HEADERS ✅ ⚠️ ¹ ❌
DASH0_MASK_RESPONSE_HEADERS ✅ ⚠️ ¹ ❌
DASH0_MASK_QUERY_PARAMS ✅ ❌ ❌
Kafka and RabbitMQ messages sent (masked with DASH0_MASK_QUERY_PARAMS) ✅ ❌ ❌
Keys inside a JSON string body, e.g. an SQS MessageBody ⚠️ JSON HTTP bodies only ✅ strings starting with { ❌

¹ Python masks HTTP headers (except from requests and Django), AWS SDK payloads, FastAPI bodies, and MongoDB and PostgreSQL results. Bodies from requests, httpx and Django, and Redis, Kafka and RabbitMQ payloads, are not masked.

In Python, patterns must match from the start of the key, so write .*email.*, not email. Some payloads are never masked, such as gRPC messages and PostgreSQL query parameters in Node.js. Secret masking by example lists them for each runtime.

Masking matches keys. Where a body is a string that is not parsed, such as an SQS MessageBody in Node.js, mask it as a whole (see all below) or by its key (e.g. .*MessageBody.*).

Custom masking rules:

Each variable takes a JSON array of regex patterns. Write patterns as .*name.* so they match anywhere in the key in every runtime. A value that is not a JSON array is ignored with a warning in the function's logs, and the fallback applies (DASH0_MASK_RULES, or the defaults).

  • DASH0_MASK_RULES - Patterns for all payloads. When set, this replaces the default patterns, so list the defaults too if you want to keep them.

    Example: DASH0_MASK_RULES='[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*passphrase.*", ".*token.*", ".*signature.*", ".*authorization.*", ".*email.*"]'

    This list also applies to environment variables unless DASH0_MASK_ENV_VARS is set. Keep .*key.*, .*secret.* and .*token.* in it, or the function's AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN are sent unmasked.

  • DASH0_MASK_ENV_VARS - Patterns for environment variables captured in traces. Falls back to DASH0_MASK_RULES (or the defaults). Every environment variable is captured, including the AWS credentials, so keep .*key.*, .*secret.* and .*token.* in the list.

    Example: DASH0_MASK_ENV_VARS='[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*token.*", ".*DATABASE_URL.*"]'

Per-payload masking rules:

These override DASH0_MASK_RULES for one kind of payload captured by the auto-instrumentation (see the table above for runtime support). When not set, that payload uses DASH0_MASK_RULES (or the defaults). Instead of a JSON array, each also accepts the value all, which replaces the whole payload with ****.

Variable Applies to
DASH0_MASK_REQUEST_BODY HTTP request bodies, SQS SendMessage/SendMessageBatch bodies
DASH0_MASK_RESPONSE_BODY HTTP response bodies, SQS ReceiveMessage bodies
DASH0_MASK_REQUEST_HEADERS HTTP request headers
DASH0_MASK_RESPONSE_HEADERS HTTP response headers
DASH0_MASK_QUERY_PARAMS HTTP query parameters

Examples:

  • Never capture outgoing HTTP request bodies or sent SQS messages: DASH0_MASK_REQUEST_BODY=all. Received messages need DASH0_MASK_RESPONSE_BODY=all. The bodies of SQS messages that trigger your function are part of the Lambda event, which only DASH0_MASK_RULES covers.
  • Keep the defaults and also mask SQS message bodies: DASH0_MASK_RULES='[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*passphrase.*", ".*token.*", ".*signature.*", ".*authorization.*", ".*MessageBody.*"]'

Manual Instrumentation

If you prefer to set up OpenTelemetry instrumentation yourself instead of relying on the extension's auto-instrumentation, you can use the manual layer and point your OTLP exporters to the extension's local endpoint. The extension will receive the telemetry, enrich it, and forward it to Dash0.

  1. Add the manual layer to your Lambda function: arn:aws:lambda:<region>:115813213817:layer:dash0-extension-manual:<version>.
  2. Configure your OTLP trace exporter to send to http://127.0.0.1:9009/v1/traces.
  3. If exporting metrics, configure your OTLP metric exporter to send to http://127.0.0.1:9009/v1/metrics.
  4. Make sure to flush all telemetry before the Lambda invocation completes (e.g., in a response hook or before returning the response).

Enrichment Attributes

The extension enriches telemetry data with additional attributes beyond what the auto-instrumentation provides.

API Gateway and Application Load Balancer

When a Lambda is invoked over HTTP, the extension recognizes the trigger shape and adds HTTP semantic-convention attributes to the handler span:

  • REST API (v1), proxy integration - detected by a top-level httpMethod field alongside requestContext.
  • HTTP API (v2), proxy integration - detected by requestContext.http alongside a top-level rawPath field.
  • Application Load Balancer - detected by a requestContext.elb object. ALB events also carry a top-level httpMethod, so this check runs first to keep them from being reported as REST API (v1).

Lambda Function URLs are also covered, with no extra detection logic needed: AWS defines the Function URL invocation event as the same shape as an HTTP API (v2) proxy integration event (requestContext.http + top-level rawPath), so it's picked up by the v2 path above. The one difference is http.route, which is always $default for Function URLs, since they have no route concept of their own - requestContext.routeKey is always $default for this trigger type.

This runs independently of the in-function OpenTelemetry SDK: the extension already parses the raw invoke event and the raw return payload for every invocation (see Manual Instrumentation for how telemetry reaches the extension). As a result, these attributes appear for every supported Lambda runtime (Node.js, Python, Java, .NET, Go), and even when auto-instrumentation is disabled and the extension builds a synthetic trace itself.

Attribute REST API (v1) source HTTP API (v2) source ALB source
http.request.method httpMethod requestContext.http.method httpMethod
url.path path rawPath path
url.scheme always https same the x-forwarded-proto header
http.route resource, as-is (e.g. /pets/{id}) requestContext.routeKey with the leading method stripped not available
server.address / server.port requestContext.domainName / always 443 same the host header, split on a trailing port; port falls back to x-forwarded-port
client.address requestContext.identity.sourceIp requestContext.http.sourceIp first entry of the x-forwarded-for header
network.protocol.version parsed from requestContext.protocol (e.g. HTTP/1.1 -> 1.1) parsed from requestContext.http.protocol not available
http.response.status_code statusCode from the handler's proxy-integration return payload ({ statusCode, headers, body }) same same

ALB specifics

ALB reports no route template (there is no resource or routeKey field), so http.route is left unset rather than filled in with the high-cardinality request path. Likewise there is no protocol field, so network.protocol.version is omitted. Because ALB listeners serve plain HTTP as well as HTTPS, url.scheme follows x-forwarded-proto instead of being hardcoded to https as it is for API Gateway.

Target groups with the lambda.multi_value_headers.enabled attribute send multiValueHeaders and multiValueQueryStringParameters instead of headers and queryStringParameters, in both directions. Both shapes are handled. A header that appears multiple times is captured as a single comma-separated value, so http.request.header.<name> stays a string in either configuration.

Health-check invocations (user-agent: ELB-HealthChecker/2.0) carry no host and no forwarding headers, so they yield just http.request.method and url.path.

Request attributes and the response status code are captured unconditionally, since none of them carry PII. Three behaviors are opt-in via environment variables, each defaulting to off so nothing changes for existing users until you ask for it. The variables are named after API Gateway for backwards compatibility, but they apply to ALB-triggered invocations too:

  • Renaming the span from the default handler name to <method> <route> (e.g. GET /pets/{id}) - DASH0_ENABLE_API_GATEWAY_SPAN_NAME. For ALB, where no route is available, the span is named after the bare method (e.g. GET), per the HTTP semantic conventions.
  • Capturing specific request or response headers as http.request.header.<name> / http.response.header.<name> - DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE / DASH0_API_GATEWAY_RESPONSE_HEADERS_TO_CAPTURE, each an explicit allow-list rather than "capture all headers", since headers can carry auth tokens or cookies.
  • Capturing the request query string as url.query - DASH0_CAPTURE_API_GATEWAY_QUERY_STRING, since query strings can carry signed-URL tokens or other secrets.

See Configuration for the exact syntax of each variable.

Span Attributes

The following attributes are added to spans by the extension (if relevant):

Attribute Type Description
faas.invocation_id string The AWS request ID of the current invocation.
faas.trigger string The event source that triggered the Lambda (e.g., aws:sqs, aws:dynamodb, aws:event_bridge). Extracted from the event payload.
faas.init_duration double The cold start initialization duration in milliseconds. Only present on cold start invocations.
dash0.faas.record_count int The number of records in a batch event (SQS, DynamoDB Streams, Kinesis, SNS).
dash0.faas.trigger_arn string The ARN of the event source (e.g., SQS queue ARN, DynamoDB stream ARN, SNS topic ARN).
dash0.faas.event_bridge_source string The source field from an EventBridge event.
dash0.faas.event_bridge_detail_type string The detail-type field from an EventBridge event.
http.request.method string The HTTP method. API Gateway- and ALB-triggered invocations only.
url.path string The request path. API Gateway- and ALB-triggered invocations only.
url.scheme string Always https for API Gateway; from x-forwarded-proto for ALB.
http.route string The matched route template, as reported by API Gateway (e.g. /pets/{id}). API Gateway-triggered invocations only - ALB reports no route.
server.address string The API Gateway domain name, or the host header for ALB.
server.port int Always 443 for API Gateway; from the host header or x-forwarded-port for ALB.
client.address string The caller's source IP. API Gateway- and ALB-triggered invocations only.
network.protocol.version string The HTTP protocol version (e.g. 1.1). API Gateway-triggered invocations only - ALB reports no protocol.
http.response.status_code int The statusCode from the handler's return payload. API Gateway- and ALB-triggered invocations only.

See API Gateway and Application Load Balancer above for how these are derived per trigger type, and which related attributes are opt-in.

Resource Attributes (Spans)

These attributes are added to the resource of span data:

Attribute Type Description
service.name string The service name, from OTEL_SERVICE_NAME or defaults to unknown_service.
process.environment_variable.<KEY> string Lambda environment variables (with sensitive values masked). Added to the span resource.

Log Attributes

The following attributes are added to log records by the extension (if relevant):

Attribute Type Description
faas.invocation_id string The AWS request ID, used to correlate logs with the invocation span.
dash0.faas.payload_type string The type of payload log record. Values: lambda_event, lambda_return_value, http_request_body, http_response_body. Only present on payload log records.

Resource Attributes (Logs)

These attributes are added to the resource of log data:

Attribute Type Description
cloud.platform string Always set to aws_lambda.
cloud.resource_id string The full ARN of the Lambda function.
cloud.account.id string The AWS account ID.
service.name string The service name, from OTEL_SERVICE_NAME or defaults to unknown_service.

Metrics

The extension creates the following histogram metrics for each Lambda invocation:

Metric Unit Description
faas.invoke_duration ms Duration of the invocation.
faas.init_duration ms Duration of the cold start initialization. Only present on cold start invocations.
dash0.faas.billed_duration ms Billed duration of the invocation.
faas.mem_usage MB Memory used by the invocation.

Custom Metrics

Besides the metrics created by the extension, functions can record their own metrics through the OpenTelemetry API. How this works depends on the runtime; the following applies to all of them:

  • Metrics are exported with the OpenTelemetry default, cumulative temporality. Set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE (cumulative, delta or lowmemory) to change it.
  • DASH0_DISABLE_TELEMETRY_METRICS only disables the metrics created by the extension, not custom metrics.
Java

With auto-instrumentation, Java functions can record their own metrics through the OpenTelemetry API (for example GlobalOpenTelemetry.getMeter("my-app").counterBuilder("orders.processed").build().add(1)). The agent exports them to the extension with an OTLP exporter that is enabled by default, and flushes them when the handler returns. Add io.opentelemetry:opentelemetry-api to your function as a regular runtime dependency (implementation in Gradle, the default compile scope in Maven); the agent only bundles a private copy. Set OTEL_METRICS_EXPORTER=none to turn metric export off.

The agent's JVM runtime metrics (jvm.*) are disabled by default. To enable them, set DASH0_JAVA_RUNTIME_METRICS_ENABLED=true. The OpenTelemetry SDK's own self-metrics about its exporter and span processor (otlp.exporter.seen, otlp.exporter.exported, queueSize, processedSpans, processedLogs) are also not exported by default; set DASH0_JAVA_SDK_SELF_METRICS_ENABLED=true to export them.

Node.js

With auto-instrumentation, Node.js functions can record their own metrics through the OpenTelemetry API. The layer registers a global MeterProvider that exports to the extension, and flushes it when the handler returns, so there is no MeterProvider to set up and no forceFlush() to call:

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('my-app');
const ordersProcessed = meter.createCounter('orders.processed');
const orderValue = meter.createHistogram('order.value', { unit: 'EUR' });

export async function handler(event) {
    ordersProcessed.add(1, { 'order.type': event.type });
    orderValue.record(event.total);
    return { statusCode: 200 };
}

Add @opentelemetry/api to your function as a regular dependency and bundle it with your code; the layer only bundles a private copy. The global MeterProvider is shared across copies of the API as long as the major version matches and the minor version of your copy is not higher than the layer's (currently 1.9), so any @opentelemetry/api from 1.0.0 up to 1.9.x works.

Only metrics your code records are exported: the layer's library instrumentations (HTTP, AWS SDK, databases, ...) and the OpenTelemetry SDK do not emit metrics of their own. Functions that record no metrics send nothing to the metrics endpoint. Custom metrics are part of auto-instrumentation, so setting DASH0_DISABLE_AUTO_INSTRUMENTATION or DASH0_DISABLE_TELEMETRY_TRACES turns them off as well.

Resource Attributes (Metrics)

These attributes are added to the resource of metric data:

Attribute Type Description
cloud.platform string Always set to aws_lambda.
cloud.resource_id string The full ARN of the Lambda function.
cloud.account.id string The AWS account ID.
service.name string The service name, from OTEL_SERVICE_NAME or defaults to unknown_service.

Serverless Framework Plugin

For Serverless Framework users, the serverless-dash0 plugin can automatically add the Dash0 layer and configuration to your Lambda functions. See the plugin README for setup instructions and configuration options.

Dockerized Lambdas

For containerized Lambda functions, use the provided Docker images in a multi-stage build. The extension images are available for Node.js, Python, and Java runtimes.

Node.js

FROM public.ecr.aws/lambda/nodejs:20

# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-node:latest /opt /opt

# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here

# Copy your function code
COPY index.js ${LAMBDA_TASK_ROOT}

CMD ["index.handler"]

Python

FROM public.ecr.aws/lambda/python:3.12

# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-python:latest /opt /opt

# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here

# Copy your function code
COPY app.py ${LAMBDA_TASK_ROOT}

CMD ["app.handler"]

Java

FROM public.ecr.aws/lambda/java:21

# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-java:latest /opt /opt

# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here

# Copy your function code
COPY target/my-function.jar ${LAMBDA_TASK_ROOT}

CMD ["com.example.Handler::handleRequest"]

About

No description, website, or topics provided.

Resources

Contributing

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages