An extension for capturing observability data from AWS Lambda invocations and shipping to Dash0.
- Layer ARNs
- Supported Regions
- Configuration
- Manual Instrumentation
- Enrichment Attributes
- Serverless Framework Plugin
- Dockerized Lambdas
This extension has five main functionalities:
- Enable auto-instrumentation for supported runtimes, which currently include Python, Node, Java.
- Receive traces from auto/manual instrumentations, enrich with data acquired in the extension, and send to Dash0.
- Detect runtime errors such as timeout or out of memory and create synthetic traces for them
- Collect all logs and send to Dash0, correlated with the trace id of the invocation.
- Create metrics for invocation duration, cold start duration, billed duration, and memory used.
See the release page for the latest ARNs of the extension layers for each runtime.
The extension layers are published to the following AWS regions:
| Region | Location |
|---|---|
us-east-1 |
US East (N. Virginia) |
us-east-2 |
US East (Ohio) |
us-west-1 |
US West (N. California) |
us-west-2 |
US West (Oregon) |
ap-south-1 |
Asia Pacific (Mumbai) |
ap-northeast-1 |
Asia Pacific (Tokyo) |
ap-northeast-2 |
Asia Pacific (Seoul) |
ap-northeast-3 |
Asia Pacific (Osaka) |
ap-southeast-1 |
Asia Pacific (Singapore) |
ap-southeast-2 |
Asia Pacific (Sydney) |
ca-central-1 |
Canada (Central) |
eu-central-1 |
Europe (Frankfurt) |
eu-west-1 |
Europe (Ireland) |
eu-west-2 |
Europe (London) |
eu-west-3 |
Europe (Paris) |
eu-north-1 |
Europe (Stockholm) |
il-central-1 |
Israel (Tel Aviv) |
sa-east-1 |
South America (São Paulo) |
-
AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper- This environment variable must be set in order to enable tracing. If this environment variable will not be set, only logs will be collected. -
DASH0_ENDPOINT- The integration endpoint for you organization in Dash0, i.e.https://ingress.eu-west-1.aws.dash0.com:4318. -
Either
DASH0_TOKENorDASH0_TOKEN_SECRET_ARNmust be set (see below).
-
DASH0_TOKEN- The API token for your Dash0 project. Either this orDASH0_TOKEN_SECRET_ARNmust be set. -
DASH0_TOKEN_SECRET_ARN- The ARN of an AWS Secrets Manager secret containing the Dash0 API token. Either this orDASH0_TOKENmust be set. If both are set,DASH0_TOKENtakes precedence. -
DASH0_TOKEN_SECRET_KEY- The JSON key within the secret to extract the token from. Required when the secret stored inDASH0_TOKEN_SECRET_ARNis a JSON object. If not set, the entire secret value is used as the token. -
DASH0_DISABLE_AUTO_INSTRUMENTATION- Auto-instrumentation can be turned off by this environment variable, which will result in creating synthetic traces by the extension for all invocations. -
DASH0_SEND_ON_INVOCATION_END- The extension has two modes of sending to the backend, either on invocation end or on the next invocation. The default istrue. Sending on invocation end will increase the billed duration of the lambda, but not the response time. Sending on next invocation will decrease the billed duration since the sending will take place in parallel of the regular execution, but might delay the sending up to 7 minutes in case of last invocation in the container. -
DASH0_EXTENSION_LOG_LEVEL- Log level for the extension itself. Valid values:trace,debug,info,warn,error. Default:warn. -
DASH0_DISTRO_DEBUG- When set to true, additional logs related to tracing and auto-instrumentation will be emitted. Default:false. -
DASH0_REQUEST_TIMEOUT- Timeout in milliseconds for HTTP requests to the backend. Default:2000. -
DASH0_CREATE_PAYLOAD_LOG_RECORDS- When set totrue(the default), the extension creates log records containing the request and response payloads for the lambda invocation and each client call. Set tofalseto disable. Default:true. -
DASH0_DISABLE_TELEMETRY_LOG_COLLECTION- When set totrue, disables collecting logs from the Lambda Telemetry API. Default:false. -
DASH0_DISABLE_TELEMETRY_METRICS- When set totrue, disables emission of the supplementary FaaS metrics (faas.invoke_duration,faas.init_duration,dash0.faas.billed_duration,faas.mem_usage). Default:false. -
DASH0_DISABLE_TELEMETRY_TRACES- When set totrue, disables both auto-instrumentation and any synthetic spans created by the extension (including error-path synthetic traces). Stronger thanDASH0_DISABLE_AUTO_INSTRUMENTATION, which only disables auto-instrumentation while still emitting synthetic spans. Default:false. -
DASH0_DATASET- When set, the extension adds aDash0-Datasetheader to all OTLP export requests, routing telemetry to the specified dataset in the Dash0 backend. -
DASH0_XRAY_TRACES_ENABLED- When set totrue, the extension preserves the original X-Ray trace context instead of creating supplementary spans. Use this when AWS X-Ray active tracing is enabled on the Lambda function. Default:false.
The four API_GATEWAY variables below also apply to Application Load Balancer-triggered invocations; the names are kept for backwards compatibility.
-
DASH0_ENABLE_API_GATEWAY_SPAN_NAME- When set totrue, renames the handler span for an HTTP-triggered invocation (API Gateway REST API v1, HTTP API v2 proxy integration, or ALB) to<method> <route>, e.g.GET /pets/{id}. ALB reports no route, so those spans are named after the bare method (e.g.GET). Off by default, so existing span names don't change under you. Default:false. -
DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE- Comma-separated, case-insensitive list of request header names to capture ashttp.request.header.<name>span attributes for API Gateway- and ALB-triggered invocations. Empty by default: headers may carry PII, auth tokens, or cookies, so none are captured unless named here.Example:
DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE=content-type,x-request-id -
DASH0_API_GATEWAY_RESPONSE_HEADERS_TO_CAPTURE- Same as above, but for the handler's response headers, captured ashttp.response.header.<name>. -
DASH0_CAPTURE_API_GATEWAY_QUERY_STRING- When set totrue, captures the request query string asurl.queryfor API Gateway- and ALB-triggered invocations. Off by default: query strings can carry signed-URL tokens or other sensitive values. Default:false. -
DASH0_DISABLE_PYTHON_DEPENDENCY_CHECK- Python only. On startup, the Python distribution checks whether its own dependencies conflict with the versions installed in the function, and skips loading the auto-instrumentation if they do. When set totrue, that check is skipped and the distribution loads regardless. Use this if the check reports a false positive; note that a real conflict may cause the function to fail at runtime. Default:false. -
DASH0_JAVA_SDK_SELF_METRICS_ENABLED- Java only. When set totrue, exports the OpenTelemetry SDK's self-metrics (otlp.exporter.seen,otlp.exporter.exported,queueSize,processedSpans,processedLogs). Default:false. -
DASH0_JAVA_RUNTIME_METRICS_ENABLED- Java only. When set totrue, exports the JVM runtime metrics (jvm.*). Default:false.
The extension masks sensitive data in the payloads it captures. Any JSON key matching one of these patterns (case-insensitive) has its value replaced with ****:
.*pass.*.*key.*.*secret.*.*credential.*.*passphrase.*.*token.*.*signature.*.*authorization.*
Secret masking by example shows what each option below does to the captured span attributes, with the payload before and after masking.
Masking applies to Lambda event and response payloads and to environment variables captured in traces, in every runtime. Masking of the payloads captured by the auto-instrumentation depends on the runtime:
| Node.js | Python | Java | |
|---|---|---|---|
| Lambda event and response payloads | ✅ | ✅ | ✅ |
Environment variables (DASH0_MASK_ENV_VARS) |
✅ | ✅ | ✅ |
Payloads captured by the auto-instrumentation (DASH0_MASK_RULES) |
✅ | ❌ | |
DASH0_MASK_RULES=all |
✅ | ❌ | ❌ |
DASH0_MASK_REQUEST_BODY |
✅ | ❌ | |
DASH0_MASK_RESPONSE_BODY |
✅ | ❌ | |
DASH0_MASK_REQUEST_HEADERS |
✅ | ❌ | |
DASH0_MASK_RESPONSE_HEADERS |
✅ | ❌ | |
DASH0_MASK_QUERY_PARAMS |
✅ | ❌ | ❌ |
Kafka and RabbitMQ messages sent (masked with DASH0_MASK_QUERY_PARAMS) |
✅ | ❌ | ❌ |
Keys inside a JSON string body, e.g. an SQS MessageBody |
✅ strings starting with { |
❌ |
¹ Python masks HTTP headers (except from requests and Django), AWS SDK payloads, FastAPI bodies, and MongoDB and PostgreSQL results. Bodies from requests, httpx and Django, and Redis, Kafka and RabbitMQ payloads, are not masked.
In Python, patterns must match from the start of the key, so write .*email.*, not email. Some payloads are never masked, such as gRPC messages and PostgreSQL query parameters in Node.js. Secret masking by example lists them for each runtime.
Masking matches keys. Where a body is a string that is not parsed, such as an SQS MessageBody in Node.js, mask it as a whole (see all below) or by its key (e.g. .*MessageBody.*).
Custom masking rules:
Each variable takes a JSON array of regex patterns. Write patterns as .*name.* so they match anywhere in the key in every runtime. A value that is not a JSON array is ignored with a warning in the function's logs, and the fallback applies (DASH0_MASK_RULES, or the defaults).
-
DASH0_MASK_RULES- Patterns for all payloads. When set, this replaces the default patterns, so list the defaults too if you want to keep them.Example:
DASH0_MASK_RULES='[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*passphrase.*", ".*token.*", ".*signature.*", ".*authorization.*", ".*email.*"]'This list also applies to environment variables unless
DASH0_MASK_ENV_VARSis set. Keep.*key.*,.*secret.*and.*token.*in it, or the function'sAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKENare sent unmasked. -
DASH0_MASK_ENV_VARS- Patterns for environment variables captured in traces. Falls back toDASH0_MASK_RULES(or the defaults). Every environment variable is captured, including the AWS credentials, so keep.*key.*,.*secret.*and.*token.*in the list.Example:
DASH0_MASK_ENV_VARS='[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*token.*", ".*DATABASE_URL.*"]'
Per-payload masking rules:
These override DASH0_MASK_RULES for one kind of payload captured by the auto-instrumentation (see the table above for runtime support). When not set, that payload uses DASH0_MASK_RULES (or the defaults). Instead of a JSON array, each also accepts the value all, which replaces the whole payload with ****.
| Variable | Applies to |
|---|---|
DASH0_MASK_REQUEST_BODY |
HTTP request bodies, SQS SendMessage/SendMessageBatch bodies |
DASH0_MASK_RESPONSE_BODY |
HTTP response bodies, SQS ReceiveMessage bodies |
DASH0_MASK_REQUEST_HEADERS |
HTTP request headers |
DASH0_MASK_RESPONSE_HEADERS |
HTTP response headers |
DASH0_MASK_QUERY_PARAMS |
HTTP query parameters |
Examples:
- Never capture outgoing HTTP request bodies or sent SQS messages:
DASH0_MASK_REQUEST_BODY=all. Received messages needDASH0_MASK_RESPONSE_BODY=all. The bodies of SQS messages that trigger your function are part of the Lambda event, which onlyDASH0_MASK_RULEScovers. - Keep the defaults and also mask SQS message bodies:
DASH0_MASK_RULES='[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*passphrase.*", ".*token.*", ".*signature.*", ".*authorization.*", ".*MessageBody.*"]'
If you prefer to set up OpenTelemetry instrumentation yourself instead of relying on the extension's auto-instrumentation, you can use the manual layer and point your OTLP exporters to the extension's local endpoint. The extension will receive the telemetry, enrich it, and forward it to Dash0.
- Add the manual layer to your Lambda function:
arn:aws:lambda:<region>:115813213817:layer:dash0-extension-manual:<version>. - Configure your OTLP trace exporter to send to
http://127.0.0.1:9009/v1/traces. - If exporting metrics, configure your OTLP metric exporter to send to
http://127.0.0.1:9009/v1/metrics. - Make sure to flush all telemetry before the Lambda invocation completes (e.g., in a response hook or before returning the response).
The extension enriches telemetry data with additional attributes beyond what the auto-instrumentation provides.
When a Lambda is invoked over HTTP, the extension recognizes the trigger shape and adds HTTP semantic-convention attributes to the handler span:
- REST API (v1), proxy integration - detected by a top-level
httpMethodfield alongsiderequestContext. - HTTP API (v2), proxy integration - detected by
requestContext.httpalongside a top-levelrawPathfield. - Application Load Balancer - detected by a
requestContext.elbobject. ALB events also carry a top-levelhttpMethod, so this check runs first to keep them from being reported as REST API (v1).
Lambda Function URLs are also covered, with no extra detection logic needed: AWS defines the Function URL invocation event as the same shape as an HTTP API (v2) proxy integration event (requestContext.http + top-level rawPath), so it's picked up by the v2 path above. The one difference is http.route, which is always $default for Function URLs, since they have no route concept of their own - requestContext.routeKey is always $default for this trigger type.
This runs independently of the in-function OpenTelemetry SDK: the extension already parses the raw invoke event and the raw return payload for every invocation (see Manual Instrumentation for how telemetry reaches the extension). As a result, these attributes appear for every supported Lambda runtime (Node.js, Python, Java, .NET, Go), and even when auto-instrumentation is disabled and the extension builds a synthetic trace itself.
| Attribute | REST API (v1) source | HTTP API (v2) source | ALB source |
|---|---|---|---|
http.request.method |
httpMethod |
requestContext.http.method |
httpMethod |
url.path |
path |
rawPath |
path |
url.scheme |
always https |
same | the x-forwarded-proto header |
http.route |
resource, as-is (e.g. /pets/{id}) |
requestContext.routeKey with the leading method stripped |
not available |
server.address / server.port |
requestContext.domainName / always 443 |
same | the host header, split on a trailing port; port falls back to x-forwarded-port |
client.address |
requestContext.identity.sourceIp |
requestContext.http.sourceIp |
first entry of the x-forwarded-for header |
network.protocol.version |
parsed from requestContext.protocol (e.g. HTTP/1.1 -> 1.1) |
parsed from requestContext.http.protocol |
not available |
http.response.status_code |
statusCode from the handler's proxy-integration return payload ({ statusCode, headers, body }) |
same | same |
ALB reports no route template (there is no resource or routeKey field), so http.route is left unset rather than filled in with the high-cardinality request path. Likewise there is no protocol field, so network.protocol.version is omitted. Because ALB listeners serve plain HTTP as well as HTTPS, url.scheme follows x-forwarded-proto instead of being hardcoded to https as it is for API Gateway.
Target groups with the lambda.multi_value_headers.enabled attribute send multiValueHeaders and multiValueQueryStringParameters instead of headers and queryStringParameters, in both directions. Both shapes are handled. A header that appears multiple times is captured as a single comma-separated value, so http.request.header.<name> stays a string in either configuration.
Health-check invocations (user-agent: ELB-HealthChecker/2.0) carry no host and no forwarding headers, so they yield just http.request.method and url.path.
Request attributes and the response status code are captured unconditionally, since none of them carry PII. Three behaviors are opt-in via environment variables, each defaulting to off so nothing changes for existing users until you ask for it. The variables are named after API Gateway for backwards compatibility, but they apply to ALB-triggered invocations too:
- Renaming the span from the default handler name to
<method> <route>(e.g.GET /pets/{id}) -DASH0_ENABLE_API_GATEWAY_SPAN_NAME. For ALB, where no route is available, the span is named after the bare method (e.g.GET), per the HTTP semantic conventions. - Capturing specific request or response headers as
http.request.header.<name>/http.response.header.<name>-DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE/DASH0_API_GATEWAY_RESPONSE_HEADERS_TO_CAPTURE, each an explicit allow-list rather than "capture all headers", since headers can carry auth tokens or cookies. - Capturing the request query string as
url.query-DASH0_CAPTURE_API_GATEWAY_QUERY_STRING, since query strings can carry signed-URL tokens or other secrets.
See Configuration for the exact syntax of each variable.
The following attributes are added to spans by the extension (if relevant):
| Attribute | Type | Description |
|---|---|---|
faas.invocation_id |
string | The AWS request ID of the current invocation. |
faas.trigger |
string | The event source that triggered the Lambda (e.g., aws:sqs, aws:dynamodb, aws:event_bridge). Extracted from the event payload. |
faas.init_duration |
double | The cold start initialization duration in milliseconds. Only present on cold start invocations. |
dash0.faas.record_count |
int | The number of records in a batch event (SQS, DynamoDB Streams, Kinesis, SNS). |
dash0.faas.trigger_arn |
string | The ARN of the event source (e.g., SQS queue ARN, DynamoDB stream ARN, SNS topic ARN). |
dash0.faas.event_bridge_source |
string | The source field from an EventBridge event. |
dash0.faas.event_bridge_detail_type |
string | The detail-type field from an EventBridge event. |
http.request.method |
string | The HTTP method. API Gateway- and ALB-triggered invocations only. |
url.path |
string | The request path. API Gateway- and ALB-triggered invocations only. |
url.scheme |
string | Always https for API Gateway; from x-forwarded-proto for ALB. |
http.route |
string | The matched route template, as reported by API Gateway (e.g. /pets/{id}). API Gateway-triggered invocations only - ALB reports no route. |
server.address |
string | The API Gateway domain name, or the host header for ALB. |
server.port |
int | Always 443 for API Gateway; from the host header or x-forwarded-port for ALB. |
client.address |
string | The caller's source IP. API Gateway- and ALB-triggered invocations only. |
network.protocol.version |
string | The HTTP protocol version (e.g. 1.1). API Gateway-triggered invocations only - ALB reports no protocol. |
http.response.status_code |
int | The statusCode from the handler's return payload. API Gateway- and ALB-triggered invocations only. |
See API Gateway and Application Load Balancer above for how these are derived per trigger type, and which related attributes are opt-in.
These attributes are added to the resource of span data:
| Attribute | Type | Description |
|---|---|---|
service.name |
string | The service name, from OTEL_SERVICE_NAME or defaults to unknown_service. |
process.environment_variable.<KEY> |
string | Lambda environment variables (with sensitive values masked). Added to the span resource. |
The following attributes are added to log records by the extension (if relevant):
| Attribute | Type | Description |
|---|---|---|
faas.invocation_id |
string | The AWS request ID, used to correlate logs with the invocation span. |
dash0.faas.payload_type |
string | The type of payload log record. Values: lambda_event, lambda_return_value, http_request_body, http_response_body. Only present on payload log records. |
These attributes are added to the resource of log data:
| Attribute | Type | Description |
|---|---|---|
cloud.platform |
string | Always set to aws_lambda. |
cloud.resource_id |
string | The full ARN of the Lambda function. |
cloud.account.id |
string | The AWS account ID. |
service.name |
string | The service name, from OTEL_SERVICE_NAME or defaults to unknown_service. |
The extension creates the following histogram metrics for each Lambda invocation:
| Metric | Unit | Description |
|---|---|---|
faas.invoke_duration |
ms | Duration of the invocation. |
faas.init_duration |
ms | Duration of the cold start initialization. Only present on cold start invocations. |
dash0.faas.billed_duration |
ms | Billed duration of the invocation. |
faas.mem_usage |
MB | Memory used by the invocation. |
Besides the metrics created by the extension, functions can record their own metrics through the OpenTelemetry API. How this works depends on the runtime; the following applies to all of them:
- Metrics are exported with the OpenTelemetry default, cumulative temporality. Set
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE(cumulative,deltaorlowmemory) to change it. DASH0_DISABLE_TELEMETRY_METRICSonly disables the metrics created by the extension, not custom metrics.
With auto-instrumentation, Java functions can record their own metrics through the OpenTelemetry API (for example GlobalOpenTelemetry.getMeter("my-app").counterBuilder("orders.processed").build().add(1)). The agent exports them to the extension with an OTLP exporter that is enabled by default, and flushes them when the handler returns. Add io.opentelemetry:opentelemetry-api to your function as a regular runtime dependency (implementation in Gradle, the default compile scope in Maven); the agent only bundles a private copy. Set OTEL_METRICS_EXPORTER=none to turn metric export off.
The agent's JVM runtime metrics (jvm.*) are disabled by default. To enable them, set DASH0_JAVA_RUNTIME_METRICS_ENABLED=true. The OpenTelemetry SDK's own self-metrics about its exporter and span processor (otlp.exporter.seen, otlp.exporter.exported, queueSize, processedSpans, processedLogs) are also not exported by default; set DASH0_JAVA_SDK_SELF_METRICS_ENABLED=true to export them.
With auto-instrumentation, Node.js functions can record their own metrics through the OpenTelemetry API. The layer registers a global MeterProvider that exports to the extension, and flushes it when the handler returns, so there is no MeterProvider to set up and no forceFlush() to call:
import { metrics } from '@opentelemetry/api';
const meter = metrics.getMeter('my-app');
const ordersProcessed = meter.createCounter('orders.processed');
const orderValue = meter.createHistogram('order.value', { unit: 'EUR' });
export async function handler(event) {
ordersProcessed.add(1, { 'order.type': event.type });
orderValue.record(event.total);
return { statusCode: 200 };
}Add @opentelemetry/api to your function as a regular dependency and bundle it with your code; the layer only bundles a private copy. The global MeterProvider is shared across copies of the API as long as the major version matches and the minor version of your copy is not higher than the layer's (currently 1.9), so any @opentelemetry/api from 1.0.0 up to 1.9.x works.
Only metrics your code records are exported: the layer's library instrumentations (HTTP, AWS SDK, databases, ...) and the OpenTelemetry SDK do not emit metrics of their own. Functions that record no metrics send nothing to the metrics endpoint. Custom metrics are part of auto-instrumentation, so setting DASH0_DISABLE_AUTO_INSTRUMENTATION or DASH0_DISABLE_TELEMETRY_TRACES turns them off as well.
These attributes are added to the resource of metric data:
| Attribute | Type | Description |
|---|---|---|
cloud.platform |
string | Always set to aws_lambda. |
cloud.resource_id |
string | The full ARN of the Lambda function. |
cloud.account.id |
string | The AWS account ID. |
service.name |
string | The service name, from OTEL_SERVICE_NAME or defaults to unknown_service. |
For Serverless Framework users, the serverless-dash0 plugin can automatically add the Dash0 layer and configuration to your Lambda functions. See the plugin README for setup instructions and configuration options.
For containerized Lambda functions, use the provided Docker images in a multi-stage build. The extension images are available for Node.js, Python, and Java runtimes.
FROM public.ecr.aws/lambda/nodejs:20
# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-node:latest /opt /opt
# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here
# Copy your function code
COPY index.js ${LAMBDA_TASK_ROOT}
CMD ["index.handler"]FROM public.ecr.aws/lambda/python:3.12
# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-python:latest /opt /opt
# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here
# Copy your function code
COPY app.py ${LAMBDA_TASK_ROOT}
CMD ["app.handler"]FROM public.ecr.aws/lambda/java:21
# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-java:latest /opt /opt
# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here
# Copy your function code
COPY target/my-function.jar ${LAMBDA_TASK_ROOT}
CMD ["com.example.Handler::handleRequest"]