Skip to content

Commit b057ba2

Browse files
authored
Format parameters as tabs (#964)
* Format parameters as tabs * Apply suggestions from code review
1 parent 317e937 commit b057ba2

File tree

1 file changed

+160
-4
lines changed

1 file changed

+160
-4
lines changed

aws/logs_monitoring/README.md

Lines changed: 160 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -341,11 +341,10 @@ Otherwise, if you are using Web Proxy:
341341

342342
The Datadog Forwarder is signed by Datadog. To verify the integrity of the Forwarder, use the manual installation method. [Create a Code Signing Configuration][19] that includes Datadog’s Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) and associate it with the Forwarder Lambda function before uploading the Forwarder ZIP file.
343343

344-
## CloudFormation parameters
344+
## Parameters
345345

346-
<div class="alert alert-warning">
347-
The following parameters are used in CloudFormation and Terraform. If you are installing the Forwarder manually, convert these parameter names from Pascal case to screaming snake case. For example, <code>DdApiKey</code> becomes <code>DD_API_KEY</code>, and <code>ExcludeAtMatch</code> becomes <code>EXCLUDE_AT_MATCH</code>.
348-
</div>
346+
{{< tabs >}}
347+
{{% tab "CloudFormation and Terraform" %}}
349348

350349
### Required
351350

@@ -487,6 +486,163 @@ To test different patterns against your logs, turn on [debug logs](#troubleshoot
487486
`LayerARN`
488487
: ARN for the layer containing the forwarder code. If empty, the script will use the version of the layer the forwarder was published with. Defaults to empty.
489488

489+
490+
[20]: https://app.datadoghq.com/organization-settings/api-keys
491+
[13]: https://docs.datadoghq.com/getting_started/site/
492+
[21]: https://docs.datadoghq.com/logs/processing/pipelines/
493+
[2]: https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function/
494+
{{% /tab %}}
495+
{{% tab "Manual" %}}
496+
497+
If you are installing the Forwarder manually, convert the parameter names from Pascal case to screaming snake case.
498+
499+
### Required
500+
501+
`DD_API_KEY`
502+
: Your [Datadog API key][20], which can be found under **Organization Settings** > **API Keys**. The API Key is stored in AWS Secrets Manager. If you already have a Datadog API Key stored in Secrets Manager, use `DD_API_KEY_SECRET_ARN` instead.
503+
504+
`DD_API_KEY_SECRET_ARN`
505+
: The ARN of the secret storing the Datadog API key, if you already have it stored in Secrets Manager. You must store the secret as a plaintext, rather than a key-value pair.
506+
507+
`DD_SITE`
508+
: The [Datadog site][13] that your metrics and logs will be sent to. Your Datadog site is {{< region-param key="dd_site" code="true" >}}.
509+
510+
### Lambda function (optional)
511+
512+
`FUNCTION_NAME`
513+
: The name of the Datadog Forwarder Lambda function. Do not change this when updating an existing CloudFormation stack, otherwise the current forwarder function will be replaced and all the triggers will be lost.
514+
515+
`MEMORY_SIZE`
516+
: Memory size for the Datadog Forwarder Lambda function.
517+
518+
`TIMEOUT`
519+
: Timeout for the Datadog Forwarder Lambda function.
520+
521+
`RESERVED_CONCURRENCY`
522+
: Reserved concurrency for the Datadog Forwarder Lambda function. If empty, use unreserved account concurrency.
523+
Datadog recommends using at least 10 reserved concurrency, but this defaults to 0 as you may need to increase your limits. If using unreserved account concurrency, you may limit other Lambda functions in your environment.
524+
525+
`LOG_RETENTION_IN_DAYS`
526+
: CloudWatch log retention for logs generated by the Datadog Forwarder Lambda function.
527+
528+
### Log forwarding (optional)
529+
530+
`DD_TAGS`
531+
: Add custom tags to forwarded logs, comma-delimited string, no trailing comma, such as `env:prod,stack:classic`.
532+
533+
`DD_MULTILINE_LOG_REGEX_PATTERN`
534+
: Use the supplied regular expression to detect for a new log line for multiline logs from S3, such as `\d{2}\/\d{2}\/\d{4}` for multiline logs beginning with pattern "11/10/2014".
535+
536+
`DD_USE_TCP`
537+
: By default, the forwarder sends logs using HTTPS through the port 443. To send logs over an
538+
SSL encrypted TCP connection, set this parameter to true.
539+
540+
`DD_NO_SSL`
541+
: Disable SSL when forwarding logs, set to true when forwarding logs through a proxy.
542+
543+
`DD_URL`
544+
: The endpoint URL to forward the logs to, useful for forwarding logs through a proxy.
545+
546+
`DD_PORT`
547+
: The endpoint port to forward the logs to, useful for forwarding logs through a proxy.
548+
549+
`DD_SKIP_SSL_VALIDATION`
550+
: Send logs over HTTPS, while not validating the certificate provided by the endpoint. This will still encrypt the traffic between the forwarder and the log intake endpoint, but will not verify if the destination SSL certificate is valid.
551+
552+
`DD_USE_COMPRESSION`
553+
: Set to false to disable log compression. Only valid when sending logs over HTTP.
554+
555+
`DD_COMPRESSION_LEVEL`
556+
: Set the compression level from 0 (no compression) to 9 (best compression). The default compression level is 6. You may see some benefit with regard to decreased outbound network traffic if you increase the compression level, at the expense of increased Forwarder execution duration.
557+
558+
`DD_FORWARD_LOG`
559+
: Set to false to disable log forwarding, while continuing to forward other observability data, such as metrics and traces from Lambda functions.
560+
561+
### Log scrubbing (optional)
562+
563+
`REDACT_IP`
564+
: Replace text matching `\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}` with `xxx.xxx.xxx.xxx`.
565+
566+
`REDACT_EMAIL`
567+
: Replace text matching `[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+` with `xxxxx@xxxxx.com`.
568+
569+
`DD_SCRUBBING_RULE`
570+
: Replace text matching the supplied regular expression with `xxxxx` (default) or `DD_SCRUBBING_RULE_REPLACEMENT` (if supplied). Log scrubbing rule is applied to the full JSON-formatted log, including any metadata that is automatically added by the Lambda function. Each instance of a pattern match is replaced until no more matches are found in each log. Using inefficient regular expression, such as `.*`, may slow down the Lambda function.
571+
572+
`DD_SCRUBBING_RULE_REPLACEMENT`
573+
: Replace text matching DD_SCRUBBING_RULE with the supplied text.
574+
575+
### Log filtering (optional)
576+
577+
`EXCLUDE_AT_MATCH`
578+
: Do not send logs matching the supplied regular expression. If a log matches both the `EXCLUDE_AT_MATCH` and `INCLUDE_AT_MATCH`, it is excluded.
579+
580+
`INCLUDE_AT_MATCH`
581+
: Only send logs matching the supplied regular expression, and not excluded by `EXCLUDE_AT_MATCH`.
582+
583+
Filtering rules are applied to the full JSON-formatted log, including any metadata that is automatically added by the Forwarder. However, transformations applied by [log pipelines][21], which occur after logs are sent to Datadog, cannot be used to filter logs in the Forwarder. Using an inefficient regular expression, such as `.*`, may slow down the Forwarder.
584+
585+
Some examples of regular expressions that can be used for log filtering:
586+
587+
- Include (or exclude) Lambda platform logs: `"(START|END) RequestId:\s`. The preceding `"` is needed to match the start of the log message, which is in a JSON blob (`{"message": "START RequestId...."}`). Datadog recommends keeping the `REPORT` logs, as they are used to populate the invocations list in the serverless function views.
588+
- Include CloudTrail error messages only: `errorMessage`.
589+
- Include only logs containing an HTTP 4XX or 5XX error code: `\b[4|5][0-9][0-9]\b`.
590+
- Include only CloudWatch logs where the `message` field contains a specific JSON key/value pair: `\"awsRegion\":\"us-east-1\"`.
591+
- The message field of a CloudWatch log event is encoded as a string. For example,`{"awsRegion": "us-east-1"}` is encoded as `{\"awsRegion\":\"us-east-1\"}`. Therefore, the pattern you provide must include `\` escape characters, like this: `\"awsRegion\":\"us-east-1\"`.
592+
593+
To test different patterns against your logs, turn on [debug logs](#troubleshooting).
594+
595+
### Advanced (optional)
596+
597+
`DD_FETCH_LAMBDA_TAGS`
598+
: Let the Forwarder fetch Lambda tags using GetResources API calls and apply them to logs, metrics, and traces. If set to true, permission `tag:GetResources` will be automatically added to the Lambda execution IAM role.
599+
600+
`DD_FETCH_LOG_GROUP_TAGS`
601+
: Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics, and traces. If set to true, permission `logs:ListTagsForResource` will be automatically added to the Lambda execution IAM role.
602+
603+
`DD_FETCH_STEP_FUNCTIONS_TAGS`
604+
: Let the Forwarder fetch Step Functions tags using GetResources API calls and apply them to logs and traces (if Step Functions tracing is enabled). If set to true, permission `tag:GetResources` will be automatically added to the Lambda execution IAM role.
605+
606+
`DD_STEP_FUNCTION_TRACE_ENABLED`
607+
: Set to true to enable tracing for all Step Functions.
608+
609+
`SOURCE_ZIP_URL`
610+
: Do not change unless you know what you are doing. Override the default location of the function source code.
611+
612+
`PERMISSIONS_BOUNDARY_ARN`
613+
: ARN for the Permissions Boundary Policy.
614+
615+
`DD_USE_PRIVATE_LINK` (DEPRECATED)
616+
: Set to true to enable sending logs and metrics through AWS PrivateLink. See [Connect to Datadog over AWS PrivateLink][2].
617+
618+
`DD_HTTP_PROXY_URL`
619+
: Sets the standard web proxy environment variables HTTP_PROXY and HTTPS_PROXY. These are the URL endpoints your proxy server exposes. Do not use this in combination with AWS Private Link. Make sure to also set `DD_SKIP_SSL_VALIDATION` to true.
620+
621+
`DD_NO_PROXY`
622+
: Sets the standard web proxy environment variable `NO_PROXY`. It is a comma-separated list of domain names that should be excluded from the web proxy.
623+
624+
`VPC_SECURITY_GROUP_IDS`
625+
: Comma separated list of VPC Security Group IDs. Used when AWS PrivateLink is enabled.
626+
627+
`VPC_SUBNET_IDS`
628+
: Comma separated list of VPC Subnet IDs. Used when AWS PrivateLink is enabled.
629+
630+
`ADDITIONAL_TARGET_LAMBDA_ARNS`
631+
: Comma separated list of Lambda ARNs that will get called asynchronously with the same `event` the Datadog Forwarder receives.
632+
633+
`INSTALL_AS_LAYER`
634+
: Whether to use the layer-based installation flow. Set to false to use the legacy installation flow, which installs a second function that copies the forwarder code from GitHub to an S3 bucket. Defaults to true.
635+
636+
`LAYER_ARN`
637+
: ARN for the layer containing the forwarder code. If empty, the script will use the version of the layer the forwarder was published with. Defaults to empty.
638+
639+
[20]: https://app.datadoghq.com/organization-settings/api-keys
640+
[13]: https://docs.datadoghq.com/getting_started/site/
641+
[21]: https://docs.datadoghq.com/logs/processing/pipelines/
642+
[2]: https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function/
643+
{{% /tab %}}
644+
{{< /tabs >}}
645+
490646
## Permissions
491647

492648
To deploy the CloudFormation Stack with the default options, you need to have the permissions below to save your Datadog API key as a secret and create an S3 bucket to store the Forwarder's code (ZIP file), and create Lambda functions (including execution roles and log groups).

0 commit comments

Comments
 (0)