# Export Teleport Audit Events with Datadog Teleport's Event Handler plugin receives audit events from the Teleport Auth Service and forwards them to your log management solution, letting you perform historical analysis, detect unusual behavior, and form a better understanding of how users interact with your Teleport cluster. Datadog is a SAAS monitoring and security platform. In this guide, we'll explain how to forward Teleport audit events to Datadog using Fluentd. ## How it works The Teleport Event Handler authenticates to the Teleport Auth Service to receive audit events over a gRPC stream, then sends those events to Fluentd as JSON payloads over a secure channel established via mutual TLS: ![Architecture of the setup shown in this guide](/docs/assets/images/datadog-diagram-6d6b654e4e3420b1c42ac8924b5ee643.png) Since the Datadog Agent can only receive logs from remote sources as JSON-encoded bytes over a [TCP or UDP connection](https://docs.datadoghq.com/agent/logs/?tab=tailfiles#custom-log-collection), the Teleport Event Handler needs to send its HTTPS payloads without using the Datadog Agent. Fluentd handles authentication to the Datadog API. ## Prerequisites - A [Datadog](https://www.datadoghq.com/) account. - Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new `fluent.conf` file you can integrate into an existing Fluentd system, or use with a fresh setup. - A server, virtual machine, Kubernetes cluster, or Docker environment to run the Teleport Event Handler plugin. This guide requires you to have completed one of the Event Handler setup guides: - [Set up the Event Handler with tctl](https://goteleport.com/docs/zero-trust-access/export-audit-events/event-handler-setup.md) - [Set up the Event Handler with the Teleport Kubernetes Operator](https://goteleport.com/docs/zero-trust-access/export-audit-events/event-handler-setup-operator.md) The instructions below demonstrate a local test of the Event Handler plugin on your workstation. You will need to adjust paths, ports, and domains for other environments. ## Step 1/2. Install the Fluentd output plugin for Datadog In order for Fluentd to communicate with Datadog, it requires the [Fluentd output plugin for Datadog](https://github.com/DataDog/fluent-plugin-datadog). Install the plugin on your Fluentd host using either `gem` or the `td-agent`, if installed: ``` Using Gem $ gem install fluent-plugin-datadog Using td-agent $ /usr/sbin/td-agent-gem install fluent-plugin-datadog ``` --- TESTING LOCALLY? If you're running Fluentd in a local Docker container for testing, you can adjust the entrypoint to an interactive shell as the root user, so you can install the plugin before starting Fluentd: ``` $ docker run -u $(id -u root):$(id -g root) -p 8888:8888 -v $(pwd):/keys -v \ $(pwd)/fluent.conf:/fluentd/etc/fluent.conf --entrypoint=/bin/sh -i --tty fluent/fluentd:edge From the container shell: $ gem install fluent-plugin-datadog $ fluentd -c /fluentd/etc/fluent.conf ``` --- ### Configure Fluentd We will modify the `fluent.conf` file generated in the prerequisite setup guide. 1. Visit Datadog and generate an API key for Fluentd by following the [Datadog documentation](https://docs.datadoghq.com/account_management/api-app-keys). 2. Copy the API key and use it to add a new `` block to `fluent.conf`: ``` @type datadog @id awesome_agent api_key abcd123-insecure-do-not-use-this host http-intake.logs.us5.datadoghq.com # Optional parameters dd_source teleport ``` 3. Edit your configuration as follows: - Add your API key to the `api_key` field. - Adjust the `host` value to match your Datadog site. See the Datadog [Log Collection and Integrations](https://docs.datadoghq.com/logs/log_collection/?tab=host) guide to determine the correct value. - `dd_source` is an optional field you can use to filter these logs in the Datadog UI. - Adjust `ca_path`, `cert_path` and `private_key_path` to point to the credential files generated in the prerequisite setup guide. If you're testing locally, the Docker command above already mounted the current working directory to `keys/` in the container. 4. Restart Fluentd after saving the changes to `fluent.conf`. ## Step 2/2. Run the Event Handler plugin In this section, you will modify the Event Handler configuration you generated and run the Event Handler to test your configuration. ### Configure the Event Handler Edit the configuration for the Event Handler, depending on your installation method. **Executable** Earlier, we generated a file called `teleport-event-handler.toml` to configure the Teleport Event Handler. This file includes setting similar to the following: ``` storage = "./storage" timeout = "10s" batch = 20 # concurrency is the number of concurrent sessions to process. By default, this is set to 5. concurrency = 5 # The window size configures the duration of the time window for the event handler # to request events from Teleport. By default, this is set to 24 hours. # Reduce the window size if the events backend cannot manage the event volume # for the default window size. # The window size should be specified as a duration string, parsed by Go's time.ParseDuration. window-size = "24h" # types is a comma-separated list of event types to search when forwarding audit # events. For example, to limit forwarded events to user logins # and new Access Requests, you can assign this field to # "user.login,access_request.create". types = "" # skip-event-types is a comma-separated list of audit log event types to skip. # For example, to forward all audit events except for new app deletion events, # you can include the following assignment: # skip-event-types = ["app.delete"] skip-event-types = [] # skip-session-types is a comma-separated list of session recording event types to skip. # For example, to forward all session events except for malformed SQL packet # events, you can include the following assignment: # skip-session-types = ["db.session.malformed_packet"] skip-session-types = [] [forward.fluentd] ca = /home/bob/event-handler/ca.crt cert = /home/bob/event-handler/client.crt key = /home/bob/event-handler/client.key url = "https://fluentd.example.com:8888/test.log" # The Event Handler appends `..log` to `session-url` when sending # session recording events. For example, if `session-url` is # `https://fluentd.example.com:8888/session`, the actual requests are sent to # paths like `/session..log`. Ensure that your log collector's # tag matching or routing rules account for this suffix (e.g., use `session.*` # as a match pattern in Fluentd or Fluent Bit). session-url = "https://fluentd.example.com:8888/session" [teleport] addr = teleport.example.com:443 identity = "identity" ``` **Helm Chart** Earlier, we generated a file called `teleport-plugin-event-handler-values.yaml` to configure the Teleport Event Handler. This file includes setting similar to the following: ``` eventHandler: storagePath: "./storage" timeout: "10s" batch: 20 # concurrency is the number of concurrent sessions to process. By default, this is set to 5. concurrency: 5 # The window size configures the duration of the time window for the event handler # to request events from Teleport. By default, this is set to 24 hours. # Reduce the window size if the events backend cannot manage the event volume # for the default window size. # The window size should be specified as a duration string, parsed by Go's time.ParseDuration. windowSize: "24h" # types is a list of event types to search when forwarding audit # events. For example, to limit forwarded events to user logins # and new Access Requests, you can assign this field to: # ["user.login", "access_request.create"] types: [] # skipEventTypes lists types of audit events to skip. For example, to forward all # audit events except for new app deletion events, you can assign this to: # ["app.delete"] skipEventTypes: [] # skipSessionTypes lists types of session recording events to skip. For example, # to forward all session events except for malformed SQL packet events, # you can assign this to: # ["db.session.malformed_packet"] skipSessionTypes: [] teleport: address: teleport.example.com:443 identitySecretName: teleport-event-handler-identity identitySecretPath: identity fluentd: url: "https://fluentd.fluentd.svc.cluster.local/events.log" # The Event Handler appends `..log` to `session-url` when sending # session recording events. For example, if `session-url` is # `https://fluentd.example.com:8888/session`, the actual requests are sent to # paths like `/session..log`. Ensure that your log collector's # tag matching or routing rules account for this suffix (e.g., use `session.*` # as a match pattern in Fluentd or Fluent Bit). sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log" certificate: secretName: "teleport-event-handler-client-tls" caPath: "ca.crt" certPath: "client.crt" keyPath: "client.key" persistentVolumeClaim: enabled: true ``` **Helm Chart with Kubernetes Operator** Your helm configuration file `teleport-plugin-event-handler-values.yaml` should contain settings similar to the following: ``` eventHandler: storagePath: "./storage" timeout: "10s" batch: 20 # concurrency is the number of concurrent sessions to process. By default, this is set to 5. concurrency: 5 # The window size configures the duration of the time window for the event handler # to request events from Teleport. By default, this is set to 24 hours. # Reduce the window size if the events backend cannot manage the event volume # for the default window size. # The window size should be specified as a duration string, parsed by Go's time.ParseDuration. windowSize: "24h" # types is a list of event types to search when forwarding audit # events. For example, to limit forwarded events to user logins # and new Access Requests, you can assign this field to: # ["user.login", "access_request.create"] types: [] # skipEventTypes lists types of audit events to skip. For example, to forward all # audit events except for new app deletion events, you can assign this to: # ["app.delete"] skipEventTypes: [] # skipSessionTypes lists types of session recording events to skip. For example, # to forward all session events except for malformed SQL packet events, # you can assign this to: # ["db.session.malformed_packet"] skipSessionTypes: [] crd: create: true namespace: operator-namespace tbot: enabled: true clusterName: teleport.example.com teleportProxyAddress: teleport.example.com:443 fluentd: url: "https://fluentd.fluentd.svc.cluster.local/events.log" sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log" certificate: secretName: "teleport-event-handler-client-tls" caPath: "ca.crt" certPath: "client.crt" keyPath: "client.key" persistentVolumeClaim: enabled: true ``` Update the following fields. **Executable** **`[teleport]`** `addr`: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443 `identity`: Fill this in with the path to the identity file you exported earlier. If you are providing credentials to the Event Handler using a `tbot` binary that runs on a Linux server, make sure the value of `identity` in the Event Handler configuration is the same as the path of the identity file you configured `tbot` to generate, `/opt/machine-id/identity`. **`[forward.fluentd]`** `ca`: Include the path to the CA certificate: /home/bob/event-handler/ca.crt `cert`: Include the path to the Fluentd client certificate. /home/bob/event-handler/client.crt `key`: Include the path to the Fluentd client key. /home/bob/event-handler/client.key `url`: Include the Fluentd URL where the audit event logs will be sent. `session-url`: Include the Fluentd URL where the session logs will be sent. **Helm Chart** **`teleport`** `address`: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443 `identitySecretName`: Fill in the `identitySecretName` field with the name of the Kubernetes secret you created earlier. `identitySecretPath`: Fill in the `identitySecretPath` field with the path of the identity file within the Kubernetes secret. If you have followed the instructions above, this will be `identity`. **`fluentd`** `url`: Include the Fluentd URL where the audit event logs will be sent. `sessionUrl`: Include the Fluentd URL where the session logs will be sent. `certificate.secretName`: Include the name of the Kubernetes secret containing the Fluentd client credentials. If you have followed the instructions above, this will be `teleport-event-handler-client-tls`. `certificate.caPath`: Include the path to the CA certificate inside the secret. `certificate.certPath`: Include the path to the Fluentd client certificate inside the secret. `certificate.keyPath`: Include the path to the Fluentd client key inside the secret. **Helm Chart with Kubernetes Operator** **`crd`** `namespace`: Include the namespace that the Teleport Kubernetes Operator is running in: operator-namespace `tokenSpecOverride`: Optionally include a specific join token specification for the bot user that `tbot` will authenticate as. **`tbot`** `clusterName`: Include the name of your Teleport cluster: teleport.example.com `teleportProxyAddress`: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account: teleport.example.com:443 **`fluentd`** `url`: Include the Fluentd URL where the audit event logs will be sent. `sessionUrl`: Include the Fluentd URL where the session logs will be sent. `certificate.secretName`: Include the name of the Kubernetes secret containing the Fluentd client credentials. If you have followed the instructions above, this will be `teleport-event-handler-client-tls`. `certificate.caPath`: Include the path to the CA certificate inside the secret. `certificate.certPath`: Include the path to the Fluentd client certificate inside the secret. `certificate.keyPath`: Include the path to the Fluentd client key inside the secret. ### Start the Event Handler Start the Teleport Event Handler by following the instructions below. **Linux server** Copy the `teleport-event-handler.toml` file to `/etc` on your Linux server. Update the settings within the `toml` file to match your environment. Make sure to use absolute paths on settings such as `identity` and `storage`. Files and directories in use should only be accessible to the system user executing the `teleport-event-handler` service such as `/var/lib/teleport-event-handler`. Next, create a systemd service definition at the path `/usr/lib/systemd/system/teleport-event-handler.service` with the following content: ``` [Unit] Description=Teleport Event Handler After=network.target [Service] Type=simple Restart=always ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled=true ExecReload=/bin/kill -HUP $MAINPID PIDFile=/run/teleport-event-handler.pid [Install] WantedBy=multi-user.target ``` If you are not using Machine & Workload Identity to provide short-lived credentials to the Event Handler, you can remove the `--teleport-refresh-enabled true` flag. Enable and start the plugin: ``` $ sudo systemctl enable teleport-event-handler $ sudo systemctl start teleport-event-handler ``` Choose when to start exporting events You can configure when you would like the Teleport Event Handler to begin exporting events when you run the `start` command. This example will start exporting from May 5th, 2021: ``` $ teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z" ``` You can only determine the start time once, when first running the Teleport Event Handler. If you want to change the time frame later, remove the plugin state directory that you specified in the `storage` field of the handler's configuration file. Once the Teleport Event Handler starts, you will see notifications about scanned and forwarded events: ``` $ sudo journalctl -u teleport-event-handler DEBU Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-21 18:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140 ... ``` **Helm chart** Run the following command on your workstation: ``` $ helm install teleport-plugin-event-handler teleport/teleport-plugin-event-handler \ --values teleport-plugin-event-handler-values.yaml \ --version 19.0.0-dev ``` **Local Docker container** Navigate to the directory where you ran the `configure` command earlier and execute the following command: ``` $ docker run --network host -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:19.0.0-dev start --config=teleport-event-handler.toml ``` This command joins the Event Handler container to the preset `host` network, which uses the Docker host networking mode and removes network isolation, so the Event Handler can communicate with the Fluentd container on `localhost`. The Logs view in Datadog should now report your Teleport cluster events: ![Datadog Logs](/docs/assets/images/datadog-logs-128146d270e231d575ea67faf784ad3e.png) ## Troubleshooting connection issues If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that: - The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the `--ttl` flag in the `tctl auth sign` command, which is 12 hours by default. - In your Teleport Event Handler configuration file, you have provided the correct host *and* port for the Teleport Proxy Service. - Start the Fluentd container prior to starting the Teleport Event Handler. The Event Handler will attempt to connect to Fluentd immediately upon startup. ## Next steps - Review the Fluentd output plugin for Datadog [README file](https://github.com/DataDog/fluent-plugin-datadog/blob/master/README.md) to learn how to customize the log format entering Datadog. - To see all of the options you can set in the values file for the `teleport-plugin-event-handler` Helm chart, consult our [reference guide](https://goteleport.com/docs/reference/helm-reference/teleport-plugin-event-handler.md).