# Managed Updates for Teleport Agents (legacy) --- WARNING This document describes Managed Updates for Agents v1, which is a legacy system that may be removed in future versions of Teleport. For Managed Updates v2 instructions, see [Managed Updates for Agents (v2)](https://goteleport.com/docs/upgrading/agent-managed-updates.md). --- Managed Updates v1 uses a script called `teleport-upgrade` that is provided by the `teleport-ent-updater` package and configured by the `cluster_maintenance_config` Teleport resource. Managed Updates v2 uses a binary called `teleport-update` that is provided by the `teleport` package and configured by the `autoupdate_version` and `autoupdate_config` resources. The original updater and resource are described in this document. Only Enterprise versions of Teleport can use Managed Updates v1. Please consider using [Managed Updates for Agents v2](https://goteleport.com/docs/upgrading/agent-managed-updates.md), as it provides a safer, simpler, more flexible, compatible, and reliable update experience compared to Managed Updates v1. --- COMPATIBILITY BETWEEN MANAGED UPDATES V1 AND V2 The Managed Updates v2 `teleport-update` binary is backwards-compatible with the `cluster_maintenance_config` resource. The Managed Updates v1 `teleport-upgrade` script is forwards-compatible with the `autoupdate_config` and `autoupdate_version` resources. Agents connected to the same cluster will all update to the same version. If the `autoupdate_config` resource is configured, it takes precedence over `cluster_maintenance_config`. This allows for a safe, non-breaking, incremental migration between Managed Updates v1 and v2. If `autoupdate_config` is not present and `autoupdate_version` is present, the `autoupdate_config` settings are implicitly derived from `cluster_maintenance_config`. Users of cloud-hosted Teleport Enterprise will be migrated to Managed Updates v2 in the first half of 2025 and should plan to migrate their agents to `teleport-update`. --- On cloud-hosted Teleport Enterprise accounts, users must set up Managed Updates for Teleport Agents to ensure that the version of Teleport running on agents remains compatible with the version running on the Auth Service and Proxy Service. If an agent does not maintain [version compatibility](https://goteleport.com/docs/upgrading/overview.md) with your Teleport cluster, connections to those agents will become degraded or lost. Cloud-hosted Teleport clusters are updated on a weekly basis. Major version updates are performed every 4 months. You can monitor and subscribe to the [Teleport Status](https://status.teleport.sh/) page to be notified of scheduled updates. Teleport supports managed agent updates for SystemD-based Linux distributions using `apt`, `yum`, and `zypper` package managers, as well as Kubernetes clusters. This guide explains how to enable Managed Updates v1 for Teleport Agents on Teleport Enterprise clusters, including both self-hosted and cloud-hosted clusters. ## How it works When Managed Updates are enabled, a Teleport updater is installed alongside each Teleport Agent. The updater communicates with the Teleport Proxy Service to determine when an update is available. When an update is available, the updater will update the Teleport Agent during the next maintenance window. However, if a critical update is available, the Teleport Agent will be updated outside the regular maintenance window. ## Prerequisites - Familiarity with the [Upgrading Compatibility Overview](https://goteleport.com/docs/upgrading/overview.md) guide, which describes the sequence in which to upgrade components in your cluster. - Teleport Agents that are not yet enrolled in Managed Updates. * A running Teleport Enterprise cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/get-started/deploy-community.md). * The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/installation.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` - To check that you can connect to your Teleport cluster, sign in with `tsh login`, then verify that you can run `tctl` commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email\@example.com to your Teleport username: ``` $ tsh login --proxy=teleport.example.com --user=email@example.com $ tctl status Cluster teleport.example.com Version 19.0.0-dev CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 ``` If you can connect to the cluster and run the `tctl status` command, you can use your current credentials to run subsequent `tctl` commands from your workstation. If you host your own Teleport cluster, you can also run `tctl` commands on the computer that hosts the Teleport Auth Service for full permissions. ## Step 1/4. Enable Managed Agent Updates If you are running a cloud-hosted Teleport Enterprise cluster, skip to [Step 2](#step-24-find-agents-to-enroll-in-managed-updates). Before enabling managed upgrades in your self-hosted Teleport cluster, you must enable a **version server**. This section shows you how to enable a version server in your cluster. ### Configure a maintenance schedule To enable Managed Updates in your cluster, you must create a cluster maintenance configuration. This configures a maintenance schedule for the Teleport cluster that agents use to determine when to check for upgrades. 1. Create a Teleport role that can manage cluster maintenance configurations through the `cluster_maintenance_config` dynamic resource. No preset Teleport roles provide this ability, so you will need to create one. Create a file called `cmc-editor.yaml` with the following content: ``` kind: role version: v7 metadata: name: cmc-editor spec: allow: rules: - resources: ['cluster_maintenance_config'] verbs: ['create', 'read', 'update', 'delete'] ``` Create the role resource: ``` $ tctl create cmc-editor.yaml ``` --- TIP You can also create and edit roles using the Web UI. Go to **Access -> Roles** and click **Create New Role** or pick an existing role to edit. --- 2. Add the role to your Teleport user: Assign the `cmc-editor` role to your Teleport user by running the appropriate commands for your authentication provider: **Local User** 1. Retrieve your local user's roles as a comma-separated list: ``` $ ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")') ``` 2. Edit your local user to add the new role: ``` $ tctl users update $(tsh status -f json | jq -r '.active.username') \ --set-roles "${ROLES?},cmc-editor" ``` 3. Sign out of the Teleport cluster and sign in again to assume the new role. **GitHub** 1. Open your `github` authentication connector in a text editor: ``` $ tctl edit github/github ``` 2. Edit the `github` connector, adding `cmc-editor` to the `teams_to_roles` section. The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization. Here is an example: ``` teams_to_roles: - organization: octocats team: admins roles: - access + - cmc-editor ``` 3. Apply your changes by saving and closing the file in your editor. 4. Sign out of the Teleport cluster and sign in again to assume the new role. **SAML** 1. Retrieve your `saml` configuration resource: ``` $ tctl get --with-secrets saml/mysaml > saml.yaml ``` Note that the `--with-secrets` flag adds the value of `spec.signing_key_pair.private_key` to the `saml.yaml` file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource. 2. Edit `saml.yaml`, adding `cmc-editor` to the `attributes_to_roles` section. The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization. Here is an example: ``` attributes_to_roles: - name: "groups" value: "my-group" roles: - access + - cmc-editor ``` 3. Apply your changes: ``` $ tctl create -f saml.yaml ``` 4. Sign out of the Teleport cluster and sign in again to assume the new role. **OIDC** 1. Retrieve your `oidc` configuration resource: ``` $ tctl get oidc/myoidc --with-secrets > oidc.yaml ``` Note that the `--with-secrets` flag adds the value of `spec.signing_key_pair.private_key` to the `oidc.yaml` file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource. 2. Edit `oidc.yaml`, adding `cmc-editor` to the `claims_to_roles` section. The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization. Here is an example: ``` claims_to_roles: - name: "groups" value: "my-group" roles: - access + - cmc-editor ``` 3. Apply your changes: ``` $ tctl create -f oidc.yaml ``` 4. Sign out of the Teleport cluster and sign in again to assume the new role. 3. Create a cluster maintenance config in a file called `cmc.yaml`. The following example allows maintenance on Monday, Wednesday and Friday between 02:00 and 03:00 UTC: ``` kind: cluster_maintenance_config spec: agent_upgrades: # Maintenance window start hour in UTC. # The maintenance window lasts 1 hour. utc_start_hour: 2 # Week days when maintenance is allowed # Possible values are: # - Short names: Sun, Mon, Tue, Wed, Thu, Fri, Sat # - Long names: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday weekdays: - Mon - Wed - Fri ``` 4. Apply the manifest using `tctl`: ``` $ tctl create cmc.yaml maintenance window has been updated ``` ### \[Optional] Assign the version served by the version server By default, the version server has a single `default` channel, serving the version of the Teleport Proxy Service. If you want to override the default version or add other channels you can use the `automatic_upgrades_channels` field in the Proxy Service configuration file: ``` proxy_service: enabled: true automatic_upgrades_channels: # Override the default version channel reachable at # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version default: static_version: v14.2.1 # Define a new version channel with a static version reachable at # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/m-static-channel/version my-static-channel: static_version: v14.2.0 # Define a new version channel forwarding requests to an upstream version server my-remote-channel: forward_url: https://updates.releases.teleport.dev/v1/stable/cloud ``` You must ensure that all Proxy Service instances share the same `automatic_upgrades_channels` configuration. If some Proxy Service instances are configured differently, you will experience agents flickering between versions as the version served is not consistent across instances. If your Proxy Service public address is teleport.example.com:443, you can query the version server with the following command: ``` $ curl "https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version" 19.0.0-dev ``` ## Step 2/4. Find agents to enroll in Managed Updates Use the `tctl inventory ls` command to list connected agents along with their current version. Use the `--upgrader=none` flag to list agents that are not enrolled in Managed Updates. ``` $ tctl inventory ls --upgrader=none Server ID Hostname Services Version Upgrader ------------------------------------ ------------- -------- ------- -------- 00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node v14.4.5 none ... ``` ## Step 3/4. Enroll agents on Linux servers in Managed Updates 1. For each agent ID returned by the `tctl inventory ls` command, copy the ID and run the following `tctl` command to access the host via `tsh`: ``` $ HOST=00000000-0000-0000-0000-000000000000 $ USER=root $ tsh ssh "${USER?}@${HOST?}" ``` 2. Determine the Teleport version to install by querying the Teleport Proxy Service. This way, the Teleport installation has the same major version as the updater. Replace stable/cloud with the name of your update channel. For cloud-hosted Teleport Enterprise accounts, this is always `stable/cloud`: ``` $ TELEPORT_VERSION="$(curl https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/stable/cloud/version | sed 's/v//')" ``` 3. Ensure that the Teleport repository is properly configured to use the stable/cloud channel, and install the `teleport-ent-updater` package. You must install `teleport-ent-updater` on each agent you would like to enroll into Managed Updates: **Managed Teleport Enterprise** ``` $ curl https://cdn.teleport.dev/install.sh | bash -s ${TELEPORT_VERSION?} cloud ``` **Self-Hosted Teleport Enterprise** 1. Follow the instructions in the Teleport [installation guide](https://goteleport.com/docs/installation/linux.md#package-repositories) to install the `teleport` binary on your Linux server for your package manager. 2. Using your package manager, install `teleport-ent-updater` on the server where you installed `teleport`. For example: ``` $ apt-get install -y teleport-ent-updater ``` The installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the [installation guide](https://goteleport.com/docs/installation/linux.md). 4. Confirm that the version of the `teleport` binary is the one you expect: ``` $ teleport version ``` Running the agent as a non-root user If you changed the agent user to run as non-root, create `/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user: ``` $ sudo mkdir -p /etc/teleport-upgrade.d/ $ sudo touch /etc/teleport-upgrade.d/schedule $ sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule ``` 5. Verify that the upgrader can see your version endpoint by checking for upgrades: ``` $ sudo teleport-upgrade dry-run ``` 6. You should see one of the following messages, depending on the target version you are currently serving: ``` no upgrades available (1.2.3 == 1.2.3) an upgrade is available (1.2.3 -> 2.3.4) ``` `teleport-upgrade` may display warnings about not having a valid upgrade schedule. This is expected immediately after install as the maintenance schedule might not be exported yet. ## Step 4/4. Enroll Kubernetes agents in Managed Updates This section assumes that the name of your `teleport-kube-agent` release is `teleport-agent`, and that you have installed it in the `teleport` namespace. 1. Confirm that you are using the Teleport Enterprise edition of the `teleport-kube-agent` chart. You should see the following when you query your `teleport-kube-agent` release: ``` $ helm -n `teleport` get values `teleport-agent` -o json | jq '.enterprise' true ``` If another value such as `null` is returned, update your existing agent `values.yaml` to use the Enterprise version: ``` enterprise: true ``` 2. Add the following chart values to the values file for the `teleport-kube-agent` chart: ``` updater: enabled: true ``` 3. Update the Teleport Helm repository to include any new versions of the `teleport-kube-agent` chart: ``` $ helm repo update teleport ``` 4. Update the Helm chart release with the new values: **Cloud-Hosted** ``` $ helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \ --values=values.yaml \ --version="18.7.3" ``` **Self-Hosted** ``` $ helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \ --values=values.yaml \ --version="19.0.0-dev" ``` 5. You can validate the updater is running properly by checking if its pod is ready: ``` $ kubectl -n teleport-agent get pods NAME READY STATUS RESTARTS AGE -0 1/1 Running 0 14m -1 1/1 Running 0 14m -2 1/1 Running 0 14m -updater-d9f97f5dd-v57g9 1/1 Running 0 16m ``` 6. Check for any deployment issues by checking the updater logs: ``` $ kubectl -n teleport logs deployment/teleport-agent-updater 2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"} ``` ## Troubleshooting Teleport Agents are not updated immediately when a new version of Teleport is released, and agent updates can lag behind the cluster by a few days. If the Teleport Agent has not been automatically updating for several weeks, you can consult the updater logs to help troubleshoot the problem: ``` $ journalctl -u teleport-upgrade ``` ### Troubleshooting managed agent upgrades on Kubernetes The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event. 1. Any deployment update will send an event, so you can trigger the upgrader by annotating the resource: ``` $ kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1' ``` 2. To suspend Managed Updates for an agent, annotate the agent deployment with `teleport.dev/skipreconcile: "true"`, either by setting the `annotations.deployment` value in Helm, or by patching the deployment directly with `kubectl`. ### Troubleshooting Managed Updates on Linux 1. If an agent is not automatically upgraded, you can invoke the upgrader manually and look at its logs: ``` $ sudo teleport-upgrade run ``` 2. To suspend Managed Updates for an agent, disable the systemd timer: ``` $ sudo systemctl disable --now teleport-upgrade.timer ``` 3. To enable and start the systemd timer after suspending: ``` $ sudo systemctl enable --now teleport-upgrade.timer ``` ## Working with the teleport-upgrade tool This section explains advanced usage of the v1 `teleport-upgrade` script for Linux. Note that this tool is no longer actively developed, and the Managed Updates v2 `teleport-update` binary should be used instead. For Managed Updates v2 instructions, see [Managed Updates for Agents (v2)](https://goteleport.com/docs/upgrading/agent-managed-updates.md). ### `teleport-upgrade` commands The `teleport-upgrade` tool provides some basic commands to verify and perform an update of the Teleport Agent. ``` $ teleport-upgrade help USAGE: /usr/sbin/teleport-upgrade Tool for automatic upgrades of Teleport Agents. Commands: run check for and potentially apply a teleport upgrade. dry-run check for new teleport version but do not upgrade. force performs an upgrade if an upgrade is available. version print the current version of /usr/sbin/teleport-upgrade. help show this help text. ``` The `dry-run` command can be used to check for an available update without performing an update. ``` Example output when teleport is already on the latest compatible version. $ teleport-upgrade dry-run [i] no upgrades available (14.3.14 == 14.3.14) [ 582 ] Example output when an update is available. $ teleport-upgrade dry-run [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] [i] within maintenance window, upgrade will be attempted. [ 596 ] ``` The `run` command performs an update if available. ``` Successful teleport update from 13.4.14 to 14.3.14. $ teleport-upgrade run [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] [i] within maintenance window, upgrade will be attempted. [ 596 ] [i] attempting apt install teleport-ent=14.3.14... [ 480 ] [...] [i] gracefully restarting Teleport (if already running) [ 449 ] Teleport updates are not attempted when outside the maintenance window. $ teleport-upgrade run [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] [i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ] ``` The `force` command performs an update immediately even when outside the maintenance window. ``` $ teleport-upgrade force [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] [i] attempting apt install teleport-ent=14.3.14... [ 480 ] [...] [i] gracefully restarting Teleport (if already running) [ 449 ] ``` ### Configuring the `teleport-upgrade` tool 1. Create the upgrade configuration directory: ``` $ sudo mkdir -p /etc/teleport-upgrade.d/ ``` 2. If you changed the agent user to run as non-root, create `/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user, assigning your-teleport-user to the name of your Teleport user. Otherwise, you can skip this step: ``` $ sudo touch /etc/teleport-upgrade.d/schedule $ sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule ``` 1. Configure the upgrader to connect to your version server and subscribe to the right release channel: ``` $ echo "teleport.example.com:443/v1/webapi/automaticupgrades/channel/default" | sudo tee /etc/teleport-upgrade.d/endpoint ``` Make sure not to include `https://` as a prefix to the server address, nor suffix the endpoint with `/version`. ### Choosing a release channel When [configuring the updater](#configuring-the-teleport-upgrade-tool), you can select a release channel. The following channels are available for APT, YUM, and Zypper repos: | Channel name | Description | | ---------------- | ---------------------------------------------------------------------------- | | `stable/` | Receives releases for the specified major release line, i.e. `v19` | | `stable/cloud` | Rolling channel that receives releases compatible with current Cloud version | | `stable/rolling` | Rolling channel that receives all published Teleport releases | ### Updating the updater The updater is designed to be minimal and relatively stable, but the updater will receive updates on occasion. Currently, the updater does not have the capability to update itself. Updates to the `teleport-ent-updater` package must be done manually. The `teleport-ent-updater` will be backwards compatible with older versions of Teleport, so you can always update the `teleport-ent-updater` package to the latest available version. ### Version locking As of Teleport `15.1.10`, a version locking mechanism is enabled by the updater. This mechanism locks the version of Teleport and prevents manual updates of the Teleport package. This prevents unintentional updates during routine system maintenance, or an accidental update by a user. The updater still has the capability to update the Teleport package, and all updates of Teleport are expected to be performed by the updater. The version locking mechanism is implemented using the features of the package managers. In case a user would like to manually update the Teleport version, this can be done with the following commands. With the `apt` package manager CLI, the `--allow-change-held-packages` flag must be provided to bypass the version lock. ``` $ apt-get install --allow-change-held-packages "teleport-ent=" ``` With the `yum` package manager CLI, the `--disableexcludes="teleport"` flag must be provided to bypass the version lock. ``` $ yum install --disablerepo="*" --enablerepo="teleport" --disableexcludes="teleport" "teleport-ent-" ``` With the `zypper` package manager CLI, the lock must be disabled and then re-enabled after the update. ``` $ zypper removelock "teleport-ent" $ zypper install --repo="teleport" "teleport-ent-" $ zypper addlock "teleport-ent" ```