# Running an HA Teleport cluster using Microsoft Azure, AKS, and Helm In this guide, we'll go through how to set up a High Availability Teleport cluster with multiple replicas in Kubernetes using Teleport Helm charts and Microsoft Azure managed services (Kubernetes Services, Database for PostgreSQL, Blob Storage). --- HAVE AN EXISTING TELEPORT CLUSTER? If you are already running Teleport on another platform, you can use your existing Teleport deployment to access your Kubernetes cluster. [Follow our guide](https://goteleport.com/docs/enroll-resources/kubernetes-access/getting-started.md) to connect your Kubernetes cluster to Teleport. --- --- TIP Teleport Enterprise Cloud takes care of this setup for you so you can provide secure access to your infrastructure right away. Get started with a [free trial](https://goteleport.com/signup?t_source=docs) of Teleport Enterprise Cloud. --- ## How it works The `teleport-cluster` Helm chart deploys the Teleport Auth Service and Teleport Proxy Service on your Azure Kubernetes Service cluster. The chart requires the following resources, which we show you how to create in this guide: - **IAM permissions for the Teleport Auth Service**. The Auth Service requires permissions to manage resources on its backend. - **cert-manager** for obtaining and renewing TLS credentials that the Proxy Service uses to run its HTTPS server. - **IAM permissions for cert-manager**. In the setup we show in this guide, `cert-manager` modifies DNS records to demonstrate domain ownership and receive TLS credentials from Let's Encrypt. To do so, the Proxy Service completes the [ACME DNS-01 challenge](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge). - **Teleport Auth Service backend components:** PostgreSQL database and an Amazon S3-compatible object storage solution. ## Prerequisites - [Kubernetes](https://kubernetes.io) >= v1.17.0 - [Helm](https://helm.sh) >= v3.4.2 Teleport's charts require the use of Helm version 3. You can [install Helm 3 by following these instructions](https://helm.sh/docs/intro/install/). Throughout this guide, we will assume that you have the `helm` and `kubectl` binaries available in your `PATH`: ``` $ helm version version.BuildInfo{Version:"v3.4.2"} $ kubectl version Client Version: version.Info{Major:"1", Minor:"17+"} Server Version: version.Info{Major:"1", Minor:"17+"} ``` Best practices for production security When running Teleport in production, you should adhere to the following best practices to avoid security incidents: - Avoid using `sudo` in production environments unless it's necessary. - Create new, non-root, users and use test instances for experimenting with Teleport. - Run Teleport's services as a non-root user unless required. Only the SSH Service requires root access. Note that you will need root permissions (or the `CAP_NET_BIND_SERVICE` capability) to make Teleport listen on a port numbered < `1024` (e.g. `443`). - Follow the **principle of least privilege**. Don't give users permissive roles when more a restrictive role will do. For example, don't assign users the built-in `access,editor` roles, which give them permissions to access and edit all cluster resources. Instead, define roles with the minimum required permissions for each user and configure **Access Requests** to provide temporary elevated permissions. - When you enroll Teleport resources—for example, new databases or applications—you should save the invitation token to a file. If you enter the token directly on the command line, a malicious user could view it by running the `history` command on a compromised system. You should note that these practices aren't necessarily reflected in the examples used in documentation. Examples in the documentation are primarily intended for demonstration and for development environments. In addition, you will need `azure-cli` 2.51 or later to follow along these instructions. Reference the Azure docs on [how to install the Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli). After installing it, make sure you are logged in by typing `az login`. This guide assumes that your user has permissions to create Azure Database for PostgreSQL instances, Azure Blob Storage accounts, and Managed Identities, and has the ability to add role assignments for those. You will also need an Azure DNS zone, and access to an AKS cluster with cert-manager [installed](https://cert-manager.io/docs/installation/) and configured to [issue certificates for said Azure DNS zone](https://cert-manager.io/docs/configuration/acme/dns01/azuredns/). In this guide we'll use [workload identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) to authenticate Teleport to PostgreSQL and Blob Storage, so you'll need to [enable workload identity and the OIDC issuer in your AKS cluster](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#update-an-existing-aks-cluster) if they're not enabled already: ``` $ az aks update --resource-group aks-rg --name aks-name --enable-oidc-issuer --enable-workload-identity ``` ## Step 1/5. Add the Teleport Helm chart repository Configure Helm to fetch Teleport charts from the Teleport Helm repository: ``` $ helm repo add teleport https://charts.releases.teleport.dev ``` Refresh the local Helm cache by fetching the latest charts: ``` $ helm repo update ``` ## Step 2/5. Set up PostgreSQL and Blob Storage For convenience, we'll create all the resources necessary in a brand new resource group; if you want to use an existing one, you can skip this step. Assign region to your Azure region: ``` $ az group create --name teleport-rg --location region ``` We're going to need a Managed Identity for Teleport to use these services. ``` $ az identity create --resource-group teleport-rg --name teleport-id ``` The recommended HA deployment of Teleport on Azure stores the cluster state and the audit log entries in a PostgreSQL instance. In this guide we'll create a publicly accessible one, but you can restrict it to your AKS cluster's IP address, or you can create it attached to the same virtual network that the cluster is using, instead. Depending on your region, you might be able to use `ZoneRedundant` high availability, or you might have to use `SameZone` high availability. ``` $ az postgres flexible-server create --resource-group teleport-rg --name teleport-pg \ --microsoft-entra-auth Enabled --password-auth Disabled \ --version 15 --high-availability SameZone --public-access All $ az postgres flexible-server parameter set --resource-group teleport-rg --server-name teleport-pg \ --name wal_level --value logical $ az postgres flexible-server restart --resource-group teleport-rg --name teleport-pg $ az postgres flexible-server microsoft-entra-admin create --resource-group teleport-rg --server-name teleport-pg \ --display-name pguser --type ServicePrincipal \ --object-id "$(az identity show --resource-group teleport-rg --name teleport-id --query principalId -o tsv)" ``` Teleport will store session recordings in a Blob Storage account. Optionally, access can be restricted to just the AKS outbound address, or the account can be made part of the virtual network that the AKS cluster is using. ``` $ az storage account create --resource-group teleport-rg --name teleportblob \ --allow-blob-public-access false $ az role assignment create --role "Storage Blob Data Owner" --assignee-principal-type ServicePrincipal \ --assignee-object-id "$(az identity show --resource-group teleport-rg --name teleport-id --query principalId -o tsv)" \ --scope "$(az storage account show --resource-group teleport-rg --name teleportblob --query id -o tsv)" ``` We'll use Workload Identity to authenticate to those services, so we'll add federated credentials for the Teleport service account used by the Auth Service. ``` $ az identity federated-credential create --resource-group teleport-rg --identity-name teleport-id \ --name aks --audience api://AzureADTokenExchange \ --subject system:serviceaccount:teleport-ns:teleport-release \ --issuer "$(az aks show --resource-group aks-rg --name aks-name --query oidcIssuerProfile.issuerUrl -o tsv)" ``` ## Step 3/5. Set values to configure the cluster License Secret Before you can install Teleport Enterprise in your Kubernetes cluster, you will need to create a secret that contains your Teleport license information. The Teleport Auth Service reads a license file to authenticate your Teleport Enterprise account. To obtain your license file, navigate to your Teleport account dashboard and log in. You can start at [teleport.sh](https://teleport.sh) and enter your Teleport account name (e.g. my-company). After logging in you will see a "GENERATE LICENSE KEY" button, which will generate a new license file and allow you to download it. ![License File Download](/docs/assets/images/license-927c35c955fa526748558be430c4b102.png) Create a secret from your license file. Teleport will automatically discover this secret as long as your file is named `license.pem`. ``` $ kubectl create namespace teleport-ns $ kubectl -n teleport-ns create secret generic license --from-file=license.pem ``` Now we'll configure the `teleport-cluster` Helm chart to use the `azure` mode. First get the client ID for the `teleport-id` identity: ``` $ az identity show --resource-group teleport-rg --name teleport-id --query clientId -o tsv teleport-client-id-uuid-123456789012 ``` Then create a file called `azure-values.yaml` containing the values you've selected above: **Open Source** ``` chartMode: azure # Name of your cluster. Use the FQDN you intend to configure in DNS later clusterName: teleport.example.com azure: databaseHost: "teleport-pg.postgres.database.azure.com" databaseUser: "pguser" sessionRecordingStorageAccount: "teleportblob.blob.core.windows.net" # Whether to mirror audit log entries to stdout in JSON format (useful for external log collectors) auditLogMirrorOnStdout: false clientID: "teleport-client-id-uuid-123456789012" highAvailability: # Number of replicas to configure replicaCount: 2 certManager: # Enable cert-manager support to get TLS certificates enabled: true # Name of the cert-manager GlobalIssuer or Issuer to use issuerName: letsencrypt-production issuerKind: ClusterIssuer # If you are running Kubernetes 1.23 or above, disable PodSecurityPolicies podSecurityPolicy: enabled: false ``` **Enterprise** ``` chartMode: azure # Name of your cluster. Use the FQDN you intend to configure in DNS later clusterName: teleport.example.com azure: databaseHost: "teleport-pg.postgres.database.azure.com" databaseUser: "pguser" sessionRecordingStorageAccount: "teleportblob.blob.core.windows.net" # Whether to mirror audit log entries to stdout in JSON format (useful for external log collectors) auditLogMirrorOnStdout: false clientID: "teleport-client-id-uuid-123456789012" highAvailability: # Number of replicas to configure replicaCount: 2 certManager: # Enable cert-manager support to get TLS certificates enabled: true # Name of the cert-manager GlobalIssuer or Issuer to use issuerName: letsencrypt-production issuerKind: ClusterIssuer # If you are running Kubernetes 1.23 or above, disable PodSecurityPolicies podSecurityPolicy: enabled: false # Indicate that this is a Teleport Enterprise deployment enterprise: true ``` Install the chart with the values from your `azure-values.yaml` file using this command: ``` $ helm install teleport-release teleport/teleport-cluster \ --create-namespace --namespace teleport-ns \ --values azure-values.yaml ``` --- NOTE You cannot change the `clusterName` after the cluster is configured, so make sure you choose wisely. We recommend using the fully-qualified domain name that you'll use for external access to your Teleport cluster. --- Once the chart is installed, you can use `kubectl` commands to view the deployment: ``` $ kubectl --namespace teleport-ns get all NAME READY STATUS RESTARTS AGE pod/teleport-auth-57989d4cb-4q2ds 1/1 Running 0 22h pod/teleport-auth-57989d4cb-rtrzn 1/1 Running 0 22h pod/teleport-proxy-c6bf55cfc-w96d2 1/1 Running 0 22h pod/teleport-proxy-c6bf55cfc-z256w 1/1 Running 0 22h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/teleport LoadBalancer 10.40.11.180 34.138.177.11 443:30258/TCP,3023:31802/TCP,3026:32182/TCP,3024:30101/TCP,3036:30302/TCP 22h service/teleport-auth ClusterIP 10.40.8.251 3025/TCP,3026/TCP 22h service/teleport-auth-v13 ClusterIP None 22h service/teleport-auth-v14 ClusterIP None 22h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/teleport-auth 2/2 2 2 22h deployment.apps/teleport-proxy 2/2 2 2 22h NAME DESIRED CURRENT READY AGE replicaset.apps/teleport-auth-57989d4cb 2 2 2 22h replicaset.apps/teleport-proxy-c6bf55cfc 2 2 2 22h ``` ## Step 4/5. Set up DNS We'll now set up DNS `A` records for `teleport.example.com` and `*.teleport.example.com`, using your Azure DNS zone rooted at `example.com`. If you're using a different DNS hosting service, follow their instructions instead. ``` $ PREFIX=teleport $ ZONE=example.com $ ZONE_RG=dns-rg $ external_ip="$(kubectl --namespace teleport-ns get service/teleport-release -o jsonpath='{.status.loadBalancer.ingress[*].ip}')" $ az network dns record-set a add-record --resource-group ${ZONE_RG} --zone-name ${ZONE} --record-set-name ${PREFIX} --ipv4-address "${external_ip}" $ az network dns record-set a add-record --resource-group ${ZONE_RG} --zone-name ${ZONE} --record-set-name "*.${PREFIX}" --ipv4-address "${external_ip}" ``` ## Step 5/5. Create a Teleport user Create a user to be able to log into Teleport. This needs to be done on the Teleport Auth Service, so we can run the command using `kubectl`: **Open Source** ``` $ kubectl --namespace teleport-ns exec deploy/teleport-release-auth -- tctl users add test --roles=access,editor User "test" has been created but requires a password. Share this URL with the user to complete user setup, link is valid for 1h: https://teleport.example.com:443/web/invite/91cfbd08bc89122275006e48b516cc68 NOTE: Make sure teleport.example.com:443 points at a Teleport proxy that users can access. ``` **Enterprise** ``` $ kubectl --namespace teleport-ns exec deploy/teleport-release-auth -- tctl users add test --roles=access,editor,reviewer User "test" has been created but requires a password. Share this URL with the user to complete user setup, link is valid for 1h: https://teleport.example.com:443/web/invite/91cfbd08bc89122275006e48b516cc68 NOTE: Make sure teleport.example.com:443 points at a Teleport proxy that users can access. ``` Load the user creation link to create a password and set up multi-factor authentication for the Teleport user via the web UI. ### High Availability In this guide, we have configured 2 replicas. This can be changed after cluster creation by altering the `highAvailability.replicaCount` value [using `helm upgrade` as detailed below](#upgrading-the-cluster-after-deployment). ## Upgrading the cluster after deployment To make changes to your Teleport cluster after deployment, you can use `helm upgrade`. Helm defaults to using the latest version of the chart available in the repo, which will also correspond to the latest version of Teleport. You can make sure that the repo is up to date by running `helm repo update`. If you want to use a different version of Teleport, pass the `--version` argument to Helm: ``` $ helm upgrade --version 14.0.0 \ teleport-release teleport/teleport-cluster \ --namespace teleport-ns \ -f azure-values.yaml ``` Here's an example where we set the chart to use 3 replicas: **Using values.yaml** Edit your `azure-values.yaml` file from above and make the appropriate changes. Upgrade the deployment with the values from your `azure-values.yaml` file using this command: ``` $ helm upgrade teleport-release teleport/teleport-cluster \ --namespace teleport-ns \ -f azure-values.yaml ``` **Using --set via CLI** Run this command, editing your command line parameters as appropriate: ``` $ helm upgrade teleport-release teleport/teleport-cluster \ --namespace teleport-ns \ --set highAvailability.replicaCount=3 ``` --- NOTE To change `chartMode`, `clusterName` or any `azure` settings, you must first uninstall the existing chart and then install a new version with the appropriate values. --- ## Uninstalling Teleport To uninstall the `teleport-cluster` chart, use `helm uninstall`. For example: ``` $ helm --namespace teleport-ns uninstall teleport-release ``` ## Next steps Now that you have deployed a Teleport cluster, read the [Manage Access](https://goteleport.com/docs/zero-trust-access/authentication.md) section to get started enrolling users and setting up RBAC. See the [high availability section of our Helm chart reference](https://goteleport.com/docs/reference/helm-reference/teleport-cluster.md) for more details on high availability.