Securing Grafana on Kubernetes with GCP IAP, Gateway API, and Terraform

This post was originally published on vidbregar.com, where you can find the best reading experience.

Follow a step-by-step guide to secure your Grafana deployment on Kubernetes using GCP IAP, Gateway API, and Terraform. This setup helps mitigate CVE risks, enables granular access control, protects against DDoS attacks, and more.

Benefits and Drawbacks

Benefits:

• All traffic is routed through GCP IAP, mitigating risks from Grafana CVE.
• IAP supports context-aware access, enabling granular access control, such as restricting access to trusted devices, locations, networks, groups, and more.
• DDoS attack protection
• Audit logging
• And more

Drawbacks:

• Using the Grafana API becomes more complicated as requests must traverse the IAP.
• Some tools, for example the Grafana Terraform provider, cannot be used directly due to API complications, as mentioned before. Workarounds include:
  • Connecting the tool from an internal network to the internal Grafana endpoint.
  • Using a temporary Kubernetes port-forward to connect the tool to localhost. While not ideal due to the extra step required, it takes only a second.

Prerequisites

• Kubernetes cluster
• Ingress or Gateway setup
• Terraform and gcloud setup

Source Code

Find the source code here.

Configure OAuth Consent Screen

Must be done manually, but only once per GCP project:

• Open Identity-Aware Proxy and click on CONFIGURE CONSENT SCREEN.
• Choose Internal for user type.
• Fill out the required fields and add an authorized domain (e.g., example.com for grafana.example.com).

Configure Grafana

grafana.ini:
  ...
  database:
    type: postgres
    host: "<db_host>:5432"
    name: grafana
    user: grafana
    # password: "" # obtained from GF_DATABASE_PASSWORD env

  users:
    allow_sign_up: false
    allow_org_create: true
    auto_assign_org: true
    auto_assign_org_id: 1
    auto_assign_org_role: Viewer
    verify_email_enabled: false
    default_theme: dark
    viewers_can_edit: false
    editors_can_admin: false

  auth:
    disable_login_form: true

  # GCP IAP
  auth.jwt:
    enabled: true
    auto_sign_up: true
    header_name: "X-Goog-Iap-Jwt-Assertion"
    username_claim: "email"
    email_claim: "email"
    jwk_set_url: "https://www.gstatic.com/iap/verify/public_key-jwk"
    expect_claims: '{"iss": "https://cloud.google.com/iap"}'

  auth.google:
     enabled: false
  ...

Deploy Grafana with Helm

Use the Helm Terraform provider to deploy Grafana.

resource "helm_release" "grafana" {
  name             = "grafana"
  repository       = "https://grafana.github.io/helm-charts"
  chart            = "grafana"
  version          = "8.8.1" # Grafana version 11.4.0
  wait             = false
  atomic           = true
  force_update     = false
  namespace        = "monitoring"
  create_namespace = true

  values = [file("${path.module}/file/values.yaml")]

  ...
}

Expose with Gateway

The Grafana Helm chart currently offers limited support for Gateway routes. To achieve greater flexibility, we deploy an additional Gateway Helm chart. (If you prefer to use Ingress, you can configure the Grafana Helm chart accordingly and skip this step.)

resource "helm_release" "gateway" {
  name         = "grafana-gateway"
  namespace    = "monitoring"
  chart        = "${path.module}/grafana-gateway"
  wait         = false
  force_update = false
  atomic       = true

  ...
}

The GCPBackendPolicy adds GCP IAP support:

spec:
  default:
    iap:
      enabled: true
      oauth2ClientSecret:
        name: "grafana-oauth-secret"

The OAuth client is created in Terraform and then passed to the Gateway Helm chart:

resource "google_iap_client" "grafana" {
  display_name = "Grafana"
  brand        = "projects/XXXXXXXXXXXX/brands/XXXXXXXXXXXX"
}

Grant Access

When the Kubernetes resources are applied, GCP automatically creates a Backend Service in the background. To grant access, you need to attach the appropriate principals to this Backend Service.

Currently, there is no Terraform resource to handle this gracefully, as the Backend Service’s name is not known in advance (it is dynamically created by GCP in the background). However, we can use the following workaround (a similar approach exists for Ingress):

resource "google_iap_web_backend_service_iam_member" "member" {
  project = var.project
  web_backend_service = (
    reverse(
      split("/",
        [for x in split(", ",
          data.kubernetes_resource.gateway.object.metadata.annotations["networking.gke.io/backend-services"],
        ) : x if strcontains(x, "monitoring-grafana-80")][0]
      )
    )[0]
  )
  role   = "roles/iap.httpsResourceAccessor"
  member = "group:[e-mail]"
}

We leverage the annotations of the Gateway Kubernetes object to extract the dynamically created Backend Service’s name. This name is then passed to the google_iap_web_backend_service_iam_member resource to grant the roles/iap.httpsResourceAccessor role to a specified Google group.

Note that it might take a few minutes for the annotations to appear after applying the code for the first time. If you encounter an error during this process, wait until the Backend Service is visible in the GCP dashboard (might take a minute), then reapply.

Conclusion

Finally, open Grafana in your browser, where you should be able to log in using GCP IAP. If access doesn’t work immediately, wait a few minutes for the IAP permissions to propagate before starting any debugging.

For admin access, you can temporarily disable the Gateway route, enable login form (disable_login_form), and use port-forwarding to log in with admin credentials from localhost. Once logged in, grant admin privileges to the account created via GCP IAP. After completing this setup, revert the changes to restore the Gateway route and continue using Grafana securely with GCP IAP.

Need help securing your Kubernetes deployments or want to upgrade your team's skills?
Let's connect