SAP BTP Kyma Metrics and Log export

Need

Everyone of you would love to see logs from your application deployed on Kubernetes nice way. We all used to luxury having Kibana, Splunk, you name it. Unfortunately with SAP Kyma it is not available anymore. You need set it up by yourself to see them collected in one place.

In the past you could have loki exposed and then look at it using your Grafana instance locally. Unfortunately that part also gone.

https://blogs.sap.com/2022/11/02/deprecation-of-loki-based-application-logs-in-sap-btp-kyma-runtime/

https://blogs.sap.com/2023/06/02/removal-of-loki-based-application-logs-in-sap-btp-kyma-runtime/

You can still navigate to Pod's logs on Kyma Dashboard or use kubectl from command line, but... that is not what we expect use in the long run.

Kyma offers Log Pipeline which I have tried to use, but for some reason I got an error with Grafana... If you would like to go through their docs, you can find that on Kyma: Application Logs page.

Solution

I have used free Grafana account to set it up their way. It was blazing fast. Start from user account page:

Alternatively you can find on the bottom of that page a shortcut:

1. find on home page "+ Connect Data" (https://<user-account>.grafana.net/connections/add-new-connection).
2. select "Kubernetes monitoring" and create token
3. follow their helm installation

Code snippet, which has been generated for my account looked like this:

helm repo add grafana https://grafana.github.io/helm-charts &&
helm repo update &&
helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring --namespace "default" --create-namespace --values - <<EOF
cluster:
  name: services
externalServices:
  prometheus:
    host: https://prometheus-prod-13-prod-us-east-0.grafana.net
    basicAuth:
      username: "1244838"
      password: PASTE_AUTH_TOKEN
  loki:
    host: https://logs-prod-006.grafana.net
    basicAuth:
      username: "721487"
      password: PASTE_AUTH_TOKEN
opencost:
  opencost:
    exporter:
      defaultClusterId: services
    prometheus:
      external:
        url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom
EOF

When I run it on Kyma I got one error due to existing pods installed by default. Helm printed out solution for it in the console. I needed to add at the end one configuration entry:

prometheus-operator-crds:
  enabled: false

After that I got a notification in console:

Release "grafana-k8s-monitoring" does not exist. Installing it now.
NAME: grafana-k8s-monitoring
LAST DEPLOYED: Thu Nov  2 15:23:43 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1

and finally I could browse logs.