Monitoring

Monitoring IBM Verify Identity Access OIDC Provider

IBM Verify Identity Access OIDC provider supports Instana, Dynatrace, and Prometheus to monitor the performance of some application runtime components.

Instana

  • CPU
  • Memory
  • Response time for individual HTTP calls

Dynatrace

  • CPU
  • Memory
  • Response time for individual HTTP calls
  • Detailed SQL queries for each HTTP call using OpenTelemetry libraries for PostgreSQL database

Prometheus

  • http_requests_total
  • response_status
  • node_memory_usage_bytes
  • http_response_time_seconds
  • Go lang based metrics

Configuration for Instana

  • The Instana agent can be installed on a Kubernetes platform by using the instructions provided here
  • A helm chart example
helm install instana-agent \
   --repo https://agents.instana.io/helm \
   --namespace instana-agent \
   --create-namespace \
   --set agent.key=QHAvLwgRSH11111zGGGnTA \
   --set agent.downloadKey=QHAvLwgRSH11111zGGGnTA \
   --set agent.endpointHost=ingress-test.instana.io \
   --set agent.endpointPort=443 \
   --set cluster.name='IVIAOP' \
   --set zone.name='jp-tok' \
   instana-agent
  • To configure sending metrics to an Instana agent, some environmental variables must be set at the IVIAOP deployment.
  • A Kubernetes environment example
env:
  - name: INSTANA_ENDPOINT_URL
    value: https://<INSTANA_ENDPOINT_URL>
  - name: INSTANA_AGENT_KEY
    value: uBp4GXpZQp11111XNcvInQ
  - name: INSTANA_AGENT_HOST
    value: 10.67.92.81
  - name: INSTANA_ENABLEMENT
    value: 'true'
Environment VariableDescription
INSTANA_ENDPOINT_URLInstana endpoint URL, it can be retrieved from the Instana tenant
INSTANA_AGENT_KEYInstana agent key which can be retrieved from the tenant
INSTANA_AGENT_HOSTThe IP address on the Instana agent running in the same Kubernetes cluster
INSTANA_ENABLEMENTBoolean flag that control pushing metric to the Instana agent
  • After the connection is successful, the Kubernetes cluster and the HTTP calls can be monitored in the Instana dashboard.

Configuration for Dynatrace

  • The Dynatrace One Agent can be installed on a Kubernetes platform by using the instructions provided here.
  • A command example
$kubectl create namespace dynatrace
$kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v1.0.0/kubernetes.yaml
$kubectl apply -f csi.yaml 
$kubectl -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300s
$kubectl -n dynatrace create secret generic dynakube --from-literal="apiToken=<apiToken>" --from-literal="dataIngestToken=<dataIngestToken>"
$kubectl apply -f cloudnative.yaml
  • Example of csi.yaml
---
# Source: dynatrace-operator/templates/Common/csi/priority-class.yaml
# Copyright 2021 Dynatrace LLC

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:
  name: dynatrace-high-priority
value: 1000000
globalDefault: false
description: "This priority class is used for Dynatrace Components in order to make sure they are not evicted in favor of other pods"
---
# Source: dynatrace-operator/templates/Common/csi/serviceaccount-csi.yaml
# Copyright 2021 Dynatrace LLC

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dynatrace-oneagent-csi-driver
  namespace: dynatrace
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
---
# Source: dynatrace-operator/templates/Common/csi/clusterrole-csi.yaml
# Copyright 2021 Dynatrace LLC

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dynatrace-oneagent-csi-driver
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - list
      - watch
      - create
      - update
      - patch
  - apiGroups:
      - storage.k8s.io
    resources:
      - csinodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
---
# Source: dynatrace-operator/templates/Common/csi/clusterrole-csi.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dynatrace-oneagent-csi-driver
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
subjects:
  - kind: ServiceAccount
    name: dynatrace-oneagent-csi-driver
    namespace: dynatrace
roleRef:
  kind: ClusterRole
  name: dynatrace-oneagent-csi-driver
  apiGroup: rbac.authorization.k8s.io
---
# Source: dynatrace-operator/templates/Common/csi/role-csi.yaml
# Copyright 2021 Dynatrace LLC

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dynatrace-oneagent-csi-driver
  namespace: dynatrace
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
rules:
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - watch
      - list
      - delete
      - update
      - create
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - watch
      - list
      - delete
      - update
      - create
  - apiGroups:
      - dynatrace.com
    resources:
      - dynakubes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - list
      - watch
---
# Source: dynatrace-operator/templates/Common/csi/role-csi.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dynatrace-oneagent-csi-driver
  namespace: dynatrace
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
subjects:
  - kind: ServiceAccount
    name: dynatrace-oneagent-csi-driver
    namespace: dynatrace
roleRef:
  kind: Role
  name: dynatrace-oneagent-csi-driver
  apiGroup: rbac.authorization.k8s.io
---
# Source: dynatrace-operator/templates/Common/csi/daemonset.yaml
# Copyright 2021 Dynatrace LLC

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
  name: dynatrace-oneagent-csi-driver
  namespace: dynatrace
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      internal.oneagent.dynatrace.com/app: csi-driver
      internal.oneagent.dynatrace.com/component: csi-driver
  template:
    metadata:
      annotations:
        dynatrace.com/inject: "false"
        kubectl.kubernetes.io/default-container: provisioner
        cluster-autoscaler.kubernetes.io/enable-ds-eviction: "false"
      labels:
        app.kubernetes.io/name: dynatrace-operator
        app.kubernetes.io/version: "1.0.0"
        app.kubernetes.io/component: csi-driver
        internal.oneagent.dynatrace.com/app: csi-driver
        internal.oneagent.dynatrace.com/component: csi-driver
    spec:
      initContainers:
      - name: csi-init
        image: public.ecr.aws/dynatrace/dynatrace-operator:v1.0.0@sha256:f742a1856f61ad127e9130e44e20dfa4d9990d1d472cbd5ce815b3599205e2e3
        imagePullPolicy: Always
        args:
        - csi-init
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: false
          runAsUser: 0
          seLinuxOptions:
            level: s0
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /data
          name: data-dir
      containers:
        # Used to receive/execute gRPC requests (NodePublishVolume/NodeUnpublishVolume) from kubelet to mount/unmount volumes for a pod
        # - Needs access to the csi socket, needs to read/write to it, needs root permissions to do so.
        # - Needs access to the filesystem of pods on the node, and mount stuff to it,needs to read/write to it, needs root permissions to do so
        # - Needs access to a dedicated folder on the node to persist data, needs to read/write to it.
      - name: server
        image: public.ecr.aws/dynatrace/dynatrace-operator:v1.0.0@sha256:f742a1856f61ad127e9130e44e20dfa4d9990d1d472cbd5ce815b3599205e2e3
        imagePullPolicy: Always
        args:
        - csi-server
        - --endpoint=unix://csi/csi.sock
        - --node-id=$(KUBE_NODE_NAME)
        - --health-probe-bind-address=:10080
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: livez
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        ports:
        - containerPort: 10080
          name: livez
          protocol: TCP
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
          readOnlyRootFilesystem: true
          runAsNonRoot: false
          runAsUser: 0
          seLinuxOptions:
            level: s0
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /var/data/kubelet/pods/
          mountPropagation: Bidirectional
          name: mountpoint-dir
        - mountPath: /data
          name: data-dir
          mountPropagation: Bidirectional
        - name: tmp-dir
          mountPath: /tmp
      - name: provisioner
        image: public.ecr.aws/dynatrace/dynatrace-operator:v1.0.0@sha256:f742a1856f61ad127e9130e44e20dfa4d9990d1d472cbd5ce815b3599205e2e3
        imagePullPolicy: Always
        args:
          - csi-provisioner
          - --health-probe-bind-address=:10090
        env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
        startupProbe:
          exec:
            command:
            - /usr/local/bin/dynatrace-operator
            - startup-probe
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 1
        
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: livez
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        ports:
          - containerPort: 10090
            name: livez
            protocol: TCP
        resources:
          limits:
            cpu: 300m
            memory: 100Mi
          requests:
            cpu: 300m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
          readOnlyRootFilesystem: true
          runAsNonRoot: false
          runAsUser: 0
          seLinuxOptions:
            level: s0
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
          - mountPath: /data
            name: data-dir
            mountPropagation: Bidirectional
          - mountPath: /tmp
            name: tmp-dir

        # Used to make a gRPC request (GetPluginInfo()) to the driver to get driver name and driver contain
        # - Needs access to the csi socket, needs to read/write to it, needs root permissions to do so.
        # Used for registering the driver with kubelet
        # - Needs access to the registration socket, needs to read/write to it, needs root permissions to do so.
      - name: registrar
        image: public.ecr.aws/dynatrace/dynatrace-operator:v1.0.0@sha256:f742a1856f61ad127e9130e44e20dfa4d9990d1d472cbd5ce815b3599205e2e3
        imagePullPolicy: Always
        env:
        - name: DRIVER_REG_SOCK_PATH
          value: /var/data/kubelet/plugins/csi.oneagent.dynatrace.com/csi.sock
        args:
        - --csi-address=/csi/csi.sock
        - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
        command:
        - csi-node-driver-registrar
        resources:
          limits:
            cpu: 20m
            memory: 30Mi
          requests:
            cpu: 20m
            memory: 30Mi
        securityContext:
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: false
          runAsUser: 0
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /registration
          name: registration-dir
        - mountPath: /var/data/kubelet/plugins/csi.oneagent.dynatrace.com/
          name: lockfile-dir
        # Used to make a gRPC request (Probe()) to the driver to check if its running
        # - Needs access to the csi socket, needs to read/write to it, needs root permissions to do so.
      - name: liveness-probe
        image: public.ecr.aws/dynatrace/dynatrace-operator:v1.0.0@sha256:f742a1856f61ad127e9130e44e20dfa4d9990d1d472cbd5ce815b3599205e2e3
        imagePullPolicy: Always
        args:
        - --csi-address=/csi/csi.sock
        - --health-port=9898
        command:
        - livenessprobe
        resources:
          limits:
            cpu: 20m
            memory: 30Mi
          requests:
            cpu: 20m
            memory: 30Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: false
          runAsUser: 0
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccountName: dynatrace-oneagent-csi-driver
      terminationGracePeriodSeconds: 30
      priorityClassName: dynatrace-high-priority
      volumes:
      # This volume is where the registrar registers the plugin with kubelet
      - name: registration-dir
        hostPath:
          path: /var/data/kubelet/plugins_registry/
          type: Directory
        # This volume is where the socket for kubelet->driver communication is done
      - name: plugin-dir
        hostPath:
          path: /var/data/kubelet/plugins/csi.oneagent.dynatrace.com/
          type: DirectoryOrCreate
      - name: data-dir
        hostPath:
          path: /var/data/kubelet/plugins/csi.oneagent.dynatrace.com/data
          type: DirectoryOrCreate
        # This volume is where the driver mounts volumes
      - name: mountpoint-dir
        hostPath:
          path: /var/data/kubelet/pods/
          type: DirectoryOrCreate
        # Used by the registrar to create its lockfile
      - name: lockfile-dir
        emptyDir: {}
        # A volume for the driver to write temporary files to
      - name: tmp-dir
        emptyDir: {}
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - key: kubernetes.io/arch
          value: arm64
          effect: NoSchedule
        - key: kubernetes.io/arch
          value: amd64
          effect: NoSchedule
        - key: kubernetes.io/arch
          value: ppc64le
          effect: NoSchedule
        - key: ToBeDeletedByClusterAutoscaler
          operator: Exists
          effect: NoSchedule
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
---
# Source: dynatrace-operator/templates/Common/csi/csidriver.yaml
# Copyright 2021 Dynatrace LLC

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: csi.oneagent.dynatrace.com
  labels:
    app.kubernetes.io/name: dynatrace-operator
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/component: csi-driver
spec:
  attachRequired: false
  podInfoOnMount: true
  volumeLifecycleModes:
    - Ephemeral


  • Example of cloudnative.yaml, replace the TENANT with the Dynatrace tenant information.
apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
  name: dynakube
  namespace: dynatrace
spec:
  # Dynatrace apiUrl including the `/api` path at the end.
  # For SaaS, set `ENVIRONMENTID` to your environment ID.
  # For Managed, change the apiUrl address.
  # For instructions on how to determine the environment ID and how to configure the apiUrl address, see https://www.dynatrace.com/support/help/reference/dynatrace-concepts/environment-id/.
  apiUrl: https://<TENANT>/api

  # Optional: Name of the secret holding the credentials required to connect to the Dynatrace tenant
  # If unset, the name of this custom resource is used
  #
  # tokens: ""

  # Optional: Defines a custom pull secret in case you use a private registry when pulling images from the Dynatrace environment
  # The secret has to be of type 'kubernetes.io/dockerconfigjson' (see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/)
  #
  # customPullSecret: "custom-pull-secret"

  # Optional: Disable certificate validation checks for installer download and API communication
  #
  # skipCertCheck: false

  # Optional: Set custom proxy settings either directly or from a secret with the field 'proxy'
  #
  # proxy:
  #   value: my-proxy-url.com
  #   valueFrom: name-of-my-proxy-secret

  # Optional: Adds custom RootCAs from a configmap
  # The key to the data must be "certs"
  # This property only affects certificates used to communicate with the Dynatrace API.
  # The property is not applied to the ActiveGate
  #
  # trustedCAs: name-of-my-ca-configmap

  # Optional: Sets Network Zone for OneAgent and ActiveGate pods
  # Make sure networkZones are enabled on your cluster before (see https://www.dynatrace.com/support/help/setup-and-configuration/network-zones/network-zones-basic-info/)
  #
  # networkZone: name-of-my-network-zone

  # Optional: If enabled, and if Istio is installed on the Kubernetes environment, the
  # Operator will create the corresponding VirtualService and ServiceEntry objects to allow access
  # to the Dynatrace cluster from agents or activeGates. Disabled by default.
  #
  # enableIstio: false

  # The namespaces which should be injected into
  # If unset, all namespace will be injected into
  # namespaceSelector has no effect on hostMonitoring or classicFullstack
  # For examples regarding namespaceSelectors, see https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements
  #
  # namespaceSelector:
  #   matchLabels:
  #     app: my-app
  #   matchExpressions:
  #    - key: app
  #      operator: In
  #      values: [my-frontend, my-backend, my-database]

  # Configuration for OneAgent instances
  #
  oneAgent:
    # Enables cloud-native fullstack monitoring and changes its settings
    # Cannot be used in conjunction with classic fullstack monitoring, application-only monitoring or host monitoring
    #
    cloudNativeFullStack:
      # Optional: Sets a node selector to control on which nodes the OneAgent will be deployed.
      # For more information on node selectors, see https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/
      #
      # nodeSelector: {}

      # Optional: Sets the priority class assigned to the OneAgent Pods. No class is set by default.
      # For more information on priority classes, see https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
      #
      # priorityClassName: priority-class

      # Optional: Specifies tolerations to include with the OneAgent DaemonSet.
      # For more information on tolerations, see https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
      #
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists

      # Optional: Adds resource settings for OneAgent container
      # Consumption of the OneAgent heavily depends on the workload to monitor
      # The values should be adjusted according to the workload
      #
      # oneAgentResources:
      #   requests:
      #     cpu: 100m
      #     memory: 512Mi
      #   limits:
      #     cpu: 300m
      #     memory: 1.5Gi

      # Optional: Adds custom arguments to the OneAgent installer
      # For a list of available options, see https://www.dynatrace.com/support/help/shortlink/linux-custom-installation
      # For a list of the limitations for OneAgents in Docker, see https://www.dynatrace.com/support/help/shortlink/oneagent-docker#limitations
      #
      # args: []

      # Optional: Adds custom environment variables to OneAgent pods
      #
      # env: []

      # Optional: Enables or disables automatic updates of OneAgent pods
      # By default, if a new version is available, the OneAgent pods are restarted to apply the update
      # If set to "false", this behavior is disabled
      # Defaults to "true"
      #
      # autoUpdate: true

      # Optional: Sets the DNS Policy for OneAgent pods
      # Defaults to "ClusterFirstWithHostNet"
      # For more information on DNS policies, see https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
      #
      # dnsPolicy: "ClusterFirstWithHostNet"

      # Optional: Adds custom annotations to OneAgent pods
      #
      # annotations:
      #   custom: annotation

      # Optional: Adds custom labels to OneAgent pods
      # Can be used to structure workloads
      #
      # labels:
      #   custom: label

      # Optional: Sets the URI for the image containing the OneAgent installer used by the DaemonSet
      # Defaults to the latest OneAgent image on the tenant's registry
      #
      # image: ""

      # Optional: If specified, indicates the OneAgent version to use
      # Defaults to the configured version on your Dynatrace environment
      # The version is expected to be provided in the semver format
      # Example: {major.minor.release}, e.g., "1.200.0"
      #
      # version: ""

      # Optional: Defines resources requests and limits for the initContainer
      # See more: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers
      #
      # initResources:
      #   requests:
      #     cpu: 100m
      #     memory: 512Mi
      #   limits:
      #     cpu: 300m
      #     memory: 1.5Gi

      # Optional: The URI of the image that contains the codemodules specific OneAgent that will be injected into pods and applications.
      # For an example of a Dockerfile creating such an image, see https://dt-url.net/operator-docker-samples
      #
      # codeModulesImage: ""

  # Configuration for ActiveGate instances.
  #
  activeGate:
    # Specifies which capabilities will be enabled on ActiveGate instances
    # The following capabilities can be set:
    # - routing
    # - kubernetes-monitoring
    # - metrics-ingest
    # - dynatrace-api
    #
    capabilities:
      - routing
      - kubernetes-monitoring
      - dynatrace-api

    # Optional: Sets how many ActiveGate pods are spawned by the StatefulSet
    # Defaults to "1"
    #
    # replicas: 1

    # Optional: Sets the image used to deploy ActiveGate instances
    # Defaults to the latest ActiveGate image on the tenant's registry
    # Example: "ENVIRONMENTID.live.dynatrace.com/linux/activegate:latest"
    #
    # image: ""

    # Recommended: Sets the activation group for ActiveGate instances
    #
    # group: ""

    # Optional: Defines a custom properties file, the file contents can be provided either as a value in this yaml or as a reference to a secret.
    # If a reference to a secret is used, then the file contents must be stored under the 'customProperties' key within the secret.
    #
    # customProperties:
    #   value: |
    #     [connectivity]
    #     networkZone=
    #   valueFrom: myCustomPropertiesConfigMap

    # Optional: Specifies resource settings for ActiveGate instances
    # Consumption of the ActiveGate heavily depends on the workload to monitor
    # The values should be adjusted according to the workload
    #
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1.5Gi

    # Optional: Sets a node selector to control on which nodes the ActiveGate will be deployed.
    # For more information on node selectors, see https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/
    #
    # nodeSelector: {}

    # Optional: Specifies tolerations to include with the ActiveGate StatefulSet.
    # For more information on tolerations, see https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
    #
    # tolerations:
    # - effect: NoSchedule
    #   key: node-role.kubernetes.io/master
    #   operator: Exists

    # Optional: Adds custom labels to ActiveGate pods
    # Can be used to structure workloads
    #
    # labels:
    #   custom: label

    # Optional: Adds custom environment variables to ActiveGate pods
    #
    # env: []

    # Optional: Specifies the name of a secret containing a TLS certificate, a TLS key and the TLS key's password to be used by ActiveGate instances
    # If unset, a self-signed certificate is used
    # The secret is expected to have the following key-value pairs
    # server.p12: TLS certificate and TLS key pair in pkcs12 format
    # password: passphrase to decrypt the TLS certificate and TLS key pair
    #
    # tlsSecretName: "my-tls-secret"

    # Optional: Sets the DNS Policy for ActiveGate pods
    # Defaults to "Default"
    # For more information on DNS policies, see https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
    #
    # dnsPolicy: "Default"

    # Optional: Specifies the priority class to assign to the ActiveGate Pods
    # No class is set by default
    # For more information on priority classes, see https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
    #
    # priorityClassName: priority-class

    # Optional: Adds custom annotations to ActiveGate pods
    #
    # annotations:
    #   custom: annotation

    # Optional: Adds TopologySpreadConstraints to the ActiveGate pods
    # For more information on TopologySpreadConstraints, see https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
    #
    # topologySpreadConstraints: []
  • Access tokens can be generated with the right scopes by using this documentation link.
  • Enable OpenTelemetry for Go by using the following link
  • Enable Attribute Capturing by using the following link
  • After the previous steps are configured successfully, all the HTTP calls and the respective DB queries are logged in Dynatrace Distributed Traces.

📘

Note

Instrumentation is added only for the PostgreSQL database library.

Configuration for Prometheus

  • To enable /metrics endpoint, the following environmental variable must be set. Example https://:/metrics.
  • A Kubernetes environment example
env:
  - name: PROMETHEUS_ENABLED
    value: 'true'
  • The following counters, gauges and historgram will be exposed
    • http_requests_total
    • response_status
    • node_memory_usage_bytes
    • http_response_time_seconds
    • Go lang based metrics
  • In Kubernetes a prometheus operator can be deployed in a different namespace where the IVIAOP is deployed.
  • Example deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config-volume
              mountPath: /etc/prometheus
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-server-conf
            defaultMode: 420

  • Example service.yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: prometheus
spec:
  selector:
    app: prometheus-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9090
  type: LoadBalancer

  • Example of the promethues configuration file
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  namespace: prometheus
data:
  prometheus.yml: |
    global:
     scrape_interval:     20s
     evaluation_interval: 10s

    scrape_configs:
      - job_name: iviaop 
        metrics_path: /metrics
        static_configs:
          - targets:
              - <IP>:<IVIAOP Port>

  • To understand more about Kubernetes SD Configuration, (see )[https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config].