Jump to: navigation, search

Observability

Genesys Web Services 8.6 provides deployment instructions and scripts which you can use when installing and configuring Observability based on Grafana's open source software solution.

The benefits of using this Observability Solution are:

  1. Help with root cause analysis.
  2. Help with resilience and reliability (MTTR, uptime, and so on).
  3. Help with faster and more informed feedback loops (think: continuous improvement).
  4. Provide centralized and searchable storage for logs and metrics for Genesys Web Services 8.6.
  5. Enable customers to directly share relevant logs and metrics with Genesys Customer Support to help with troubleshooting during incidents happening in an on-premise environment.

The Observability Solution is founded upon Grafana’s Observability Stack (open source) which consists of the following components:

  • Grafana - analytics and visualization
  • Mimir - storage for metrics
  • Loki - log aggregation backend
  • Tempo – tracing backend
  • Grafana Agent - vendor-neutral telemetry collector

Architecture

The Observability Solution architecture consists of:

  • OBS Core Stack (Grafana, Loki, Mimir, Tempo) running on a single designated instance.
  • Grafana Agents configured on instances where Genesys’ software is running and sends data to the OBS Core Stack.
  • On-demand export from OBS Core Stack to Genesys (the nature of "On-demand" is completely controlled by customer).
Observability Solution Architecture

Observability platform installation

Important
Genesys has provided deployment instructions and scripts which install the Grafana platform with Genesys Web Services 8.6. However, Genesys cannot provide support for any issues encountered with the Grafana platform.
Important
In order to ensure you are using the latest instructions and to gain access to the RPM package referenced below, refer to the Observability installation package included in Genesys Web Services 8.6.

OBS_instance/README.md

Installation procedure

Complete the following steps to install Grafana.

  1. Get the repository keys.
    wget -q -O gpg.key https://rpm.grafana.com/gpg.key
    sudo rpm --import gpg.key
  2. Create /etc/yum.repos.d/grafana.repo with the following parameters.
    [grafana]
    name=grafana
    baseurl=https://rpm.grafana.com
    repo_gpgcheck=1
    enabled=1
    gpgcheck=1
    gpgkey=https://rpm.grafana.com/gpg.key
    sslverify=1
    sslcacert=/etc/pki/tls/certs/ca-bundle.crt
  3. Install packages.
    sudo dnf install grafana-enterprise loki mimir tempo
  4. Update the loki configuration file (/etc/loki/config.yml) with the following parameters:
    # /etc/loki/config.yml
    auth_enabled: false
    server:
      http_listen_port: 3100
      grpc_listen_port: 9096
      grpc_server_max_recv_msg_size: 10000000
      grpc_server_max_send_msg_size: 10000000
    common:
      instance_addr: 127.0.0.1
      path_prefix: /tmp/loki
      storage:
        filesystem:
          chunks_directory: /tmp/loki/chunks
          rules_directory: /tmp/loki/rules
      replication_factor: 1
      ring:
        kvstore:
          store: inmemory
    query_range:
      results_cache:
        cache:
          embedded_cache:
            enabled: true
            max_size_mb: 100
    schema_config:
      configs:
        - from: 2022-11-30
          store: tsdb
          object_store: filesystem
          schema: v12
          index:
            prefix: index_
            period: 24h
    
    ruler:
      alertmanager_url: http://localhost:9093
    #### custom part
    compactor:
      working_directory: /tmp/loki/retention
      shared_store: filesystem
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2m
      retention_delete_worker_count: 150
    limits_config:
      retention_period: 7d
      ingestion_rate_mb: 7
      ingestion_burst_size_mb: 20
  5. Update the Mimir configuration file (/etc/mimir/config.yml) with the following parameters:
    # /etc/mimir/config.yml
    
    multitenancy_enabled: false
    target: all,alertmanager,overrides-exporter
    server:
      http_listen_port: 9009
      # Configure the server to allow messages up to 100MB.
      grpc_server_max_recv_msg_size: 104857600
      grpc_server_max_send_msg_size: 104857600
      grpc_server_max_concurrent_streams: 1000
    ingester:
      ring:
        # The address to advertise for this ingester.  Will be autodiscovered by
        # looking up address on eth0 or en0; can be specified if this fails.
        instance_addr: 127.0.0.1
        # We want to start immediately and flush on shutdown.
        min_ready_duration: 0s
        final_sleep: 0s
        num_tokens: 512
        # Use an in memory ring store, so we don't need to launch a Consul.
        kvstore:
          store: memberlist
        replication_factor: 1
    blocks_storage:
      tsdb:
        dir: /tmp/mimir/tsdb
      filesystem:
        dir: ./data/tsdb
    compactor:
      data_dir: /tmp/mimir/compactor
    ruler_storage:
      backend: local
      local:
        directory: /tmp/mimir/rules
    ##### custom part
    limits:
      compactor_blocks_retention_period: 2w
      max_global_series_per_user: 0
      ingestion_rate: 1000000
      ingestion_burst_size: 20000000
    memberlist:
      join_members: [ localhost ]
  6. Update the Tempo configuration file (/etc/tempo/config.yml) with the following parameters.
    #/etc/tempo/config.yml
    server:
      http_listen_port: 3200
      grpc_listen_port: 9097
    query_frontend:
      search:
        duration_slo: 5s
        throughput_bytes_slo: 1.073741824e+09
      trace_by_id:
        duration_slo: 5s
    distributor:
      receivers:
        jaeger:
          protocols:
            thrift_http:
            grpc:
            thrift_binary:
            thrift_compact:
        zipkin:
        otlp:
          protocols:
            http:
            grpc:
        opencensus:
    
    ingester:
      max_block_duration: 5m
      #enable_inet6: false
    
    compactor:
      compaction:
        block_retention: 24h
      #enable_inet6: false
    
    metrics_generator:
      registry:
        external_labels:
          source: tempo
          cluster: docker-compose
        #enable_inet6: false
      storage:
        path: /tmp/tempo/generator/wal
        remote_write:
          - url: http://localhost:9009/api/v1/write
            send_exemplars: true
    storage:
      trace:
        backend: local
        wal:
          path: /tmp/tempo/wal
        local:
          path: /tmp/tempo/blocks
    overrides:
      defaults:
        global:
          max_bytes_per_trace: 10000000
        metrics_generator:
          processors: [service-graphs, span-metrics] # enables metrics generator
  7. Create Grafana's default data source file (/etc/grafana/provisioning/datasources/grafana-provisioning-datasources.yaml) with the following parameters:
    # goes to /etc/grafana/provisioning/datasources/grafana-provisioning-datasources.yaml
    apiVersion: 1
    datasources:
      - name: Mimir
        uid: mimir
        type: prometheus
        access: proxy
        orgId: 1
        url: http://localhost:9009/prometheus
        version: 1
        editable: true
        jsonData:
          httpHeaderName1: 'X-Scope-OrgID'
          alertmanagerUid: 'alertmanager'
        secureJsonData:
          httpHeaderValue1: 'demo'
        isDefault: true
      - name: Mimir Alertmanager
        uid: alertmanager
        type: alertmanager
        access: proxy
        orgId: 1
        url: http://localhost:9009/
        version: 1
        editable: true
        jsonData:
          httpHeaderName1: 'X-Scope-OrgID'
          implementation: 'cortex'
        secureJsonData:
          httpHeaderValue1: 'demo'
      - name: Loki
        uid: loki
        editable: true
        type: loki
        access: proxy
        url: http://localhost:3100
        jsonData:
          derivedFields:
            - datasourceUid: tempo
              matcherRegex: "traceId\":\"(\\w+)\""
              name: traceId
              url: '$${__value.raw}'
              urlDisplayLabel: ''
            - datasourceUid: tempo
              matcherRegex: "trace_id\":\"(\\w+)\""
              name: trace_id
              url: '$${__value.raw}'
              urlDisplayLabel: ''
          timeout: 125
      - name: Tempo
        uid: tempo
        editable: true
        type: tempo
        access: proxy
        url: http://localhost:3200
  8. Update the file /etc/grafana/provisioning/dashboards/sample.yaml with the following parameters:
    # /etc/grafana/provisioning/dashboards/sample.yaml
    apiVersion: 1
    providers:
     - name: 'default'
       orgId: 1
       folder: ''
       folderUid: ''
       type: file
       options:
         path: /var/lib/grafana/dashboards
  9. Create a folder /var/lib/grafana/dashboards.
  10. Copy attached dashboards files from the path: OBS_instance/dashboards to /var/lib/grafana/dashboards.
  11. Copy attached recording rules to /tmp/mimir/rules/anonymous.
  12. Enable and start services.
    systemctl enable loki
    systemctl start loki
    systemctl enable mimir
    systemctl start mimir
    systemctl enable tempo
    systemctl start tempo
    systemctl enable grafana-server
    systemctl start grafana-server
  13. Check the statuses of the newly deployed services.
    systemctl status loki
    systemctl status mimir
    systemctl status tempo
    systemctl status grafana-server
  14. Check the accessibility of Grafana UI using the following URL and update the default password:
    https://your_host_ip:3000/

Grafana Agent installation

The following steps must be repeated for any host which has Genesys Web Services 8.6 deployed. grafana_agent_static/README.md

  1. Copy the .tar.gz package to the nodes where the agent needs to be installed.
  2. Untar the archive.
  3. From the grafana_agent_static/config folder, run the agent_install.sh script with the following two arguments.
    1. the hostname of Loki/Mimir/Grafana instance
    2. the path to folder with GWS logs location, that is, /logs:
      ls -l /logs
      total 16
      drwxrwxr-x 2 genesys genesys 16384 Mar  5 14:20 gws-api-v2
      drwxrwxr-x 2 genesys genesys    38 Mar  5 11:28 gws-service-platform
  4. Script should be executed with root permissions, that is, sudo ./agent_install.sh <OBS_HOSTNAME_OR_IP> <LOG_FOLDER>

For example, sudo ./agent_install.sh d-pat-u1040.us.int.genesyslab.com /mnt/logs/dir_with_logs_from_gws_components/

This page was last edited on July 23, 2024, at 18:26.
Comments or questions about this documentation? Contact us for support!