Install and Configure Alertmanager

14.3 Install and Configure Alertmanager

Prometheus scrapes data from the MicroTx coordinator at regular intervals. Prometheus Alertmanager sends notifications based on the specified thresholds and rules. If you have configured an email ID to receive notifications, Alertmanager sends notifications to the configured email ID.

Alerts are sent based on the thresholds or rules mentioned in the prometheus.yaml file. For example, users are notified when the MicroTx service or database service is not available. You can customize the alerts. To add a custom alert, see https://prometheus.io/docs/alerting/latest/notifications/.

Install Prometheus. See Set Up Prometheus and Grafana.
Open the prometheus.yaml file.
In Kubernetes environments, use the prometheus.yaml which is located at installation_directory/otmm-RELEASE/otmm/visualization/alert/kubernetes. The MicroTx coordinator uses the configuration details provided in the prometheus.yaml to monitor and manage alerts. This file also defines the rules to generate alerts.

This file contains details about the instance from which Prometheus scrapes the metrics and it defines the threshold for which alert messages are sent. For description about each property and the permitted values, see https://prometheus.io/docs/prometheus/latest/configuration/configuration/.

To install Prometheus in your local environment, use the prometheus.yaml and rules.yaml files which are located at installation_directory/otmm-RELEASE/otmm/visualization/alert/local. The MicroTx coordinator uses the configuration details provided in the prometheus.yaml to monitor and manage alerts. The rules.yaml file defines the rules to generate alerts.

Provide values for the following properties.
- Enter the scrape_interval and evaluation_interval to specify the time interval at which Prometheus scrapes the metrics from the coordinator.
```
server:
    global:
        scrape_interval: 1m
        evaluation_interval: 1m
```
- Enter a unique value for job_name and specify the instance that you want to connect to. Prometheus scrapes the metrics logs from the specified instance.
```
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
        - localhost:9090
   - job_name: "tcs"
     static_configs:
       - targets: 
         - otmm-tcs.otmm.svc.cluster.local:9000
   - job_name: kube-state-metrics
     metrics_path: /metrics
     static_configs:
       - targets:
         - prometheus-kube-state-metrics.otmm.svc.cluster.local:8080
```
  Where,
  - scrape_config is the name of the job in Prometheus.
  - otmm-tcs.otmm.svc.cluster.local:9000 is the name of the MicroTx coordinator service and the port number at which the service is running. Alert messages are generated for this service.
  - prometheus-kube-state-metrics.otmm.svc.cluster.local:8080 is the target from which you want to scrape the metrics. Prometheus exposes a set of metrics by default. When you install Prometheus in a cluster, a set of services are deployed which includesprometheus-kube-state-metrics. Use this service to query the state, health, or performance of any Kubernetes resource.

Run the following commands to download the Helm Charts from the prometheus-community repo.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Reinstall Prometheus using the prometheus.yaml file that you have downloaded.
```
helm install -f prometheus.yaml prometheus prometheus-community/prometheus
```
Download the Prometheus Alertmanager from https://prometheus.io/download/#alertmanager.
Run the following command to install Prometheus Alertmanager.
```
helm install [RELEASE_NAME] prometheus-community/alertmanager
```
Where, [RELEASE_NAME] is the release number of Prometheus Alertmanager that you want to install.

You can configure to receive alerts on Slack or email. See the Prometheus Alertmanager documentation.

If you have installed Prometheus and Alertmanager in your local envirnonment, you can access Prometheus at http://localhost:9090/ and Alertmanager at http://localhost:9093/.

Parent topic: Monitor and Trace Transactions