Alertmanager-evateam: integration with EvaTeam

Integrating Alertmanager with EvaTeam: The alertmanager-evateam Plugin for Task Automation

The alertmanager-evateam plugin ensures direct integration between Alertmanager from Prometheus or VictoriaMetrics and the EvaTeam tracker. When an alert fires, a task is automatically created in the EvaTeam project. When the alert resolves, a comment is added or the status is updated. The solution has been deployed in production for over six months, demonstrating stability without failures.

Implemented in Java using the Quarkus framework and native GraalVM compilation, it minimizes resource consumption and accelerates startup.

Integration Workflow

The alert processing scheme includes the following stages:

• Collection of Prometheus or VictoriaMetrics metrics.
• Alertmanager evaluates rules, generates an alert, and sends it via webhook_config to the plugin endpoint.
• The plugin parses the alert JSON, creates a new task in EvaTeam, or updates an existing one.
• If send_resolved: true, a resolution comment is added or the task status changes.

The key mechanism is that task parameters are defined via the eva__ prefix in the alert labels and annotations. This allows customizing behavior at the level of each alerting rule without changes to the plugin configuration.

Deployment via Containers

Images are available on Docker Hub. Quick test:

podman run -it --rm --name alertmanager-evateam \
    -p 8080:8080 \
    hubbitus/alertmanager-evateam:latest

For development, a prometheus + alertmanager + grafana stack is provided in the _DEV.scripts/prometheus-alertmanager-grafana directory with podman-compose or docker-compose.

Configuration in Alertmanager

Add a receiver:

receivers:
  - name: 'data-evateam'
    webhook_configs:
      - url: 'https://alertmanager-evateam.your.domain'
        send_resolved: true

For common values, use alert_relabel_configs:

alerting:
  alert_relabel_configs:
    - target_label: eva__project
      replacement: data-alerts
    - target_label: eva__field__issue_type_name
      replacement: Task

Key Control Fields

• eva__project: EvaTeam project key (e.g., data-alerts).
• eva__field__issue_type_name: Task type (Task, Incident).
• eva__field__<field>: Arbitrary fields (assignee, priority, cf_env).
• eva__identification_field_name: Field for searching existing tasks (default cf_alert_id).
• eva__identification_field_value: Identifier template (${context.alert.hashCode()}).
• eva__bql_to_find_issue_for_update: BQL query for updating the task.
• eva__comment_in_present_issues: Comment template.

Example alert rule:

- name: DATA
  rules:
    - alert: DataTest0
      expr: 'promhttp_metric_handler_requests_total > 1'
      labels:
        severity: warning
        eva__field__severity: High
      annotations:
        eva__project: data-alerts
        eva__field__issue_type_name: Task
        eva__field__cf_env: PROD
        summary: DataTest0 summary
        description: |
          Some description VALUE: {{$value}}

Handling Fields with Special Characters and Templating

For field names with spaces or slashes, apply pairs:

annotations:
  eva__field__name__1: 'Component/s'
  eva__field__value__1: 'DQ-issues+alerts, DevOps+infrastructure'
  eva__field__name__result: 'Final Result'
  eva__field__value__result: 'Some result description'

Templates use context: eva__field__tags: '["severity:${context.field("severity")}"]'. Full list of variables in AlertContext.

EvaTeam API Specifics

The EvaTeam API is flexible: it accepts kwargs for any fields without a strict schema. However, validation is weak — typos in field names are ignored without errors. Mandatory testing: verify task creation with target fields before production.

Building and Testing from Source

Build:

./gradlew build
java -jar build/quarkus-app/quarkus-run.jar

Native build:

./gradlew build -Dquarkus.native.enabled=true \
    -Dquarkus.package.jar.enabled=false \
    -Dquarkus.native.container-build=true

Tests (unit + integration, require live EvaTeam):

./gradlew test --info
./gradlew testNative --info

Key Takeaways

• Automates the full cycle: from Alertmanager alert to EvaTeam task with updates on resolution.
• Flexible configuration via eva__ prefixes in labels/annotations without plugin edits.
• GraalVM native build for low resource consumption.
• Requires thorough testing due to weak EvaTeam API validation.
• Suitable for production: stable operation for six months.

— Editorial Team