Commit graph

66 commits

Author SHA1 Message Date
Mateusz Matuszkowiak
53234ab0e5 Use max instead of min
Since these services are replicated (two per service),
we should check for the max, not min, as one of the replicas
could be down, but still the API is Up.

Change-Id: I6dad9b74d65415f5f4f91e00c732792010487619
2018-03-20 14:12:30 +01:00
Dmitry Kalashnik
be606a43c4 Allow to set custom template for email receivers
Change-Id: I9fb7fbd1946e564e1e245ea226c5bbe7440f25d4
Closes-Bug: PROD-18240
2018-03-05 17:34:51 +04:00
Dmitry Kalashnik
cdad46d5ad Allow to use multimatching for alermanager routes
Change-Id: I2c0d439055efc0d218c8f247aea264d425fca838
2018-03-01 16:35:41 +04:00
Bartosz Kupidura
8bdf3ed090 Add support for prometheus 2.0
New version changes:
* different alerts format
* rewritten storage (some config flags removed)

Closes-Bug: PROD-16609
Change-Id: I805fa322e4744e98177d6c3e29589ebc6fb917a2
2018-01-03 12:26:10 +01:00
Ondrej Smola
94baab3660 iadded posibility to define two endpoint in one receiver + test pillars + update Readme
This option is usefull when all alarms should be send to two or more receivers simultaneously - without need to use continue parameter

Change-Id: I6b8d833776549614d0381378a2e92cb5bc05af77
2017-11-21 10:18:34 +01:00
mcp-jenkins
2f5ed47dc9 Merge "Add prometheus alerts" 2017-09-28 11:26:00 +00:00
Bartosz Kupidura
2f8a10309e Merge "Add pushgateway and alertmanager dashboards" 2017-09-28 11:24:32 +00:00
Bartosz Kupidura
6fce6098d7 Add prometheus alerts
* PrometheusRushMode
* PrometheusRemoteStorageQueue
* AlertmanagerNotificationFailed

Change-Id: I5a875e7b9861f860bac501da55f0e8b20e799d52
2017-09-27 16:48:33 +02:00
Bartosz Kupidura
20a437a58e Add pushgateway and alertmanager dashboards
Change-Id: I83d705833f73bd827f5b752b215d3be53dc278dd
2017-09-27 16:20:06 +02:00
Ildar Svetlov
5932165d11 Alignment Remote Storage Adapter Grafana dashboard
Change-Id: Ic61a852118eb932c837689d1eafbeba313adeb2d
Closes-bug: PROD-14712
2017-09-25 14:11:12 +04:00
Bartosz Kupidura
37925ad206 Get kubernetes status from grains
Change-Id: If921742f5f2601bcd974922d272f0d8f4aa2b0b0
2017-09-20 12:17:23 +02:00
Ales Komarek
22f7ab3572 Merge "Trigger the target down alert after 2 minutes" 2017-09-15 09:51:58 +00:00
Konstantin Hontar
7b14deaf31 Add Slack and mail receiver to AlertManager config
Change-Id: Ib82ef0a7d029f84a7f92a90384bac02a0870b521
2017-09-14 13:24:04 +02:00
Simon Pasquier
cd90c9f842 Trigger the target down alert after 2 minutes
Otherwise the alert fires as soon as Prometheus can't scrape a target.
It is too aggressive in case of transient connectivity issues or
endpoint restart.

Change-Id: Ib3de5b141db7a7f2397bf332844a9c44d38f2d3c
2017-09-12 15:14:21 +02:00
Olivier Bourdon
477ea05109 Fix for removing local addresses as exporters addresses
Change-Id: Ifac8c8960497cb6dfe95e493583f07cee4d0833d
2017-09-01 11:00:50 +02:00
Ildar Svetlov
1aaa416bdc Add Prometheus Stats dashboard
Change-Id: Icad3d5bee52546288f178ab980f60d8355a36a13
2017-08-30 12:34:54 +00:00
Bartosz Kupidura
dc36045033 Allow to pass metrics_path for static jobs
Change-Id: Ifa4fa71241bf8d2c7e0a4960192d3499252516aa
2017-08-21 11:32:02 +02:00
Bartosz Kupidura
d949dcef88 Revert "Add Jenkins to Prometheus targets"
This reverts commit db1f492198.

Change-Id: I4db9c61f1061021e02e2e28ce6132449a7cb5586
2017-08-21 09:18:58 +00:00
Konstantin Hontar
db1f492198 Add Jenkins to Prometheus targets
Change-Id: Ie9853b9860b34536ded4a7583c48c735e75bb7b7
2017-08-18 13:12:03 +03:00
Bartosz Kupidura
94c1a3c99a Pass all options for static target from formula
Change-Id: I21c84341d5a585862d21f75a4d263f8f5781675d
2017-08-04 18:33:56 +02:00
Ales Komarek
dd19c7f616 Merge changes from topics 'prometheus/alert', 'rename-prometheus-alerts'
* changes:
  Add Prometheus alerts
  Rename Prometheus alerts for consistency
2017-07-26 12:21:51 +00:00
Olivier Bourdon
5b6b583c42 Add Prometheus alerts
Change-Id: I4ad10555d728d62c8e6504659d30558f95b410ac
2017-07-26 13:45:51 +02:00
mcp-jenkins
90b074f356 Merge "Add Grafana dashboard with Prometheus datasource for remote storage adapter" 2017-07-25 15:21:03 +00:00
Olivier Bourdon
9d68c5658f Add Grafana dashboard with Prometheus datasource for remote storage adapter
Change-Id: I7e18f50a4fad80ee194d21af058ee945089408f1
2017-07-24 16:13:50 +02:00
Simon Pasquier
2959ab41f3 Rename Prometheus alerts for consistency
Change-Id: I96fb789bf73af22d56fc6c6980626647f87409d4
2017-07-24 15:38:28 +02:00
Olivier Bourdon
0e92c3d41f Fix Prometheus dashboard
Change-Id: I8256086f6c89b781e3eae047afd24c0ce7833bef
2017-07-24 15:05:44 +02:00
Olivier Bourdon
793be69566 Reorder related panel items
Change-Id: I4df4224a78a31fde52a3beb538fbce0048c0cdad
2017-07-24 14:56:29 +02:00
Olivier Bourdon
a69805ee62 Remove non-existing metrics
Change-Id: Ief97b794d167c9b307b6ac5b4906bf360e1933fb
2017-07-24 14:53:51 +02:00
Olivier Bourdon
ac64feebbf Remove aliasColors
Change-Id: I816944c1dffd3217bdc4a849458fea2f3203135c
2017-07-24 14:53:00 +02:00
Olivier Bourdon
4943a0ded1 Fix exporters deployment error
On compute nodes we get:
cmp01.virtual-mcp11-contrail.local:
----------
          ID: libvirt_exporter_service
    Function: service.running
        Name: libvirt-exporter
      Result: False
     Comment: The following requisites were not found:
                                 watch:
                                     file: libvirt_exporter_service_config_file

Change-Id: I31069ec37702f8466da27b76c0becc6802cea272
2017-07-21 08:18:47 +02:00
Simon Pasquier
1ac10079c9 Split infra and workload monitoring for K8S
This change allows to enable the monitoring only for the K8S
infrastructure (default) or for both the infrastructure and the
workloads running on top of K8S.

Change-Id: Idc68576f2d64840c0454fadc204174027bbaa454
2017-07-19 12:00:03 +02:00
Olivier Bourdon
3b4ad1fb29 Fix jmx-exporter service configuration
Change-Id: Ie889b5f8d485aa7f41166fc3f90bb6554681c68e
2017-07-11 16:07:48 +02:00
Olivier Bourdon
31bfa9d94c Fix exporters configuration to be generic
Change-Id: If5cf3662f96311a5f37055c0f62934ba5407a978
2017-07-11 16:07:45 +02:00
Olivier Bourdon
e1539aee0e Add Prometheus metrics relabel configuration for exporters
Tested succesfully with the following test cases:
            source_labels: instance
            source_labels: instance,__name__,test
            source_labels: [instance,__name__,test]
            source_labels:
              - instance
              - __name__
              - test

Change-Id: I9750bb0a4204335dd202cc4020445e850824c497
Depends-On: I2f85fe2f9cc67d2d03d5e8dad5ac0b74e82b0fcc
2017-06-29 11:01:15 +02:00
Bartosz Kupidura
9a986bebf7 Merge grains with pillars for static targets
Change-Id: I2f85fe2f9cc67d2d03d5e8dad5ac0b74e82b0fcc
2017-06-28 09:22:42 +02:00
jenkins-mk jenkins-mk
220497bb8e Merge "Add Prometheus exporters" 2017-06-27 11:53:38 +00:00
jenkins-mk jenkins-mk
f67908f290 Merge "Generate 'route' config entry with macro" 2017-06-27 10:28:07 +00:00
Bartosz Kupidura
a5123c928a Merge "Support for custom add custom labels from reclass" 2017-06-27 10:28:01 +00:00
Olivier Bourdon
e8c7be754f Add Prometheus exporters
Change-Id: I71538bc6b39edaa6a0856c8bba9de11b88cec67a
2017-06-27 10:30:24 +02:00
Aleksey Kasatkin
ba40adbc35 Fix typo in job name for k8s services in prometheus.yml
Right wording is:

    job_name: 'kubernetes-service-endpoints'

E.g., see 14d0604aba/documentation/examples/prometheus-kubernetes.yml (L97)

Change-Id: I997ffc96b1ab6b81e2e634d0be48ad73daf2f077
2017-06-26 13:42:45 +00:00
Bartosz Kupidura
b845f3fc69 Support for custom add custom labels from reclass
This change will enable possibility to add custom
labels to alerts.
This can be used ex. to provite routing key for
pushkin (notification service)

If given key already exists in alert, it will NOT
be overriden.

prometheus:
  server:
    config:
      alert:
        labels_add:
          route: 'mail,skype'

Change-Id: I0b21d8a9695e8a15d0d6b80e6ddd0476682c80c2
2017-06-25 14:53:01 +02:00
Bartosz Kupidura
ce0834f152 Generate 'route' config entry with macro
Change-Id: Id258bea57281e761fdeab10c30edcf32e695163e
2017-06-25 14:02:42 +02:00
Bartosz Kupidura
9b3d774430 We need empty {} to evaluate alerts
For some alerts we use reclass to get variables
(ex. threshold). To get those reclass variables
we use `prometheus_server` dict, which on nodes
without prometheus class is None.
We need to provide at least empty dict in that
case.

Change-Id: Ic1f2c1776c4852701dcf9f8a76f933926b2d9aec
2017-06-22 15:27:49 +02:00
Bartosz Kupidura
45d0bb8fe7 Remove hardcoded username
Change-Id: I3103dc4c174be7b8c199508b0b53941e687012bf
Depends-On: Ib4c218af6e880963cadff807246d65ff68ddf5e1
2017-06-19 14:54:45 +02:00
Filip Pytloun
f4755dfd02 Merge "Add remote_write" 2017-05-30 10:22:05 +00:00
Bartosz Kupidura
13f39f2ac1 Add remote_write
Minor fix for application naming

Change-Id: Ie725b43a5aa9967c31ef482429176bccc21d9f02
2017-05-30 11:42:58 +02:00
Swann Croiset
84f01423df Fix alert typo when using FOR directive
Change-Id: I52d62ff13041cabea331d4d7ca24a6cf525efafd
2017-05-26 19:11:59 +02:00
Bartosz Kupidura
e7de11cf46 Use builtin autodiscovery of alertmanager
Change-Id: I36bdb3a4e90687d32724e5e248d5862567c36f83
Depends-On: I91e23f17858deb2ef937fcd42948641eb371474b
2017-05-25 10:46:56 +02:00
Bartosz Kupidura
abec256072 Create data directory for am and prometheus
Change-Id: Ie89009480c06cafb15bb06ded263e49ef194dfc9
2017-05-22 10:56:34 +02:00
Martin Polreich
e9dc06012c Added Kitchen tests and Travis
Change-Id: Ie5620ef3c416df76287d7864dbc46e0b9c66ab79
2017-05-19 15:26:00 +02:00