Commit graph

105 commits

Author SHA1 Message Date
Ales Komarek
22f7ab3572 Merge "Trigger the target down alert after 2 minutes" 2017-09-15 09:51:58 +00:00
Konstantin Hontar
7b14deaf31 Add Slack and mail receiver to AlertManager config
Change-Id: Ib82ef0a7d029f84a7f92a90384bac02a0870b521
2017-09-14 13:24:04 +02:00
Simon Pasquier
cd90c9f842 Trigger the target down alert after 2 minutes
Otherwise the alert fires as soon as Prometheus can't scrape a target.
It is too aggressive in case of transient connectivity issues or
endpoint restart.

Change-Id: Ib3de5b141db7a7f2397bf332844a9c44d38f2d3c
2017-09-12 15:14:21 +02:00
Olivier Bourdon
477ea05109 Fix for removing local addresses as exporters addresses
Change-Id: Ifac8c8960497cb6dfe95e493583f07cee4d0833d
2017-09-01 11:00:50 +02:00
Ildar Svetlov
1aaa416bdc Add Prometheus Stats dashboard
Change-Id: Icad3d5bee52546288f178ab980f60d8355a36a13
2017-08-30 12:34:54 +00:00
Bartosz Kupidura
dc36045033 Allow to pass metrics_path for static jobs
Change-Id: Ifa4fa71241bf8d2c7e0a4960192d3499252516aa
2017-08-21 11:32:02 +02:00
Bartosz Kupidura
d949dcef88 Revert "Add Jenkins to Prometheus targets"
This reverts commit db1f492198.

Change-Id: I4db9c61f1061021e02e2e28ce6132449a7cb5586
2017-08-21 09:18:58 +00:00
Konstantin Hontar
db1f492198 Add Jenkins to Prometheus targets
Change-Id: Ie9853b9860b34536ded4a7583c48c735e75bb7b7
2017-08-18 13:12:03 +03:00
Bartosz Kupidura
94c1a3c99a Pass all options for static target from formula
Change-Id: I21c84341d5a585862d21f75a4d263f8f5781675d
2017-08-04 18:33:56 +02:00
Ales Komarek
dd19c7f616 Merge changes from topics 'prometheus/alert', 'rename-prometheus-alerts'
* changes:
  Add Prometheus alerts
  Rename Prometheus alerts for consistency
2017-07-26 12:21:51 +00:00
Olivier Bourdon
5b6b583c42 Add Prometheus alerts
Change-Id: I4ad10555d728d62c8e6504659d30558f95b410ac
2017-07-26 13:45:51 +02:00
mcp-jenkins
90b074f356 Merge "Add Grafana dashboard with Prometheus datasource for remote storage adapter" 2017-07-25 15:21:03 +00:00
Olivier Bourdon
9d68c5658f Add Grafana dashboard with Prometheus datasource for remote storage adapter
Change-Id: I7e18f50a4fad80ee194d21af058ee945089408f1
2017-07-24 16:13:50 +02:00
Simon Pasquier
2959ab41f3 Rename Prometheus alerts for consistency
Change-Id: I96fb789bf73af22d56fc6c6980626647f87409d4
2017-07-24 15:38:28 +02:00
Olivier Bourdon
0e92c3d41f Fix Prometheus dashboard
Change-Id: I8256086f6c89b781e3eae047afd24c0ce7833bef
2017-07-24 15:05:44 +02:00
Olivier Bourdon
793be69566 Reorder related panel items
Change-Id: I4df4224a78a31fde52a3beb538fbce0048c0cdad
2017-07-24 14:56:29 +02:00
Olivier Bourdon
a69805ee62 Remove non-existing metrics
Change-Id: Ief97b794d167c9b307b6ac5b4906bf360e1933fb
2017-07-24 14:53:51 +02:00
Olivier Bourdon
ac64feebbf Remove aliasColors
Change-Id: I816944c1dffd3217bdc4a849458fea2f3203135c
2017-07-24 14:53:00 +02:00
Olivier Bourdon
4943a0ded1 Fix exporters deployment error
On compute nodes we get:
cmp01.virtual-mcp11-contrail.local:
----------
          ID: libvirt_exporter_service
    Function: service.running
        Name: libvirt-exporter
      Result: False
     Comment: The following requisites were not found:
                                 watch:
                                     file: libvirt_exporter_service_config_file

Change-Id: I31069ec37702f8466da27b76c0becc6802cea272
2017-07-21 08:18:47 +02:00
Simon Pasquier
1ac10079c9 Split infra and workload monitoring for K8S
This change allows to enable the monitoring only for the K8S
infrastructure (default) or for both the infrastructure and the
workloads running on top of K8S.

Change-Id: Idc68576f2d64840c0454fadc204174027bbaa454
2017-07-19 12:00:03 +02:00
Olivier Bourdon
3b4ad1fb29 Fix jmx-exporter service configuration
Change-Id: Ie889b5f8d485aa7f41166fc3f90bb6554681c68e
2017-07-11 16:07:48 +02:00
Olivier Bourdon
31bfa9d94c Fix exporters configuration to be generic
Change-Id: If5cf3662f96311a5f37055c0f62934ba5407a978
2017-07-11 16:07:45 +02:00
Olivier Bourdon
e1539aee0e Add Prometheus metrics relabel configuration for exporters
Tested succesfully with the following test cases:
            source_labels: instance
            source_labels: instance,__name__,test
            source_labels: [instance,__name__,test]
            source_labels:
              - instance
              - __name__
              - test

Change-Id: I9750bb0a4204335dd202cc4020445e850824c497
Depends-On: I2f85fe2f9cc67d2d03d5e8dad5ac0b74e82b0fcc
2017-06-29 11:01:15 +02:00
Bartosz Kupidura
9a986bebf7 Merge grains with pillars for static targets
Change-Id: I2f85fe2f9cc67d2d03d5e8dad5ac0b74e82b0fcc
2017-06-28 09:22:42 +02:00
jenkins-mk jenkins-mk
220497bb8e Merge "Add Prometheus exporters" 2017-06-27 11:53:38 +00:00
jenkins-mk jenkins-mk
f67908f290 Merge "Generate 'route' config entry with macro" 2017-06-27 10:28:07 +00:00
Bartosz Kupidura
a5123c928a Merge "Support for custom add custom labels from reclass" 2017-06-27 10:28:01 +00:00
Olivier Bourdon
e8c7be754f Add Prometheus exporters
Change-Id: I71538bc6b39edaa6a0856c8bba9de11b88cec67a
2017-06-27 10:30:24 +02:00
Aleksey Kasatkin
ba40adbc35 Fix typo in job name for k8s services in prometheus.yml
Right wording is:

    job_name: 'kubernetes-service-endpoints'

E.g., see 14d0604aba/documentation/examples/prometheus-kubernetes.yml (L97)

Change-Id: I997ffc96b1ab6b81e2e634d0be48ad73daf2f077
2017-06-26 13:42:45 +00:00
Bartosz Kupidura
b845f3fc69 Support for custom add custom labels from reclass
This change will enable possibility to add custom
labels to alerts.
This can be used ex. to provite routing key for
pushkin (notification service)

If given key already exists in alert, it will NOT
be overriden.

prometheus:
  server:
    config:
      alert:
        labels_add:
          route: 'mail,skype'

Change-Id: I0b21d8a9695e8a15d0d6b80e6ddd0476682c80c2
2017-06-25 14:53:01 +02:00
Bartosz Kupidura
ce0834f152 Generate 'route' config entry with macro
Change-Id: Id258bea57281e761fdeab10c30edcf32e695163e
2017-06-25 14:02:42 +02:00
Bartosz Kupidura
9b3d774430 We need empty {} to evaluate alerts
For some alerts we use reclass to get variables
(ex. threshold). To get those reclass variables
we use `prometheus_server` dict, which on nodes
without prometheus class is None.
We need to provide at least empty dict in that
case.

Change-Id: Ic1f2c1776c4852701dcf9f8a76f933926b2d9aec
2017-06-22 15:27:49 +02:00
Bartosz Kupidura
45d0bb8fe7 Remove hardcoded username
Change-Id: I3103dc4c174be7b8c199508b0b53941e687012bf
Depends-On: Ib4c218af6e880963cadff807246d65ff68ddf5e1
2017-06-19 14:54:45 +02:00
Filip Pytloun
f4755dfd02 Merge "Add remote_write" 2017-05-30 10:22:05 +00:00
Bartosz Kupidura
13f39f2ac1 Add remote_write
Minor fix for application naming

Change-Id: Ie725b43a5aa9967c31ef482429176bccc21d9f02
2017-05-30 11:42:58 +02:00
Swann Croiset
84f01423df Fix alert typo when using FOR directive
Change-Id: I52d62ff13041cabea331d4d7ca24a6cf525efafd
2017-05-26 19:11:59 +02:00
Bartosz Kupidura
e7de11cf46 Use builtin autodiscovery of alertmanager
Change-Id: I36bdb3a4e90687d32724e5e248d5862567c36f83
Depends-On: I91e23f17858deb2ef937fcd42948641eb371474b
2017-05-25 10:46:56 +02:00
Bartosz Kupidura
abec256072 Create data directory for am and prometheus
Change-Id: Ie89009480c06cafb15bb06ded263e49ef194dfc9
2017-05-22 10:56:34 +02:00
Martin Polreich
e9dc06012c Added Kitchen tests and Travis
Change-Id: Ie5620ef3c416df76287d7864dbc46e0b9c66ab79
2017-05-19 15:26:00 +02:00
Bartosz Kupidura
bb791375c6 Disable by default kubernetes and dns target
Change-Id: Ie5ad3d8b630d2e634f143099641f06aa9f75e30f
2017-05-12 17:30:35 +02:00
Bartosz Kupidura
953510a4d8 Render alertmanager config without yaml filter
Change-Id: Ie86bb95ab7bf37291347ca4533fe20bbd95484e1
2017-05-12 13:52:48 +02:00
Tomáš Kukrál
3d960a15b4
fix meta/salt with missing pillar
Change-Id: I43333c76bb3814d2c6b98849df109ef97523dc7e
2017-05-11 22:06:19 +02:00
Bartosz Kupidura
eb97a444ed Enable {%raw%} in grafana dashboard template
Change-Id: Iff39e2a916bc320a498568d3f4f02f2a83d03227
2017-05-11 12:15:00 +02:00
Filip Pytloun
d92d2d7ce8 Use salt support to generate grains
Change-Id: I7cc5f00608127e0bf4df5bbd58db4cae9816a402
2017-05-09 10:23:12 +02:00
Bartosz Kupidura
c9e81f45db Import prometheus.server in collector
Change-Id: I42e17bd63df43d9bfec3a302bfecf5290c69e0a8
2017-04-28 14:34:50 +02:00
Martin Polreich
1404088f3c Added ability to disable endpoints, alerts and
recording rules and override alerts/recording
rules from reclass

Change-Id: Ie79409f350f4e7d223955c177718350c38ed8302
2017-04-28 11:03:21 +02:00
Swann Croiset
91d5be301a Add Grafana dashboard for prometheus performance metrics
Change-Id: I8e2eed69a9da6c5f87eccc3a4272085cf1fb636f
2017-04-26 15:25:17 +00:00
Bartosz Kupidura
1cb5cbdba4 Send updated grains to Salt mine
Change-Id: I8e90dd0e7c2eabe79046bad0e86a4c020a991fc7
2017-04-25 12:15:11 +02:00
Bartosz Kupidura
9f4d99cb3a Add more generic auto-discovery for statics
Change-Id: I4d20d6f126ec4698bbdae34c8476fe325363bced
2017-04-13 10:21:19 +02:00
Bartosz Kupidura
5f644a1921 Typo
Change-Id: I7e04706411018b117e8d2ec523667f6048cf932e
2017-04-11 15:34:25 +02:00
Bartosz Kupidura
2b784c85b9 Add support.yml for alerts and recording rules
Change-Id: If1927033922c350257999f59ba3031445689e11b
2017-04-11 12:17:08 +02:00
Bartosz Kupidura
453e290495 Add ssl options for etcd
Change-Id: I45afdf25d8d3ebc1ceab4a38d8cc61435fed1700
2017-03-30 15:43:41 +02:00
Bartosz Kupidura
8642002cf8 Get dns autodiscovery from reclass
Change-Id: I8850be1a118662ebf75a3e0fc2540691ab027220
2017-03-30 12:05:57 +02:00
Bartosz Kupidura
597b0678bf Better grain handling for telegraf
Replace '-' to '_' in prometheus config dir
Add possibility to add recording rules

Change-Id: Ifc85a95ae62cb0f1b810a3b4f718e8d92f9c6209
2017-03-29 13:54:26 +02:00
Bartosz Kupidura
2f70396a78 Add prometheus, alertmanager, pushgateway configs
Change-Id: I7f146e0207ada58f0723b14d0a625a6bce67ea9f
2017-03-17 17:50:23 +01:00