85 lines
2.9 KiB
Plaintext
85 lines
2.9 KiB
Plaintext
[#prometheus]
|
|
= Prometheus
|
|
|
|
OliveTin supports basic Prometheus metrics, and the project is interested to hear about what more metrics people would find useful, as well!
|
|
|
|
To enable Prometheus support;
|
|
|
|
.`config.yaml`
|
|
[source,yaml]
|
|
----
|
|
logLevel: INFO
|
|
|
|
prometheus:
|
|
enabled: true
|
|
defaultGoMetrics: false
|
|
----
|
|
|
|
It is required to restart OliveTin after changing the `prometheus` configuration.
|
|
|
|
The `defaultGoMetrics` option will enable the default Go metrics, which expose metrics like go_memstats_alloc_bytes, or go_memstats_heap_alloc_bytes,
|
|
and generally most people don't need these.
|
|
|
|
This will give you metrics available at http://yourserver:1337/metrics. The page should look something like this;
|
|
|
|
[source]
|
|
----
|
|
# HELP olivetin_actions_requested_count The actions requested count
|
|
# TYPE olivetin_actions_requested_count counter
|
|
olivetin_actions_requested_count 0
|
|
# HELP olivetin_action_executions_total Total number of finished action executions grouped by result.
|
|
# TYPE olivetin_action_executions_total counter
|
|
olivetin_action_executions_total{result="success"} 0
|
|
olivetin_action_executions_total{result="failed"} 0
|
|
olivetin_action_executions_total{result="blocked"} 0
|
|
olivetin_action_executions_total{result="timeout"} 0
|
|
olivetin_action_executions_total{result="error"} 0
|
|
# HELP olivetin_action_execution_duration_seconds Action execution duration in seconds from start to finish.
|
|
# TYPE olivetin_action_execution_duration_seconds histogram
|
|
olivetin_action_execution_duration_seconds_bucket{le="0.1"} 0
|
|
# HELP olivetin_config_action_count Then number of actions in the config file
|
|
# TYPE olivetin_config_action_count gauge
|
|
olivetin_config_action_count 18
|
|
# HELP olivetin_config_reloaded_count The number of times the config has been reloaded
|
|
# TYPE olivetin_config_reloaded_count counter
|
|
olivetin_config_reloaded_count 1
|
|
----
|
|
|
|
=== Failed job monitoring
|
|
|
|
Finished action executions are counted in `olivetin_action_executions_total` with a `result` label:
|
|
|
|
[cols="1,2"]
|
|
|===
|
|
| `success` | Command finished with exit code 0
|
|
| `failed` | Command ran but exited with a non-zero code
|
|
| `timeout` | Command exceeded its configured timeout
|
|
| `blocked` | Execution was blocked (ACL, rate limit, concurrency, or queue limit)
|
|
| `error` | Execution failed before the command ran (for example, invalid arguments)
|
|
|===
|
|
|
|
`olivetin_action_execution_duration_seconds` records how long each finished execution took.
|
|
|
|
Example Prometheus alert rules:
|
|
|
|
[source,yaml]
|
|
----
|
|
groups:
|
|
- name: olivetin
|
|
rules:
|
|
- alert: OliveTinActionFailed
|
|
expr: increase(olivetin_action_executions_total{result="failed"}[15m]) > 0
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: OliveTin action failed with non-zero exit code
|
|
|
|
- alert: OliveTinActionTimedOut
|
|
expr: increase(olivetin_action_executions_total{result="timeout"}[15m]) > 0
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: OliveTin action timed out
|
|
----
|
|
|