UlricQin

{{$time_duration := sub now.Unix $event.FirstTriggerTime }}{{if $event.IsRecovered}}{{$time_duration = sub $event.LastEvalTime $event.FirstTriggerTime }}{{end}}
{{if $event.IsRecovered}}🟢{{else}}🔴{{end}} {{$event.RuleName}} 

---
**💨告警级别**: {{$event.Severity}}级
{{- if $event.RuleNote}}
**📜规则备注**: {{$event.RuleNote}}   
{{- end -}}
{{if not $event.IsRecovered}}
**⏱️发送时间**: {{timeformat $event.TriggerTime}}   

UlricQin

#### {{if $event.IsRecovered}}<font color="#008800">💚{{$event.RuleName}}</font>{{else}}<font color="#FF0000">💔{{$event.RuleName}}</font>{{end}}
---
{{$time_duration := sub now.Unix $event.FirstTriggerTime }}{{if $event.IsRecovered}}{{$time_duration = sub $event.LastEvalTime $event.FirstTriggerTime }}{{end}}
**💨告警级别**: {{$event.Severity}}级   

{{if $event.RuleNote}}
**📜规则备注**: {{$event.RuleNote}}   
{{end}}

{{if not $event.IsRecovered}}

UlricQin

http server 代码如下：

package main

import (
	"net/http"
)

func main() {

UlricQin

{
    "name": "Host Table NG",
    "tags": "",
    "ident": "",
    "uuid": 1756720567064000,
    "configs": {
        "var": [
            {
                "name": "prom",
                "label": "PROM",

UlricQin

package main

import (
	"encoding/json"
	"fmt"

	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

UlricQin

1. 修改webapi.conf，具体调整如下：

修改企微机器人那个NotifyChannels，把“企微机器人”换成“企微用户”
新增一个 NotifyChannels：

[[NotifyChannels]]
Label = "企微机器人"
Key = "webot"

修改 ContactKeys，修改这个：

UlricQin

{
    "name": "Kafka - by jmx_exporter",
    "tags": "",
    "configs": {
        "var": [
            {
                "type": "query",
                "name": "cluster",
                "definition": "label_values(kafka_brokers, cluster)"
            },

UlricQin

{
    "name": "Kafka - by kafka_exporter",
    "tags": "Kafka Prometheus",
    "configs": {
        "var": [
            {
                "name": "cluster",
                "definition": "label_values(kafka_brokers,  cluster)",
                "type": "query"
            }

UlricQin

---
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames: [
  "kafka.log:type=Log,name=Size,topic=*,partition=*",
  "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions",
  "kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount",
  "kafka.server:type=ReplicaManager,name=LeaderCount",
  "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec",
  "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec",

UlricQin

cpu 使用率

container!="POD" 是过滤掉 pause 容器。这个 promql 比较难以理解，分子是：每秒内，容器用了多少CPU时间，分母是：每秒内，被限制使用多少CPU时间

sum(
  irate(container_cpu_usage_seconds_total[3m])
) by (pod,id,namespace,container,ident,image)
/
sum(