监控报警系统

Author: xiaomeng79

02数据存取/10clickhouse.md

@@ -4,3 +4,35 @@
 
 - [ClickHouse](https://github.com/ClickHouse/ClickHouse)
 - [使用场景](https://zhuanlan.zhihu.com/p/98439924)
+
+## 表引擎
+
+### MergeTree
+
+- ReplacingMergeTree ：在数据合并期间删除排序键值相同的重复项。
+- CollapsingMergeTree ：增加状态列`Sign`，主键相同，没成对的状态行保留，多线程情况下-1和1可能存在乱序，导致无法被删除。
+- VersionedCollapsingMergeTree ： 为了解决CollapsingMergeTree乱序写入情况下无法正常折叠问题，新增一列`Version`，主键相同，且Version相同、Sign相反的行，在Compaction时会被删除。
+- SummingMergeTree ： 把所有具有相同主键的行合并为一行，并根据指定的字段进行累加。
+- AggregatingMergeTree ： SummingMergeTree只能累加，这个能进行各种聚合，需要指定聚合函数。
+
+#### 功能
+- 按照主键排序
+- 如果指定了分区键，则可以使用分区
+- 支持数据复制
+- 支持数据采样
+- 并发/索引/分区/crud
+
+### Log
+
+  为了需要写入许多小数据量（少于一百万行）的表的场景而开发的。
+
+#### 功能
+- 数据被顺序append写到磁盘上；
+- 不支持delete、update；
+- 不支持index；
+- 不支持原子性写；
+- insert会阻塞select操作。
+
+### Integration
+
+  该系统表引擎主要用于将外部数据导入到ClickHouse中，或者在ClickHouse中直接操作外部数据源。

17基础设施/0目录.md

@@ -15,4 +15,6 @@
 
 4. [Gateway](./4Gateway.md)
 
-5. [PKI体系](./5PKI.md)
+5. [PKI体系](./5PKI.md)
+
+6. [监控报警系统](./monitor/README.md) 

17基础设施/monitor/README.md

@@ -0,0 +1,24 @@
+# 监控报警系统
+
+## 主要组件
+### prometheus
+
+- [prometheus](https://prometheus.io/) 时序数据库 [prometheus-book](https://yunlzheng.gitbook.io/prometheus-book/) | [prometheus.wang](https://www.prometheus.wang/)
+- 收集客户端
+  - [node_exporter](https://github.com/prometheus/node_exporter)
+  - [blackbox_exporter](https://github.com/prometheus/blackbox_exporter) | [dashboards: 9965-blackbox_exporter](https://grafana.com/grafana/dashboards/9965)
+
+### grafana
+
+- [grafana](https://grafana.com) 可视化 | [创建警报管理](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/)
+- [plugins](https://grafana.com/grafana/plugins/)
+  - [clickhouse](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource/)
+- [dashboards](https://grafana.com/grafana/dashboards/)
+  - [(8919) node_exporter](https://grafana.com/grafana/dashboards/8919)
+  - [(9965) blackbox_exporter](https://grafana.com/grafana/dashboards/9965)
+  - [(13606) clickhouse-performance](https://grafana.com/grafana/dashboards/13606/)
+
+### PrometheusAlert
+
+- [PrometheusAlert](https://github.com/feiyu563/PrometheusAlert) 告警中心
+

17基础设施/monitor/docker-compose.yml

@@ -0,0 +1,128 @@
+version: '3.2'
+
+services:
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    restart: unless-stopped
+    ports:
+      - '9090:9090'
+    user: '0'
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--storage.tsdb.path=/prometheus/data'
+      - '--storage.tsdb.retention=90d'
+      - '--web.enable-lifecycle'
+    volumes:
+      - ./etc/prometheus:/etc/prometheus
+      - ./data/prometheus/data:/prometheus/data
+  #   depends_on:
+  #     - cadvisor
+
+  # cadvisor:
+  #   image: google/cadvisor:latest
+  #   container_name: cadvisor
+  #   restart: unless-stopped
+  #   ports:
+  #     - '8080:8080'
+  #   volumes:
+  #     - /:/rootfs:ro
+  #     - /var/run:/var/run:rw
+  #     - /sys:/sys:ro
+  #     - /var/lib/docker/:/var/lib/docker:ro
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    container_name: node-exporter
+    restart: unless-stopped
+    ports:
+      - '9100:9100'
+    command:
+      - '--path.procfs=/host/proc'
+      - '--path.sysfs=/host/sys'
+      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
+      - '--collector.textfile.directory=/node_exporter/prom'
+    volumes:
+      - /proc:/host/proc
+      - /sys:/host/sys
+      - /:/rootfs
+      - ./etc/node_exporter/prom:/node_exporter/prom
+
+  node-exporter-01:
+    image: prom/node-exporter:latest
+    container_name: node-exporter-01
+    restart: unless-stopped
+    ports:
+      - '9101:9100'
+    command:
+      - '--path.procfs=/host/proc'
+      - '--path.sysfs=/host/sys'
+      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
+      - '--collector.textfile.directory=/node_exporter/prom'
+    volumes:
+      - /proc:/host/proc
+      - /sys:/host/sys
+      - /:/rootfs
+      - ./etc/node_exporter/prom:/node_exporter/prom
+
+  grafana:
+    image: grafana/grafana:8.5.11
+    container_name: grafana
+    restart: unless-stopped
+    ports:
+      - '3000:3000'
+    user: '0'
+    volumes:
+      - ./data/grafana8:/var/lib/grafana
+      - ./etc/grafana/grafana.ini:/etc/grafana/grafana.ini
+    environment:
+      - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=vertamedia-clickhouse-datasource
+      - GF_SECURITY_ADMIN_PASSWORD=password
+      - GF_USERS_ALLOW_SIGN_UP=false
+    
+  prometheusalert:
+    image: feiyu563/prometheus-alert:latest
+    container_name: prometheusalert
+    restart: unless-stopped
+    ports:
+      - '8080:8080'
+    user: '0'
+    volumes:
+      - ./etc/prometheusalert/app.conf:/app/conf/app.conf
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=password
+      - PA_LOGIN_USER=prometheusalert
+      - PA_LOGIN_PASSWORD=prometheusalert
+      - PA_TITLE=PrometheusAlert
+      - PA_OPEN_FEISHU=1
+      - PA_OPEN_DINGDING=1
+      - PA_OPEN_WEIXIN=1
+
+  clickhouse:
+    image: yandex/clickhouse-server
+    container_name: clickhouse
+    restart: unless-stopped
+    ports:
+      - 34424:34424
+      - 9000:9000
+      - 9009:9009
+      - 8123:8123
+    user: '0'
+    volumes:
+      - ./data/clickhouse/data:/var/lib/clickhouse
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+
+  blackbox:
+    image: prom/blackbox-exporter:latest
+    container_name: blackbox
+    restart: unless-stopped
+    command:
+      - "--config.file=/etc/blackbox/blackbox.yml"
+    ports:
+      - 9115:9115
+    user: '0'
+    volumes:
+      - ./etc/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml

17基础设施/monitor/etc/blackbox/blackbox.yml

@@ -0,0 +1,9 @@
+modules:
+  http_2xx:
+    prober: http
+  http_post_2xx:
+    prober: http
+    http:
+      method: POST
+  tcp_connect:
+    prober: tcp