Skip to content

Commit c781fc9

Browse files
zhufuyizhufuyi
committed
add REAME
1 parent 46527ef commit c781fc9

File tree

10 files changed

+760
-50
lines changed

10 files changed

+760
-50
lines changed

‎.github/RELEASE.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
## Change log
22

3-
1. Change page field size to limit.
4-
2. Supporting custom bingdingXXX and response in the web(protobuf).
5-
3. Match message name xxxID in the protobuf file.
3+
1. Modify the returned ID type, it will affect the ID types of GetByID and List for `⓵Create web service based on sql`, which are consistent with the ID types in the database.
4+
5+
> If you are using code for `⓵Create web service based on sql` before v1.8.6, do not modify the sponge version under go.mod and upgrade to v1.8.6 or above. Otherwise, the List interface will return empty data because the original `size` field has become invalid (replaced by `limit` field).

‎pkg/grpc/metrics/monitor-example-cn.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
### 启动Prometheus和Grafana服务
2+
3+
**(1) prometheus服务**
4+
5+
这是 [prometheus服务启动脚本](https://github.com/zhufuyi/sponge/tree/main/test/server/monitor/prometheus)，启动prometheus服务：
6+
7+
```bash
8+
docker-compose up -d
9+
```
10+
11+
在浏览器访问prometheus主页 [http://localhost:9090](http://localhost:9090/) 。
12+
13+
<br>
14+
15+
**(2) grafana服务**
16+
17+
这是 [grafana服务启动脚本](https://github.com/zhufuyi/sponge/tree/main/test/server/monitor/grafana)，启动grafana服务：
18+
19+
```bash
20+
docker-compose up -d
21+
```
22+
23+
在浏览器访问 grafana 主页面 [http://localhost:33000](http://localhost:33000) ，设置prometheus的数据源 `http://localhost:9090` 。
24+
25+
> [!attention] 在grafana导入监控面板的json的**datasource**值，必须与在grafana设置的prometheus的数据源名称(这里是**Prometheus**)要一致，否则图标上无法显示数据。
26+
27+
<br>
28+
29+
### web服务监控示例
30+
31+
以`⓵基于sql创建web服务`代码为例，默认提供指标接口 [http://localhost:8080/metrics](http://localhost:8080/metrics) 。
32+
33+
**(1) 在prometheus添加监控目标**
34+
35+
打开prometheus配置文件`prometheus.yml`，添加采集目标：
36+
37+
```bash
38+
- job_name: 'http-edusys'
39+
scrape_interval: 10s
40+
static_configs:
41+
- targets: ['localhost:8080']
42+
```
43+
44+
> [!attention] 在启动Prometheus服务前，必须将文件`prometheus.yml`权限改为`0777`，否则使用vim修改`prometheus.yml`文件无法同步到容器中。
45+
46+
执行请求使prometheus配置生效 `curl -X POST http://localhost:9090/-/reload` ，稍等一会，然后在浏览器访问 [http://localhost:9090/targets](http://localhost:9090/targets)，检查新添加的采集目标是否生效。
47+
48+
<br>
49+
50+
**(2) 在grafana添加监控面板**
51+
52+
把 [http 监控面板](https://github.com/zhufuyi/sponge/blob/main/pkg/gin/middleware/metrics/gin_grafana.json) 导入到grafana，如果监控界面没有数据显示，检查json里的数据源名称与grafana配置prometheus数据源名称是否一致。
53+
54+
<br>
55+
56+
**(3) 压测接口，观察监控数据**
57+
58+
使用[wrk](https://github.com/wg/wrk)工具压测接口
59+
60+
```bash
61+
# 接口1
62+
wrk -t2 -c10 -d10s http://192.168.3.27:8080/api/v1/teacher/1
63+
64+
# 接口2
65+
wrk -t2 -c10 -d10s http://192.168.3.27:8080/api/v1/course/1
66+
```
67+
68+
监控界面如下图所示：
69+
70+
![http-grafana](https://go-sponge.com/assets/images/http-grafana.jpg)
71+
72+
<br>
73+
74+
### grpc服务监控示例
75+
76+
以`⓶基于sql创建grpc服务`代码为例，默认提供指标接口 [http://localhost:8283/metrics](http://localhost:8283/metrics) 。
77+
78+
**(1) 在prometheus添加监控目标**
79+
80+
打开prometheus配置文件`prometheus.yml`，添加采集目标：
81+
82+
```yaml
83+
- job_name: 'rpc-server-user'
84+
scrape_interval: 10s
85+
static_configs:
86+
- targets: ['localhost:8283']
87+
```
88+
89+
> [!attention] 在启动Prometheus服务前，必须将文件`prometheus.yml`权限改为`0777`，否则使用vim修改`prometheus.yml`文件无法同步到容器中。
90+
91+
执行请求使prometheus配置生效 `curl -X POST http://localhost:9090/-/reload` ，稍等一会，然后在浏览器访问 [http://localhost:9090/targets](http://localhost:9090/targets)， 检查新添加的采集目标是否生效。
92+
93+
<br>
94+
95+
**(2) 在grafana添加监控面板**
96+
97+
把 [grpc server 监控面板](https://github.com/zhufuyi/sponge/blob/main/pkg/grpc/metrics/server_grafana.json) 导入到grafana，如果监控界面没有数据显示，检查json里的数据源名称与grafana配置prometheus数据源名称是否一致。
98+
99+
<br>
100+
101+
**(3) 压测grpc api，观察监控数据**
102+
103+
使用`Goland` IDE打开`internal/service/teacher_client_test.go`文件，对**Test_teacherService_methods** 或 **Test_teacherService_benchmark** 下各个方法进行测试。
104+
105+
监控界面如下图所示。
106+
![rpc-grafana](https://go-sponge.com/assets/images/rpc-grafana.jpg)
107+
108+
<br>
109+
110+
上面是grpc服务端的监控，grpc的客户端的监控也类似，[grpc client 监控面板](https://github.com/zhufuyi/sponge/blob/main/pkg/grpc/metrics/client_grafana.json) 。
111+
112+
<br>
113+
114+
### 在prometheus自动添加和移除监控目标
115+
116+
实际使用中服务数量比较多，手动添加监控目标到prometheus比较繁琐，也容易出错。prometheus支持使用`consul`的服务注册与发现进行动态配置，自动添加和移除监控目标。
117+
118+
在本地启动 consul 服务，这是 [consul 服务启动脚本](https://github.com/zhufuyi/sponge/tree/main/test/server/consul)，启动consul服务：
119+
120+
```bash
121+
docker-compose up -d
122+
```
123+
124+
打开 prometheus 配置 prometheus.yml，添加consul配置：
125+
126+
```yaml
127+
- job_name: 'consul-micro-exporter'
128+
consul_sd_configs:
129+
- server: 'localhost:8500'
130+
services: []
131+
relabel_configs:
132+
- source_labels: [__meta_consul_tags]
133+
regex: .*user.*
134+
action: keep
135+
- regex: __meta_consul_service_metadata_(.+)
136+
action: labelmap
137+
```
138+
139+
执行请求使prometheus配置生效 `curl -X POST http://localhost:9090/-/reload` 。
140+
141+
在prometheus配置好consul服务发现之后，接着把服务的地址信息推送到consul，推送信息 user_exporter.json 文件内容如下：
142+
143+
```json
144+
{
145+
"ID": "user-exporter",
146+
"Name": "user",
147+
"Tags": [
148+
"user-exporter"
149+
],
150+
"Address": "localhost",
151+
"Port": 8283,
152+
"Meta": {
153+
"env": "dev",
154+
"project": "user"
155+
},
156+
"EnableTagOverride": false,
157+
"Check": {
158+
"HTTP": "http://localhost:8283/metrics",
159+
"Interval": "10s"
160+
},
161+
"Weights": {
162+
"Passing": 10,
163+
"Warning": 1
164+
}
165+
}
166+
```
167+
168+
> curl -XPUT --data @user_exporter.json http://localhost:8500/v1/agent/service/register
169+
170+
稍等一会，然后在浏览器打开 [http://localhost:9090/targets](http://localhost:9090/targets) 检查新添加的采集目标是否生效。然后关闭服务，稍等一会，检查是否自动移除采集目标。
171+
172+
> [!tip] 在web或grpc务中，通常是使用程序代码自动把json信息提交给consul，不是通过命令，web或grpc务正常启动服务后，Prometheus就可以动态获取到监控目标，web或grpc务停止后，Prometheus自动移除监控目标。
173+
174+
<br>

‎pkg/grpc/metrics/monitor-example-en.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
### Starting Prometheus and Grafana Services
2+
3+
**(1) Prometheus Service**
4+
5+
Here is the [script for starting the Prometheus service](https://github.com/zhufuyi/sponge/tree/main/test/server/monitor/prometheus). Start the Prometheus service:
6+
7+
```bash
8+
docker-compose up -d
9+
```
10+
11+
Access the Prometheus homepage in your browser at [http://localhost:9090](http://localhost:9090/).
12+
13+
<br>
14+
15+
**(2) Grafana Service**
16+
17+
Here is the [script for starting the Grafana service](https://github.com/zhufuyi/sponge/tree/main/test/server/monitor/grafana). Start the Grafana service:
18+
19+
```bash
20+
docker-compose up -d
21+
```
22+
23+
Access the main Grafana page in your browser at [http://localhost:33000](http://localhost:33000), and configure the Prometheus data source to be `http://localhost:9090`.
24+
25+
> [!attention] When importing JSON dashboards into Grafana, the **datasource** value in the JSON must match the name of the Prometheus data source you set in Grafana (in this case, **Prometheus**), or the graphs won't display data.
26+
27+
<br>
28+
29+
### Web Service Monitoring Example
30+
31+
Taking the code for the `⓵ Web Service Based on SQL` as an example, it provides a default metrics interface at [http://localhost:8080/metrics](http://localhost:8080/metrics).
32+
33+
**(1) Adding Monitoring Targets to Prometheus**
34+
35+
Open the Prometheus configuration file `prometheus.yml` and add scraping targets:
36+
37+
```bash
38+
- job_name: 'http-edusys'
39+
scrape_interval: 10s
40+
static_configs:
41+
- targets: ['localhost:8080']
42+
```
43+
44+
> [!attention] Before starting the Prometheus service, make sure to change the permissions of the `prometheus.yml` file to `0777`. Otherwise, changes made to `prometheus.yml` using `vim` won't be synchronized with the container.
45+
46+
Trigger the Prometheus configuration to take effect by executing `curl -X POST http://localhost:9090/-/reload`. Wait for a moment, then access [http://localhost:9090/targets](http://localhost:9090/targets) in your browser to check if the newly added scraping target is active.
47+
48+
<br>
49+
50+
**(2) Adding Monitoring Dashboards to Grafana**
51+
52+
Import the [HTTP monitoring dashboard](https://github.com/zhufuyi/sponge/blob/main/pkg/gin/middleware/metrics/gin_grafana.json) into Grafana. If the monitoring interface does not display data, check if the data source name in the JSON matches the Prometheus data source name in Grafana's configuration.
53+
54+
<br>
55+
56+
**(3) Load Testing the API and Observing Monitoring Data**
57+
58+
Use the [wrk](https://github.com/wg/wrk) tool to perform load testing on the API:
59+
60+
```bash
61+
# test 1
62+
wrk -t2 -c10 -d10s http://192.168.3.27:8080/api/v1/teacher/1
63+
64+
# test 2
65+
wrk -t2 -c10 -d10s http://192.168.3.27:8080/api/v1/course/1
66+
```
67+
68+
The monitoring interface will look like the following image:
69+
70+
![http-grafana](https://go-sponge.com/assets/images/http-grafana.jpg)
71+
72+
<br>
73+
74+
### GRPC Service Monitoring Example
75+
76+
Taking the code for the `⓶Create grpc service based on sql` as an example, it provides a default metrics interface at [http://localhost:8283/metrics](http://localhost:8283/metrics).
77+
78+
**(1) Adding Monitoring Targets to Prometheus**
79+
80+
Open the Prometheus configuration file `prometheus.yml` and add scraping targets:
81+
82+
```yaml
83+
- job_name: 'rpc-server-user'
84+
scrape_interval: 10s
85+
static_configs:
86+
- targets: ['localhost:8283']
87+
```
88+
89+
> [!attention] Before starting the Prometheus service, make sure to change the permissions of the `prometheus.yml` file to `0777`. Otherwise, changes made to `prometheus.yml` using `vim` won't be synchronized with the container.
90+
91+
Trigger the Prometheus configuration to take effect by executing `curl -X POST http://localhost:9090/-/reload`. Wait for a moment, then access [http://localhost:9090/targets](http://localhost:9090/targets) in your browser to check if the newly added scraping target is active.
92+
93+
<br>
94+
95+
**(2) Adding Monitoring Dashboards to Grafana**
96+
97+
Import the [grpc service monitoring dashboard](https://github.com/zhufuyi/sponge/blob/main/pkg/grpc/metrics/server_grafana.json) into Grafana. If the monitoring interface does not display data, check if the data source name in the JSON matches the Prometheus data source name in Grafana's configuration.
98+
99+
<br>
100+
101+
**(3) Load Testing GRPC Methods and Observing Monitoring Data**
102+
103+
Open the `internal/service/teacher_client_test.go` file using the `Goland` IDE and test various methods under **Test_teacherService_methods** or **Test_teacherService_benchmark**.
104+
105+
The monitoring interface will look like the following image:
106+
107+
![rpc-grafana](https://go-sponge.com/assets/images/rpc-grafana.jpg)
108+
109+
<br>
110+
111+
The monitoring for the grpc client is similar to the server monitoring, and you can find the [grpc client monitoring dashboard](https://github.com/zhufuyi/sponge/blob/main/pkg/grpc/metrics/client_grafana.json) here.
112+
113+
<br>
114+
115+
### Automatically Adding and Removing Monitoring Targets in Prometheus
116+
117+
In real-world scenarios, managing monitoring targets in Prometheus manually can be cumbersome and error-prone, especially when dealing with a large number of services. Prometheus supports dynamic configuration using service discovery with tools like `Consul` to automatically add and remove monitoring targets.
118+
119+
Start a local Consul service using the [Consul service start script](https://github.com/zhufuyi/sponge/tree/main/test/server/consul):
120+
121+
```bash
122+
docker-compose up -d
123+
```
124+
125+
Open the Prometheus configuration file `prometheus.yml` and add Consul configuration:
126+
127+
```yaml
128+
- job_name: 'consul-micro-exporter'
129+
consul_sd_configs:
130+
- server: 'localhost:8500'
131+
services: []
132+
relabel_configs:
133+
- source_labels: [__meta_consul_tags]
134+
regex: .*user.*
135+
action: keep
136+
- regex: __meta_consul_service_metadata_(.+)
137+
action: labelmap
138+
```
139+
140+
Trigger the Prometheus configuration to take effect by executing `curl -X POST http://localhost:9090/-/reload`.
141+
142+
Once you've configured Prometheus with Consul service discovery, you can push service address information to Consul. Push the information using a JSON file named `user_exporter.json` with content like this:
143+
144+
```json
145+
{
146+
"ID": "user-exporter",
147+
"Name": "user",
148+
"Tags": [
149+
"user-exporter"
150+
],
151+
"Address": "localhost",
152+
"Port": 8283,
153+
"Meta": {
154+
"env": "dev",
155+
"project": "user"
156+
},
157+
"EnableTagOverride": false,
158+
"Check": {
159+
"HTTP": "http://localhost:8283/metrics",
160+
"Interval": "10s"
161+
},
162+
"Weights": {
163+
"Passing": 10,
164+
"Warning": 1
165+
}
166+
}
167+
```
168+
169+
> curl -XPUT --data @user_exporter.json http://localhost:8500/v1/agent/service/register
170+
171+
Wait for a moment, then open [http://localhost:9090/targets](http://localhost:9090/targets) in your browser to check if the newly added scraping target is active. Then, stop the service and wait for a moment to check if the target is automatically removed.
172+
173+
> [!tip] In web or grpc service, the JSON information is usually submitted to Consul automatically via program code, not through commands. After the web or grpc service starts normally, Prometheus can dynamically discover monitoring targets. When the web or grpc service stops, Prometheus automatically removes the monitoring target.
174+
175+
<br>

0 commit comments

Comments
 (0)