docs: update monitoring metrics document with static-server probes
and correct gitserver named-port configuration
This commit is contained in:
parent
fd232354cc
commit
7eb9c5a7fb
164
docs/metrics.md
164
docs/metrics.md
@ -2,18 +2,27 @@
|
|||||||
|
|
||||||
## 概览
|
## 概览
|
||||||
|
|
||||||
本项目为以下服务暴露 Prometheus 指标:
|
| 服务 | 端点 | 端口 | 说明 |
|
||||||
|
|
||||||
| 服务 | 端点 | 端口 | 类型 |
|
|
||||||
|------|------|------|------|
|
|------|------|------|------|
|
||||||
| app (主站) | `GET /metrics` | 8080 | actix-web |
|
| app | `GET /health` | 8080 | 主站 API (含 `/metrics`) |
|
||||||
| gitserver | `GET :8021/health` | 8021 | actix-web |
|
| static | `GET /health` | 8081 | 静态文件服务 (avatar, blob, media) |
|
||||||
| git-hook | `GET :8083/metrics` | 8083 | hyper (内置) |
|
| gitserver | `GET /health` | 8021 | Git 智能协议 (HTTP + SSH) |
|
||||||
| email-worker | `GET :8084/metrics` | 8084 | hyper (内置) |
|
| git-hook | `GET /health` | 8083 | Hook worker (含 `/metrics`) |
|
||||||
|
| email-worker | `GET /health` | 8084 | 邮件队列 worker (含 `/metrics`) |
|
||||||
|
|
||||||
## 采集配置
|
## 健康检查返回格式
|
||||||
|
|
||||||
### Prometheus scrape 配置
|
所有服务的 `/health` 返回格式一致:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "status": "ok", "db": "ok", "cache": "ok" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- `db: ok` — PostgreSQL 可用
|
||||||
|
- `cache: ok` — Redis 可用
|
||||||
|
- 任一不可用时返回 HTTP `503`,`status: unhealthy`
|
||||||
|
|
||||||
|
## Prometheus 采集配置
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
scrape_configs:
|
scrape_configs:
|
||||||
@ -22,11 +31,6 @@ scrape_configs:
|
|||||||
- targets: ['app:8080']
|
- targets: ['app:8080']
|
||||||
metrics_path: '/metrics'
|
metrics_path: '/metrics'
|
||||||
|
|
||||||
- job_name: 'gitdata-gitserver'
|
|
||||||
static_configs:
|
|
||||||
- targets: ['gitserver:8021']
|
|
||||||
metrics_path: '/metrics'
|
|
||||||
|
|
||||||
- job_name: 'gitdata-git-hook'
|
- job_name: 'gitdata-git-hook'
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ['git-hook:8083']
|
- targets: ['git-hook:8083']
|
||||||
@ -38,30 +42,11 @@ scrape_configs:
|
|||||||
metrics_path: '/metrics'
|
metrics_path: '/metrics'
|
||||||
```
|
```
|
||||||
|
|
||||||
## 健康检查端点
|
> 注:gitserver 和 static-server 目前未暴露 `/metrics`,仅用于健康检查。
|
||||||
|
|
||||||
### gitserver
|
|
||||||
- **端点**: `GET /health`
|
|
||||||
- **端口**: 8021
|
|
||||||
- **返回**:
|
|
||||||
```json
|
|
||||||
{ "status": "ok", "db": "ok", "cache": "ok" }
|
|
||||||
```
|
|
||||||
或 `503` 当 db/cache 任一不可用
|
|
||||||
|
|
||||||
### git-hook
|
|
||||||
- **端点**: `GET /health`
|
|
||||||
- **端口**: 8083
|
|
||||||
- **返回**: 同上
|
|
||||||
|
|
||||||
### email-worker
|
|
||||||
- **端点**: `GET /health`
|
|
||||||
- **端口**: 8084
|
|
||||||
- **返回**: 同上
|
|
||||||
|
|
||||||
## 指标列表
|
## 指标列表
|
||||||
|
|
||||||
### git-hook 指标 (`/metrics`, port 8083)
|
### git-hook 指标 (`GET :8083/metrics`)
|
||||||
|
|
||||||
| 指标名 | 类型 | 标签 | 说明 |
|
| 指标名 | 类型 | 标签 | 说明 |
|
||||||
|--------|------|------|------|
|
|--------|------|------|------|
|
||||||
@ -74,17 +59,17 @@ scrape_configs:
|
|||||||
| `hook_sync_branches_changed_total` | Counter | — | 同步时产生的分支变更总数 |
|
| `hook_sync_branches_changed_total` | Counter | — | 同步时产生的分支变更总数 |
|
||||||
| `hook_sync_tags_changed_total` | Counter | — | 同步时产生的标签变更总数 |
|
| `hook_sync_tags_changed_total` | Counter | — | 同步时产生的标签变更总数 |
|
||||||
|
|
||||||
**任务类型 (`task_type`)**:
|
**`task_type` 标签值**:
|
||||||
- `Sync` — 完整同步 (refs + commits + tags + LFS + fsck + gc + skills)
|
- `Sync` — 完整同步 (refs + commits + tags + LFS + fsck + gc + skills)
|
||||||
- `Fsck` — 仅校验仓库完整性
|
- `Fsck` — 仅校验仓库完整性
|
||||||
- `Gc` — 仅垃圾回收
|
- `Gc` — 仅垃圾回收
|
||||||
|
|
||||||
**使用示例**:
|
**PromQL 示例**:
|
||||||
```promql
|
```promql
|
||||||
# 任务成功率
|
# 任务成功率
|
||||||
hook_tasks_success_total / hook_tasks_total
|
hook_tasks_success_total / hook_tasks_total
|
||||||
|
|
||||||
# 失败率 (按类型)
|
# Sync 任务失败率
|
||||||
rate(hook_tasks_failed_total{task_type="Sync"}[5m])
|
rate(hook_tasks_failed_total{task_type="Sync"}[5m])
|
||||||
|
|
||||||
# 仓库锁定频率
|
# 仓库锁定频率
|
||||||
@ -93,36 +78,33 @@ rate(hook_tasks_locked_total[15m])
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### email-worker 指标 (`/metrics`, port 8084)
|
### email-worker 指标 (`GET :8084/metrics`)
|
||||||
|
|
||||||
| 指标名 | 类型 | 标签 | 说明 |
|
| 指标名 | 类型 | 说明 |
|
||||||
|--------|------|------|------|
|
|--------|------|------|
|
||||||
| `email_queued_total` | Counter | — | 写入 Redis Stream 的邮件总数 (生产端) |
|
| `email_queued_total` | Counter | 写入 Redis Stream 的邮件总数 (生产端) |
|
||||||
| `email_consumed_total` | Counter | — | 从队列消费的邮件总数 (每批次) |
|
| `email_consumed_total` | Counter | 从队列消费的邮件总数 |
|
||||||
| `email_batch_size` | Counter | — | 消费批次大小累计值 |
|
| `email_batch_size` | Counter | 消费批次大小累计值 |
|
||||||
| `email_validation_skipped_total` | Counter | — | 收件人地址校验失败被跳过的邮件数 |
|
| `email_validation_skipped_total` | Counter | 收件人地址校验失败被跳过的邮件数 |
|
||||||
| `email_build_errors_total` | Counter | — | 邮件消息构建失败的次数 |
|
| `email_build_errors_total` | Counter | 邮件消息构建失败的次数 |
|
||||||
| `email_send_attempts_total` | Counter | — | SMTP 发送尝试总次数 (含重试) |
|
| `email_send_attempts_total` | Counter | SMTP 发送尝试总次数 (含重试) |
|
||||||
| `email_sent_total` | Counter | — | 成功发送的邮件数 |
|
| `email_sent_total` | Counter | 成功发送的邮件数 |
|
||||||
| `email_send_failures_total` | Counter | — | 经 3 次重试后最终失败的邮件数 |
|
| `email_send_failures_total` | Counter | 经 3 次重试后最终失败的邮件数 |
|
||||||
|
|
||||||
**使用示例**:
|
**PromQL 示例**:
|
||||||
```promql
|
```promql
|
||||||
# 邮件发送成功率
|
# 邮件发送成功率
|
||||||
email_sent_total / (email_sent_total + email_send_failures_total)
|
email_sent_total / (email_sent_total + email_send_failures_total)
|
||||||
|
|
||||||
# 队列积压率
|
|
||||||
rate(email_consumed_total[5m]) / rate(email_queued_total[5m])
|
|
||||||
|
|
||||||
# 校验失败率
|
# 校验失败率
|
||||||
rate(email_validation_skipped_total[5m]) / rate(email_queued_total[5m])
|
rate(email_validation_skipped_total[5m]) / rate(email_queued_total[5m])
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### app / gitserver 指标 (`/metrics`)
|
### app 指标 (`GET :8080/metrics`)
|
||||||
|
|
||||||
由 `observability` crate 导出,包含以下通用指标:
|
由 `observability` crate 导出:
|
||||||
|
|
||||||
| 指标名 | 类型 | 说明 |
|
| 指标名 | 类型 | 说明 |
|
||||||
|--------|------|------|
|
|--------|------|------|
|
||||||
@ -132,19 +114,47 @@ rate(email_validation_skipped_total[5m]) / rate(email_queued_total[5m])
|
|||||||
| `ai_input_tokens_total` | Counter | 累计输入 token 数 |
|
| `ai_input_tokens_total` | Counter | 累计输入 token 数 |
|
||||||
| `ai_output_tokens_total` | Counter | 累计输出 token 数 |
|
| `ai_output_tokens_total` | Counter | 累计输出 token 数 |
|
||||||
| `ai_function_calls_total` | Counter | AI function/tool 调用次数 |
|
| `ai_function_calls_total` | Counter | AI function/tool 调用次数 |
|
||||||
| `http_requests_total` | Counter | HTTP 请求总数 |
|
| `http_requests_total` | Counter | HTTP 请求总数 (via `service="app"`) |
|
||||||
| `http_request_duration_ms_total` | Counter | HTTP 请求累计耗时 (ms) |
|
| `http_request_duration_ms_total` | Counter | HTTP 请求累计耗时 (ms) |
|
||||||
| `http_requests_by_status_class` | Gauge | 按状态码分类的请求数 (`2xx`, `4xx`, `5xx`) |
|
| `http_requests_by_status_class` | Gauge | 按状态码分类的请求数 (`2xx`, `4xx`, `5xx`) |
|
||||||
|
|
||||||
## Kubernetes 探针配置
|
## Kubernetes 探针配置
|
||||||
|
|
||||||
### gitserver
|
> 以下为 values.yaml 中实际生效的配置值。Helm 模板使用具名端口 `health` 引用容器端口。
|
||||||
|
|
||||||
|
### app
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet:
|
httpGet:
|
||||||
path: /health
|
path: /health
|
||||||
port: 8021
|
port: 8080
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
|
||||||
|
startupProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 0
|
||||||
|
periodSeconds: 10
|
||||||
|
failureThreshold: 30
|
||||||
|
```
|
||||||
|
|
||||||
|
### static
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8081
|
||||||
initialDelaySeconds: 5
|
initialDelaySeconds: 5
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
timeoutSeconds: 3
|
timeoutSeconds: 3
|
||||||
@ -153,7 +163,29 @@ livenessProbe:
|
|||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet:
|
httpGet:
|
||||||
path: /health
|
path: /health
|
||||||
port: 8021
|
port: 8081
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
timeoutSeconds: 3
|
||||||
|
failureThreshold: 3
|
||||||
|
```
|
||||||
|
|
||||||
|
### gitserver
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: health # 具名端口,映射到容器 8021
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 3
|
||||||
|
failureThreshold: 3
|
||||||
|
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: health
|
||||||
initialDelaySeconds: 5
|
initialDelaySeconds: 5
|
||||||
periodSeconds: 5
|
periodSeconds: 5
|
||||||
timeoutSeconds: 3
|
timeoutSeconds: 3
|
||||||
@ -204,7 +236,7 @@ readinessProbe:
|
|||||||
failureThreshold: 3
|
failureThreshold: 3
|
||||||
```
|
```
|
||||||
|
|
||||||
## 告警规则示例 (Alertmanager / Grafana)
|
## 告警规则示例
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# Git Hook Worker Down
|
# Git Hook Worker Down
|
||||||
@ -254,9 +286,13 @@ readinessProbe:
|
|||||||
|------|------|
|
|------|------|
|
||||||
| `GET /sitemap.xml` | sitemapindex,引用所有子 sitemap |
|
| `GET /sitemap.xml` | sitemapindex,引用所有子 sitemap |
|
||||||
| `GET /sidemap/static` | 固定页面 (首页、auth、营销页) |
|
| `GET /sidemap/static` | 固定页面 (首页、auth、营销页) |
|
||||||
| `GET /sidemap/users` | 公开用户页面 |
|
| `GET /sidemap/users` | 公开用户页面 (按用户名字母排序) |
|
||||||
| `GET /sidemap/projects` | 公开项目页面 |
|
| `GET /sidemap/projects` | 公开项目页面 (按项目名字母排序) |
|
||||||
| `GET /sidemap/repos` | 公开仓库页面 |
|
| `GET /sidemap/repos` | 公开仓库页面 (按仓库名字母排序) |
|
||||||
| `GET /robots.txt` | robots.txt,声明 Sitemap 位置 |
|
| `GET /robots.txt` | robots.txt,声明 Sitemap 位置 |
|
||||||
|
|
||||||
用户/项目/仓库 sitemap 数据通过 Redis 缓存 (8h TTL),`robots.txt` 中的 Sitemap URL 动态读取 `APP_DOMAIN_URL` 环境变量并强制使用 `https://` 前缀。
|
**缓存策略**:用户/项目/仓库 sitemap 数据通过 Redis 缓存,TTL 8 小时,访问不续期,过期自动重新生成。缓存在 key `sidemap:{users,projects,repos}` 下。
|
||||||
|
|
||||||
|
**域名**:`robots.txt` 中的 Sitemap URL 动态读取 `APP_DOMAIN_URL` 环境变量并强制使用 `https://` 前缀。
|
||||||
|
|
||||||
|
**禁止爬取**:`/sidemap/` 子路径在 `robots.txt` 中未声明,由主站统一控制,仅 `sitemapindex` 对外可见。
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user