feat(W4): daily full reconciliation — user/role drift repair + ReconcileLogs audit
- w4-body.js: ONE-DIRECTIONAL sync (NocoDB status = SOT, never reverse);
drift matrix: missing user/role (provision w/ username lookup to avoid
duplicates), dangling refs (adopt/recreate), disabled<->status, canonical
base_path '/' + permission 16384, scopes vs calculateEffectiveGrants,
username drift detect+log_only; verify-after-repair on every change;
password never pushed on repair (方案 C); dry-run via ?dryRun=true
- gen_w4.js: daily 03:00 cron + webhook alist-w4-reconcile; API-import-safe
(UUID node ids, no pinData, webhookId)
- ReconcileLogs table mktsb7le6fqr2u1 (add_reconcilelogs_table.py, idempotent,
v2 API); FK column nc_jitu___Customers_id for bt-link writes (v2 ignores
link column itself)
- test_w4_reconcile.js: 34 assertions / 12 mock worlds, ALL PASSED
- live-verified against production: dry-run (3 drifts + 1 error found),
real repair (3 repaired, verify all passed, 4 log rows), idempotent rerun
- NocoDB 2026.08.x pitfalls documented (v1 col endpoints gone, row delete
via body [{Id}]); n8n API import/activation pitfalls (activeVersionId)
- docs: W4-DESIGN.md new; DATABASE_SCHEMA 8 tables; N8N_WORKFLOWS, SETUP,
PROJECT_STATE updated
This commit is contained in:
@@ -0,0 +1,78 @@
|
||||
# W4 — Daily Full Reconciliation / 全量对账 + drift 修复 + 对账日志
|
||||
|
||||
> 2026-08-23 定稿并实测(本地单元测试 + 直连生产 dry-run + 真实修复跑 + 幂等复跑)。
|
||||
> 与 W3 的差异:W3 每 5 分钟只对账 **scopes**(事件对账的补网);
|
||||
> W4 每日全量对账 **用户/角色/绑定的完整状态**,修复 W3 不管的 drift,并落 ReconcileLogs 审计。
|
||||
|
||||
## 触发
|
||||
|
||||
- **Schedule Trigger**:每日 03:00(cron `0 3 * * *`)。
|
||||
- **Webhook**(`POST /webhook/alist-w4-reconcile`)手动触发;`?dryRun=true` 或 env `W4_DRY_RUN=true` = 只出计划,零写入。
|
||||
|
||||
## 单向同步铁律(用户定稿)
|
||||
|
||||
- **NocoDB `Customers.status` 是 business source of truth;AList enabled/disabled 只是 enforcement state。**
|
||||
- W4 **永不**反向修改 NocoDB status。管理员在 AList 手工 disable 用户 → W4 按 NocoDB status 修回;绝不反过来。
|
||||
- 密码**不随 repair 推送**(方案 C:NocoDB plaintext 可能旧,推送会把客户自改的 AList 密码覆盖掉)。仅 provision 新建用户时用 NocoDB 密码。
|
||||
|
||||
## Drift 矩阵(自动修复)
|
||||
|
||||
| 场景 | 期望状态 | 处理 |
|
||||
|---|---|---|
|
||||
| **active 客户缺 user/role** | 存在 + enabled + canonical 配置 | 自动 provision:**先按 username 查 AList**(避免 NocoDB 丢 id 时建重)→ 没有才 create(role → user 顺序,空 role 会被 AList 拒) |
|
||||
| **alist_user_id / alist_role_id 悬空**(指向已删记录) | — | username / role name 反查 → 找到即 **adopt**(回写 bookkeeping id,不新建)→ 找不到才 recreate → 重新 sync |
|
||||
| **非 active(pending/suspended/expired/deleted)** | user disabled + scopes 空 | 有 user → disable + 清空 scopes;**pending 从未 provision → 保持无 AList user,不自动建**(纯 NocoDB 标签) |
|
||||
| **user.disabled 漂移** | 与 status 一致 | 按 status 单向修(active→enable,非 active→disable) |
|
||||
| **base_path / permission 漂移** | `/` + 16384 | 强制修回(= W1 已验证的 canonical 配置,不重新发明) |
|
||||
| **role binding 漂移**(user 没绑 role) | user.role = [专属 role id] | rebind |
|
||||
| **scopes 漂移 vs desired** | = calculateEffectiveGrants | 整体重写 role `permission_scopes`(与 W3 同算法) |
|
||||
| **username drift**(NocoDB 名 ≠ AList 名) | — | **detect + log_only**(不假设 rename API,不做 destructive migration) |
|
||||
| **客户行缺 username** | — | reconcile_error 记日志 + summary.errors |
|
||||
|
||||
## ReconcileLogs 表(新建,2026-08-23)
|
||||
|
||||
- 表 id `mktsb7le6fqr2u1`;列:`run_id` / `customer`(bt→Customers) / `drift_type` / `details` / `action` / `result` / `timestamp`
|
||||
- **只记 drift / repair / error 行**——健康客户不产生任何行(不为每天的全量跑刷日志)。
|
||||
- drift_type 选项:user_missing / role_missing / dangling_user_ref / dangling_role_ref /
|
||||
user_disabled_drift / scopes_drift / base_path_drift / permission_drift /
|
||||
role_binding_drift / username_drift / verify_failed / reconcile_error
|
||||
- result 选项:repaired / log_only / failed / skipped
|
||||
- ⚠ **写 customer 链接必须用 FK 列 `nc_jitu___Customers_id`**(v2 records API 写 link 列本身:标量/数组/对象全被忽略,实测确认),env `NOCODB_FK_RECONCILELOG_CUSTOMER` 可覆盖。
|
||||
|
||||
## 修复后必须 verify
|
||||
|
||||
每个 customer 修复后 **GET 回读 AList user/role 实测状态**,与期望比对;
|
||||
不通过 → 记 `verify_failed` 行(result=failed,不计入 repaired)。
|
||||
dry-run 不 verify(没写入可验)。
|
||||
|
||||
## 输出 summary
|
||||
|
||||
```
|
||||
{ runId, dryRun, at, customersScanned, driftsDetected, repaired, skipped,
|
||||
errors: [{customerId, username, error}], details: [{customerId, ..., drifts[]}],
|
||||
logRowsWritten, plannedLogRows (dry-run), logWriteSkipped (env 缺失时) }
|
||||
```
|
||||
|
||||
## 与 W1/W2/W3 的交互
|
||||
|
||||
- 写回 Customers 只有 `alist_user_id` / `alist_role_id`(adopt 时)→ W1 的 old/new 比对视为 echo → 短路,无 loop。
|
||||
- 不写 `sync_status` / `sync_required` / `last_sync`。
|
||||
- W4 不碰 AccessGrants(保持原定位:手工人为授权入口,源 b)。
|
||||
- AccessGrants/CustomerProducts/Customers 无 webhook 指向 W4,写 ReconcileLogs 无 loop。
|
||||
|
||||
## 本地实跑(绕过 n8n,直连生产)
|
||||
|
||||
`NOCODB_TOKEN=... ALIST_ADMIN_PASSWORD=... node scripts/live_dry_run_w4.js`(dry-run)
|
||||
与 `scripts/run_real_w4.js`(本机脚本版 run_real_w4.js 在 hermes scripts/,凭据走 env)。
|
||||
2026-08-23 实测记录:
|
||||
1. dry-run:5 客户,3 drift(TEST-Alice: dangling_role_ref + base_path_drift + role_binding_drift),1 error(Id 7 缺 username),3 skip,零写入 ✅
|
||||
2. 真实跑:3 repaired + 4 行 ReconcileLogs(FK 正确),verify 全过 ✅
|
||||
3. 复跑:0 drift / 0 repair / 4 skip / 仅 Id-7 error 照记 → 幂等 ✅
|
||||
|
||||
## 坑(已沉淀,勿再踩)
|
||||
|
||||
- n8n API 导入校验严格:**节点 id 必须 UUID、顶层不能有 `pinData`**,否则 400 "additional properties"。
|
||||
- n8n webhook 节点必须带 `webhookId`,否则生产 URL 永不注册(404 "not registered")。
|
||||
- n8n 1.123 API 激活返回 `active:true` 但运行时激活可能被权限拒绝(日志 "attempted to activate a workflow without permissions")→ 激活状态在 DB 是 `workflow_entity.activeVersionId`(不是 active 列),boot 时加载 active + activeVersionId 的工作流注册 webhook/cron。
|
||||
- NocoDB 2026.08.x:v1 column GET/PATCH 端点已删除;列 id 从 `GET /api/v2/meta/tables/{id}` 拿;v2 的列 create/PATCH 响应返回的是**表对象**不是列。
|
||||
- NocoDB v2 行删除:`DELETE /records` + body `[{Id: N}, ...]`(路径 /records/{id} 和 where 参数都不行)。
|
||||
Reference in New Issue
Block a user