solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md

CI/CD Monitor Agent — Persistent Memory

• 07-23 verify 4510cea (App: chuẩn-hóa mã dự-án normalize + PE attach-gate FE both-apps) [em-main spawn, return CLEAN]: VERDICT PASS 5/5 — Run #407 (id=521) success ~5m36s (17:11:54→17:17:30+07); gate 532 EXACT từ log (45 Domain + 487 Infra, 0 fail/skip) — +4 vs prev run #406/8b6df0a (483I) = 4 ProjectCodeNormalizationTests (delta-proof: dotnet summary KHÔNG in tên-test riêng → dùng delta prev-run + build-success + all-pass). NO-Mig: git diff 8b6df0a..4510cea -- *Migrations* EMPTY + prod history-top AddPeApprovedBudgetSnapshot (Mig 67, từ S133 KHÔNG phải push này) = repo-HEAD unchanged + sys.tables 89 — code-only no-drift by-construction. Bundle JS rotate cả 2 app / CSS FROZEN cả 2 (đúng dự-đoán — PeDetailTabs.tsx ×2 = pure button-gating logic, no new Tailwind utility): admin js DGPE6Nkb→B2j1LQ-v css CBzUlgOA frozen · eoffice js DlTwAgJj→C3wbJegk css BHsBUA8e frozen; real-size 1.660.902/1.567.232B, LM 17:16:01/17:16:54+07 in-window; SPA-trap control fake-hash index-ZZfakehash0.js=900B ct=text/html (discriminator OK). smoke 4×200 (health/live+ready, admin, eoffice). Baseline shift: Infra 464(S118)→483(run406)→487, Mig-latest = Mig 67 ("Mig 66" essentials-line STALE — updated). Pre-deploy snapshot captured WHILE running (đúng bài #242 verify-post-success). Tag [run-407, pass-5-5, js-rotate-css-frozen, delta-proof-+4, code-only-no-mig]

• (S134 #399 6664a61 PE lũy-kế tạm-tính (0-mig) GO 5/5 — gate 528 [45+483, +8 PePendingAccumulation]; js+css rotate ×2 admin BVCukFP5/CBzUlgOA eoffice DoxyEOxh/BHsBUA8e; byte-verify #77 urllib-BYTES; 🔑 ct=text/html discriminator SẠCH HƠN size-919B [shell size drift per-build]; body.count(marker.encode()) bulletproof cp1252 · verbatim → archive/2026-07.md via _INDEX)

• (S133 #398 d4fbced PE budget-freeze + Mig 67 AddPeApprovedBudgetSnapshot APPLIED-PROD GO 5/5 — gate 520 [45+475]; 11-col ApprovedBudget* + backfill 18/18 DaDuyet + 89-table; js+CSS rotate ×2 [badge utility-class → CSS content-hash đổi, benign #69]; python text=True crash cp1252 → text=False bytes+json.loads · verbatim → archive/2026-07.md via _INDEX)

Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 archive/ on-demand · L3 RAG search_memory just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → git d2f52ba + archive/2026-05-{runs,q2,q3,q4}.md + archive/2026-06.md + archive/2026-07.md. 🗺️ Lookup map: archive/_INDEX.md — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + 2026-0{5,6}.gist.md theo nhu cầu. See 🔄 Curate-trigger log for archive-move history.

---

🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.

---

⚠️ cicd-lane return-garble tracker (#53)

cicd-spawn đôi khi trả memory-curation slice ("Compressing S112/S111") thay verdict (×5/6 qua S114; cumulative ×18 all-agents → feedback_agent_return_garble_recover, root-cause = curate-at-return). Recover: em-main-solo curl-verify. S115 (#385)+S116 (#386) cicd-lane CLEAN-return no-garble ×2; S117 (#387) garble→SendMessage-resume (em-main curl cross-verify khớp, verdict disk-true) — return-mechanics garbled NHƯNG verdict-substance OK. S118 (#388+#389) cicd-lane garble ×2 → em-main-solo curl+sqlcmd recover (#388 durability tra.bui=20 seed-race "9 upgraded" · #389 Reports.Read→Admin-only+401), both verdict-substance disk-verified OK; curate-at-return root-cause persists (harvest-curator S118-end bắt nhãn "CLEAN-return" tự-dán sai → habitual pattern).

---

🚨 Recurring CI/CD bug patterns (catch priority)

• #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded .gitea/workflows/deploy.yml. KHÔNG revert.
• #40 npm cache tsc not found — build_fe_admin fail post cache: npm. DISABLED rolled back a21790d. KHÔNG re-enable.
• #41 paths-ignore docs-only skip — code commit không trigger CI? Check git diff --name-only HEAD~1 HEAD vs paths-ignore 7-entry từ D10 S111 8aa3869 (#495): ['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']. Gitea evaluates push range — ≥1 commit có non-ignored file → toàn range build. ⚠️ AS-6 BASELINE SHIFT: từ #495, memory-push chứa memory-budget.json/agent-memory-file/run-trace HẾT trigger CI (pattern #490/#492 DEAD).
• #25 IIS WebSocket — notification-hub/negotiate 401/404 prod. Fix: WebSocket module enable web.config site api (skill iis-deploy-runbook).
• #48 SQLite tie-break — OrderByDescending(CreatedAt).First() pick wrong khi 2+ .Add() cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy.
• Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).
  • 🔴 #69 (S72 #312) — FE bundle hash NON-DETERMINISTIC per rebuild. deploy.yml Remove-Item fe-*\* -Exclude web.config+Copy-Item dist\* runs UNCONDITIONALLY every run (path-filter gates whole-workflow, NOT per-step). ⟹ "BE/governance ⟹ frozen" can be FALSE; rotation possible every deploy. To detect REAL FE change, diff fe-*/src in commit/range, NOT hash-delta.
  • ⚠️ SPA-fallback 200 trap (S72): server rewrites /*→/index.html so GET /assets/index-<ANYTHING>.js returns 200 even for fake hash (control ZZdoesnotexist0.js→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html <script src>, GET that exact path + check size_download LARGE (real ~1.6MB vs fake ~919b) + Last-Modified in deploy window + 2nd-fetch byte-stable (no mid-deploy transient).
  • 🟢 #69-REFINE (S83 #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change. #69 "rotates EVERY deploy" NOT absolute: identical Vite source ⟹ byte-identical output ⟹ same content-hash. MOST RELIABLE ship-proof = Last-Modified on real bundle IN deploy-window + real-size, NOT hash-delta (hash-delta false-NEG when src unchanged, false-POS risk per #69). 0-FE-change ⟹ hash EITHER same (#333/#485-487/#495/#385-S115) OR rotated (#312/#489) — both benign; ship-proof = fresh LM-in-window + real-size + marker.
• Migration drift prod vs repo — compare ls .../Persistence/Migrations/*.cs vs sqlcmd __EFMigrationsHistory. Fix: check Program.cs/DbInitializer app.MigrateDatabase() + app pool recycle. Mig-applied proof = __EFMigrationsHistory top==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.

---

📋 5-stage checklist (EVERY run)

• Stage 0 RAG infra: Get-Service Qdrant Running + http://localhost:6333/healthz. Collection proj_solution_erp.
• Stage 1 Push+filter: git log -1 --format='%H %s' + git log origin/main..HEAD empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
• Stage 2 Gitea poll (max 10 iter × 60s): API .../actions/tasks?limit=5 (NOT /runs 404). Match head_sha (NOT run_number — cancelled runs shift numbering). ⚠️ updated_at stale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter.
• Stage 3 Test gate: baseline 509 PASS (45 Domain + 464 Infra; S118 #389 c808b9a — 505+4 AuthorizePolicyRegressionTests ReportsController Reports.Read gate; Infra 460→464, Domain frozen 45; prior 505 [#388 c2135ba ProcurementMasterAccessSeed], 501, 495, 486, 458, 440, 421). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (tasks terminal=status:success, conclusion NOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK.
• Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=accessToken route /api/auth/login) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns 411 IIS Length-Required pre-auth NOT 404; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.
  • Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
  • Discovery: ASP.NET 10 record enum needs numeric input unless JsonStringEnumConverter (SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth \\\\SQLEXPRESS 4-backslash. INFRASTRUCTURE seed MUST run (NOT inside if(!demoSeedDisabled)); DEMO seed gated → gotcha #51.
• Stage 5 Report PASS/FAIL + evidence + MEMORY update.

---

⚠️ Anti-patterns (DO NOT)

1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

---

🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)

• Gitea: git.baocaogiaoduc.vn/vietreport-admin/solution-erp · workflow .gitea/workflows/deploy.yml · paths-ignore 7-entry (D10 S111 #495): ['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']. Anon API works (public repo) when GITEA_TOKEN absent.
• Prod: api/admin/eoffice .solutions.com.vn · SSH ssh vietreport-vps (Administrator, id_ed25519) · IIS phys paths: API C:\inetpub\solution-erp\api · admin \fe-admin · user \fe-user. DB .\SQLEXPRESS/SolutionErp/vrapp SQL-auth. Conn key = ConnectionStrings.Default (NOT DefaultConnection). DB pw = buKL3TGBkD0wDDbYVw65QeX9 (User Id=vrapp; ⚠️ Password field has NO vrapp/ prefix — S89 login-failed when prefix included; read verbatim from prod appsettings.Production.json ConnectionStrings.Default when $env:PROD_DB_PASSWORD empty). ⚠️ skill-doc path C:\inetpub\apps\SolutionErp\Api STALE.
  • SSH→PS quoting (S42): nested bash→ssh→powershell mangles $var/\" → use iconv UTF-16LE | base64 → powershell -EncodedCommand $B64; OR write ps1-file + powershell -File $(cygpath -w). sqlcmd over SSH direct -W -h -1 pw-inline. sys-catalog string-concat → COLLATE DATABASE_DEFAULT (collation conflict).
• Tests baseline: 532 PASS (45 Domain + 487 Infra · 0 fail/skip; run #407 4510cea 07-23; prior climb 528/520/509/505/501/495/486/458/440/421). Phase 9 UAT skip OK. 🔑 Concrete count via web-UI log https://git.baocaogiaoduc.vn/vietreport-admin/solution-erp/actions/runs/{run_number}/jobs/0/logs (~18KB, NO /api/v1/ prefix — api/v1 runs+tasks logs BOTH 404) → per-project Passed! - Failed: 0, Passed: N, Skipped: 0. Delta-proof new-tests = compare prev-run count (dotnet summary hides names). status=success alone ⟹ gate-passed (tests pre-deploy).
• Mig latest repo: Mig 67 AddPeApprovedBudgetSnapshot (20260717032812, S133 d4fbced — PE freeze-budget 11-col AddColumn + backfill 18/18, no-new-table; VERIFIED-APPLIED-PROD history-top + sys.tables 89). Prev Mig 66 AddPeApprovedPriceRange / 65 backfill-invariant / 62 ExpandSupplierFields (full table → docs/CLAUDE.md). Path src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/ (67 .cs excl Designer/Snapshot). Prod check sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC. ⚠️ sys.tables(is_ms_shipped=0)=89 — AddColumn/data-only/enum-additive don't change count; no-new-table commit ⟹ 89 correct, don't FAIL. NO-Mig proof = history NOT-advance + empty git diff <base>..<head> -- '*Migrations*'.
• Bundle hash live: ship-baseline = run #407 (07-23) admin js B2j1LQ-v/css CBzUlgOA · eoffice js C3wbJegk/css BHsBUA8e (real-size 1,660,902/1,567,232b; LM 17:16:01/17:16:54+07). PER #69 TWO-MODE = SNAPSHOT-per-run NOT stable-baseline (rotate on real-src, freeze byte-id on 0-src) — ALWAYS re-snapshot LIVE post-success; ship-proof = LM-in-window + real-size (vs 900b ct=text/html SPA-trap) + marker-grep, NOT hash-delta. js-rotate/css-frozen = FE-logic-no-style fingerprint (S134-refine: ct=text/html cleaner discriminator than 919b, shell size drifts per-build). ⚠️ id≠run_number (head_sha-keyed).
• Bearer: admin admin@solutions.com.vn/Admin@123456 (full) · UAT nv.test@solutions.com.vn/TestUser@123456 (Drafter CCM, gotcha #44 check).
• API Serilog log (S118 correction): real sink = C:\inetpub\solution-erp\api\logs\solution-erp-YYYYMMDD.log (lowercase logs, rolling daily). ⚠️ skill-runbook path …\Logs\log-YYYYMMDD.txt STALE → empty grep. MinimumLevel Default=Information (DbInitializer LogInformation seed/revoke lines DO persist) — Microsoft/System overridden Warning (no "Now listening" INFO). Startup seed-race audit pattern (durability deploys): grep this log for revoke-line THEN seeder-line same-second → "N added + M upgraded" proves seeder actively flipped rows revoke knocked down (0-upgraded = seeder no-op = FAIL flag).

🔑 Critical config (flag commit nếu tái xuất)

Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)

---

🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)

Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.

📊 Run stats baseline

BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate. Recent runs all ~4m40-5m.

---

📅 Recent runs (compressed — verbatim → archive via _INDEX.md)

• (S128 #395 289ba96 + S129 #396 295c70c + S132 #397 72f68d1 GOVERNANCE/SCRIPTS/TOOLING GO — gate 509 [45+464, 0 fail/skip]; smoke 4×200; bundle benign #69 [S128 FULL-ROTATE from S117 → yCkX-2Bp/DKSD4p9k; S129+S132 FROZEN byte-id 0-fe-src] ship-proof = LM-in-window + real-size ≠919b NOT hash-delta; NO-BE/NO-Mig [Mig 66/89-table]; 🔑 AS-6 #41: soi TỪNG file vs 7-glob — .ps1 thoát scripts/**.md glob + .claude/rag.json / .session-counter.json thoát ⟹ non-.md governance VẪN trigger CI; conclusion:None dù success → trust status [tái-xác-nhận S126/S128/S129/S132] · verbatim → archive/2026-07.md)
• (S118 #389 c808b9a SECURITY-AUTHZ ReportsController→Reports.Read GO 5m34s [gate 509=45+464 (+4 authz-reflection vs 505); DB Reports.Read→Admin-ONLY sqlcmd (Procurement CanRead=0 → non-admin 403 by-construction); no-token 401 class-[Authorize] intact; NO-FE/NO-Mig; cicd-lane GARBLED #53 → em-main curl+sqlcmd recovered, verdict disk-TRUE] · +S117 #387 316a82f PE duyệt-khoảng Min–Max + Mig 66 AddPeApprovedPriceRange APPLIED-PROD [gate 501; js-rotate×2 css-frozen; byte-verify #77 4-str PRESENT; Mig66 COL_LENGTH=9+sys.tables 89; 🔑 pre-deploy snapshot BEFORE poll-success] · verbatim → archive/2026-07.md via _INDEX)
• (S111-S116 #383-#386/#495/#496 PE multi-NCC per-hạng-mục (Mig 65 backfill) + nhập-giá-ÂM + submit-guard + Supplier-import Phase-B/v2 (Mig63 AddSupplierImportSourceFields/Mig64 AddSupplierPublishState APPLIED-PROD, IsPublic 37/37) + H22-WAL-adopt (deploy.yml self-change +2 paths-ignore) + GOVERNANCE GO [gate 458→495; js-rotate/css-frozen; 🔑 AS-6 baseline-shift memory-push HẾT trigger; 411-refine bodyless-POST→411 → re-probe-with-body 401/415≠404; #69-REFINE 0-src FROZEN-proof; byte-verify #77 str-PRESENT] · verbatim → archive/2026-07.md via _INDEX)
• (S98-S110 + S126 #485-#494/#394 GOVERNANCE/MEMORY/TOOLING-push FROZEN-streak sau ROTATE #489 (#69 TWO-MODE; ship-proof = LM-in-window + real-size, hash-delta phụ); gate 440→509; Mig 62 + 89-table frozen · 🔑 #494 LM-newer-than-verified-window = missed-run detector; bare /health=404 UNMAPPED (đừng FAIL — thật /health/live+/ready) · verbatim → archive/2026-07.md via _INDEX)
• (PASS-pattern library S69b→S104 #307-#494 all GO — 🔑 reusable verify-recipes: SCHEMA-ADD+DATA-UN-CRAM [#488 Supplier 9→27 Mig62: AddColumn×18-exact + 89-invariant + un-cram 4-NCC + DTO-live] · DATA-FIX-BACKFILL [#484 Mig61: Mig-top==HEAD + target-rows-flip-EXACT + control-untouched + derived-API-FLIP-live] · DATA-SEED-PASS [#477: pre/post-DB-row-delta + DLL-mtime + w3wp-recycle]; #482 Mig60 EF-ternary-SQL 200≠500; #472 PROD Identity NON-prefix Roles; #470 no-resurrect DB-count-diff; Mig54-62 applied-prod; #348 411→401 · verbatim → git + 2026-06/07.md)
• Older (S83 #325 ← S29 #232) → git d2f52ba + archive/2026-06.md (incl #291 06-16 FAIL [gotcha #65]); pre-S38 → archive/2026-05-*.

---

🔄 Curate trigger

• ~30KB → archive recent runs → L2 archive/<period>.md (byte-exact append) + _INDEX.md substring pointer. Dup failure patterns → merge. Stale >3mo → remove.

• Last curate: 2026-07-14 S118 (self #389-run: S117 #387 verbatim→archive/2026-07.md moved-not-cut byte-id, baselines 505→509; prev S115 self 21.6→<17KB): #384/#383/#496/#495 (S114/S113/S112/S111) verbatim → archive/2026-07.md + _INDEX +4 pointers → S114 compact-stub + S111-113 merged-stub; top S114 garble-block → 1-line tracker.
• Prev: S111 07-12 self 20.5→17 (#490-#494→archive) · S107 07-10 self 20.4→16.9 (#489/#488→archive) · S104 07-06 self 20.5→17.9 · S98 07-02 em-main 19.9→17.06 (2026-07.md NEW) · S80 06-20 em-main 38.8→16 · S70 06-17 Harness-9 65.2→23.2 · earlier S66 · S40.