solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md

CI/CD Monitor Agent — Persistent Memory

Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 archive/ on-demand · L3 RAG search_memory just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → git d2f52ba + archive/2026-05-{runs,q2,q3,q4}.md + archive/2026-06.md + archive/2026-07.md. 🗺️ Lookup map: archive/_INDEX.md — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + 2026-0{5,6}.gist.md theo nhu cầu. S80 curate: moved 3 huge run-entries (#330/#318/#325) → archive. S98 curate: moved 2 huge S97 entries → archive/2026-07.md.

---

🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.

---

🚨 Recurring CI/CD bug patterns (catch priority)

• #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded .gitea/workflows/deploy.yml. KHÔNG revert.
• #40 npm cache tsc not found — build_fe_admin fail post cache: npm. DISABLED rolled back a21790d. KHÔNG re-enable.
• #41 paths-ignore docs-only skip — code commit không trigger CI? Check git diff --name-only HEAD~1 HEAD vs paths-ignore: ['docs/**','**/*.md','.claude/skills/**']. Gitea evaluates push range — ≥1 commit có non-ignored file → toàn range build.
• #25 IIS WebSocket — notification-hub/negotiate 401/404 prod. Fix: WebSocket module enable web.config site api (skill iis-deploy-runbook).
• #48 SQLite tie-break — OrderByDescending(CreatedAt).First() pick wrong khi 2+ .Add() cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy.
• Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).
  • 🔴 #69 (S72 Run #312) — FE bundle hash NON-DETERMINISTIC per rebuild. deploy.yml Remove-Item fe-*\* -Exclude web.config + Copy-Item dist\* runs UNCONDITIONALLY every run (path-filter gates whole-workflow trigger, NOT per-step). Identical FE source ⟹ DIFFERENT hash each deploy (Vite/rolldown non-reproducible chunk-id). PROVEN: governance-only 18fced6 (0 files in fe-*/src) rotated BOTH bundles. ⟹ "BE-only/governance ⟹ bundle frozen" is FALSE; rotation EXPECTED every deploy. To detect REAL FE change, diff fe-*/src in commit/range, NOT hash delta.
  • ⚠️ SPA-fallback 200 trap (S72): server rewrites /*→/index.html so GET /assets/index-<ANYTHING>.js returns 200 even for fake hash (control ZZdoesnotexist0.js→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html <script src>, GET that exact path + check size_download LARGE (real ~1.6MB vs fake ~919b) + Last-Modified in deploy window + 2nd-fetch byte-stable (no mid-deploy transient).
  • 🟢 #69-REFINE (S83 Run #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change. #69 said "rotates EVERY deploy" — NOT absolute. Run #333 (BE+tests+governance, ZERO fe-*/src diff) shipped FE (Last-Modified 17:10:35/17:11:27 BOTH in run-window 17:06→17:11) but hash UNCHANGED (admin CsJetgZH/user BVS0ApIm = #330 baseline). ⟹ identical Vite source ⟹ byte-identical output ⟹ same content-hash. Rolldown chunk-id apparently MORE deterministic than #69 feared (or only rotates on real src delta). MOST RELIABLE ship-proof = Last-Modified header on the real bundle file IN deploy-window, NOT hash-delta (hash-delta has false-NEGATIVE when src unchanged + false-POSITIVE risk noted in #69). Always cross-ref git diff fe-*/src scope: 0-FE-change ⟹ expect SAME hash + fresh Last-Modified.
• Migration drift prod vs repo — compare ls .../Persistence/Migrations/*.cs vs sqlcmd __EFMigrationsHistory. Fix: check Program.cs/DbInitializer app.MigrateDatabase() + app pool recycle. Mig-applied proof = __EFMigrationsHistory top==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.

---

📋 5-stage checklist (EVERY run)

• Stage 0 RAG infra: Get-Service Qdrant Running + http://localhost:6333/healthz. Collection proj_solution_erp.
• Stage 1 Push+filter: git log -1 --format='%H %s' + git log origin/main..HEAD empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
• Stage 2 Gitea poll (max 10 iter × 60s): API .../actions/tasks?limit=5 (NOT /runs 404). Match head_sha (NOT run_number — cancelled runs shift numbering). ⚠️ updated_at stale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter.
• Stage 3 Test gate: baseline 440 PASS (45 Domain + 395 Infra; S96 #480 0c2f9f6 — 436 + new PeSuggestedPriceSetterAuthzTests [price-lock-after-submit authz]; prior S96 #479 436, S96/S95 434, S90 #465 421). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (tasks terminal=status:success, conclusion NOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK.
• Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=accessToken route /api/auth/login) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns 411 IIS Length-Required pre-auth NOT 404; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.
  • Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
  • Discovery: ASP.NET 10 record enum needs numeric input unless JsonStringEnumConverter (SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth \\\\SQLEXPRESS 4-backslash. INFRASTRUCTURE seed MUST run (NOT inside if(!demoSeedDisabled)); DEMO seed gated → gotcha #51.
• Stage 5 Report PASS/FAIL + evidence + MEMORY update.

---

⚠️ Anti-patterns (DO NOT)

1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

---

🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)

• Gitea: git.baocaogiaoduc.vn/vietreport-admin/solution-erp · workflow .gitea/workflows/deploy.yml · paths-ignore ['docs/**','**/*.md','.claude/skills/**']. Anon API works (public repo) when GITEA_TOKEN absent.
• Prod: api/admin/eoffice .solutions.com.vn · SSH ssh vietreport-vps (Administrator, id_ed25519) · IIS phys paths: API C:\inetpub\solution-erp\api · admin \fe-admin · user \fe-user. DB .\SQLEXPRESS/SolutionErp/vrapp SQL-auth. Conn key = ConnectionStrings.Default (NOT DefaultConnection). DB pw = buKL3TGBkD0wDDbYVw65QeX9 (User Id=vrapp; ⚠️ Password field has NO vrapp/ prefix — S89 login-failed when prefix included; read verbatim from prod appsettings.Production.json ConnectionStrings.Default when $env:PROD_DB_PASSWORD empty). ⚠️ skill-doc path C:\inetpub\apps\SolutionErp\Api STALE.
  • SSH→PS quoting (S42): nested bash→ssh→powershell mangles $var/\" → use iconv UTF-16LE | base64 → powershell -EncodedCommand $B64; OR write ps1-file + powershell -File $(cygpath -w). sqlcmd over SSH direct -W -h -1 pw-inline. sys-catalog string-concat → COLLATE DATABASE_DEFAULT (collation conflict).
• Tests baseline: 440 PASS (45 Domain + 395 Infra · 0 fail/skip; S96 Run #480 sha 0c2f9f6 — 436 + new PeSuggestedPriceSetterAuthzTests; prior S96 #479 436, S96/S95 434, S90 #465 421). Phase 9 UAT skip per chunk OK.
• Mig latest repo: Mig 61 20260701142752_BackfillEndedByLevelFinalizeFromChangelog (S97 #484; data-only NO-schema, Sql() UPDATE EndedByLevelFinalize=1 cho phiếu có finalize-marker ở PurchaseEvaluationChangelogs — fix Mig 60 soi NHẦM bảng — VERIFIED-APPLIED-PROD history-HEAD + A/005/A/008 flipped 0→1, A/010 stayed 0). Prev Mig 60 AddPeEndedByLevelFinalize (S97 #482 AddColumn bit default 0). Path src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/. Prod check sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5. ⚠️ sys.tables (is_ms_shipped=0) = 89 (S97 LIVE-verified pre/post #482 — narrative/spec-88 STALE, 89=truth; AddColumn/data-only migs don't change count → commit touches no schema ⟹ 89 correct don't FAIL. Cross-ref COMMIT scope vs ambient count). enum-additive (e.g. ApprovalAttachment=5) = int-stored NO-mig → history NOT-advance + empty git diff -- '*Migrations*'.
• Bundle hash live (S100 #486 verified 2026-07-02): admin js mEGqlNWc/css Bbtesa6B · user js Dnrp4PU1/css BmuPz_Dy — FROZEN qua #485→#486 (0 fe-src diff ⟹ byte-identical, #69-REFINE ×3: #333/#485/#486) nhưng Last-Modified fresh mỗi deploy = ship-proof. Rotate cuối = post-#484 (từ #483 CeTA7Z82/CGaAyXen). ⚠️ id ≠ run_number; head_sha-keyed. ⚠️ PER #69: hash MAY rotate (real src-diff) OR frozen (0 fe-src diff) — SNAPSHOT per-run, NOT baseline; ALWAYS re-snapshot LIVE pre-deploy. Real FE ship = diff fe-*/src in range + real-size-vs-fake (1.64/1.54MB vs ~919b SPA-trap) + Last-Modified-in-window, NOT hash delta alone.
• Bearer: admin admin@solutions.com.vn/Admin@123456 (full) · UAT nv.test@solutions.com.vn/TestUser@123456 (Drafter CCM, gotcha #44 check).

🔑 Critical config (flag commit nếu tái xuất)

Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)

---

🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)

Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.

📊 Run stats baseline

BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate. Recent runs all ~4m40-5m.

---

📅 Recent runs (compressed — full verbatim → archive/2026-06.md via archive/_INDEX.md)

• 🟢 S100 #486 (rn 372) 5ee32c0 GO 5m17s (15:32:11→15:37:28) — GOVERNANCE-ONLY (19 file = 18 .md/docs ignored + scripts/governance-detectors.ps1 non-ignored → trigger #41; 5 docs-only commits trước [525a01a…513aa09] đúng 0-run): Mig 61 frozen history-top + sys.tables 89 · smoke 200×7 (health-ready/admin/eoffice + auth accessToken + PE-list + contracts + menus) · bundle hash FROZEN 2 app + Last-Modified in-window 15:36:08/15:37:01+07 + real-size 1.64/1.54MB vs fake-919b control = ship-proof. #69-REFINE ×3 (#333→#485→#486): 0 fe-src diff ⟹ hash frozen + fresh copy. GOVERNANCE-PASS stable ×3 (#476/#485/#486). 🔧 Poll: python time.sleep in-script chạy ĐƯỢC foreground (chỉ bash sleep builtin bị block) — hết cần busy-wait spin.
• (S98 #485 (rn 371) 8a249c5 GO 5m18s GOVERNANCE-ONLY [22 .md + memory-budget.json non-ignored → #41] — Mig61 frozen · hash FROZEN + LM-in-window 11:01:48/11:02:42 + real-size = ship-proof · #69-REFINE lần 2 · verbatim git c112be9)
• 🟢🟢 S96+S97 FINAL-SESSION-REVIEW (2026-07-01 end) — 11-commit day #478-#484 ALL-DEPLOYED-HEALTHY, verdict CLEAN (whole-session GO trên LIVE prod): CI-success ×7 + Mig61 history-top + backfill A/005=1/A/008=1/A/010=0 exact + sys.tables 88-excl/89-incl reconcile + health 4×200 + auth/PE-list/SignalR wired (#25 clear) + BE-dll+w3wp in-#484-window. ⚠️ bundle ROTATED post-#484: admin js mEGqlNWc/user Dnrp4PU1, CSS frozen — #69 CONFIRMED: rebuild rotate MỌI deploy kể cả data-fix; rotation ≠ ship-signal (real-FE-ship = diff fe-src + real-size + Last-Modified-in-window). 🔑 WHOLE-SESSION GO = every-commit-CI-success + Mig-top==HEAD + backfill-exact + control-untouched + 4×health-200 + auth+PE+SignalR-wired + BE-dll+w3wp-in-window + bundles-real-in-window. Full verbatim → archive/2026-07.md via _INDEX (curate S98).
• 🟢 S97 #484 (run_number 370) fb2dfa9 GO ~5m — DATA-FIX Mig 61 backfill EndedByLevelFinalize từ Changelogs (fix Mig 60 wrong-table, gotcha #79): sqlcmd A/005=1 + A/008=1 (0→1 = fix ăn) · A/010=0 control · Mig61 __EFMigrationsHistory top · endsBeforeCeo FLIP live A/005=True · smoke 200×3 + PE-list 200 · baseline 440. 🔑 DATA-FIX-BACKFILL PASS = CI-success + Mig-top==HEAD + sqlcmd-target-rows-flip-EXACT + control-untouched + derived-API-field-FLIP-live + smoke-200. Full verbatim → archive/2026-07.md via _INDEX (curate S98).
• (endsBeforeCeo-saga PE finalize UAT cluster anh Kiệt: S97 #482 8037fa0 CROSS-STACK + Mig60 AddPeEndedByLevelFinalize [config→RUNTIME · Mig-APPLIED col-bit-default-CONVERT(0) backfill=0 sys.tables=89-pre/post · 🔴 EF-TERNARY-SQL-Server-TRANSLATE-LIVE list-200-NOT-500 + detail-endsBeforeCeo-False-runtime · JS Cf0o8dx8/x8rppRAK→DR9q2VUy/Bgz8u4E5 CSS-frozen · #77 field-PRESENT] · S97-badge #483 9e12e80 FE-ONLY 8-file badge "✅ Đã được CEO duyệt" [JS→CeTA7Z82/CGaAyXen CSS-frozen · #77 new-str-BOTH+prev-additive · Mig60-frozen list-200] · S97 #481 1c8fb6a FE-ONLY 2-fix [badge không trình tới→Kết thúc tại {name}+opt-out→opt-in · JS→Cf0o8dx8/x8rppRAK CSS-frozen · #77 new-PRESENT/old-ABSENT] — all baseline-440 all PASS, full verbatim → git + 2026-06.md via _INDEX)
• (S96 #480 0c2f9f6 cross-stack PE 3-commit-range khóa-giá+mark-read+opt-out NO-mig baseline-440+new-PeSuggestedPriceSetterAuthzTests JS-rotate/CSS-frozen · S96 #479 36cbf8b PE 2-commit mark-read-stale-noti baseline-436 JS-rotate/CSS-frozen · all PASS Mig59-frozen · full verbatim → git + archive 2026-06.md via _INDEX)
• (S96 #477 ba4b8e0 Infra data-seed NCC DONGDUONG pre=3→post=4 total=21-EXACT · 🔑 DATA-SEED-PASS = pre/post-DB-row-EXACT-delta + DLL-mtime + w3wp-recycle + Mig-frozen + API-payload-contains-row · ⚠️ sqlcmd -W diacritic-LIKE false-NEG use _-wildcard; base64-EncodedCmd-FAIL→.ps1-File · S95 #476 GOVERNANCE-ONLY .ps1-triggers-CI 🔑 GOVERNANCE-PASS=CI-green+JS-rotate(#69)+3×200+Mig-frozen · S93 #473 PE tree-CODE · S92 #472 permission-revoke [⚠️ PROD Identity tables NON-prefixed Roles NOT AspNetRoles] · S91 #470 GO-LIVE 🔑 PATTERN no-resurrect = pre+post-DB-count-diff + dll-in-window + w3wp≥run-window [integrated-auth -E] · S90 #465 PE draft-private → all PASS, full verbatim → git + 2026-06.md via _INDEX)
• (S89 #348 1aa3bdb PE 1→N multi-winner endpoint-411→401 · S88-quad #347 73cce1f PE D1 re-key+CEO-notify · S88-bis #346 0d3c0cf FE-only js+css-BOTH-rotate · S88 #345 9b1ee0c Mig 59 AddCcmBudgetPeriodToPurchaseEvaluation PATCH-401+7-authz · S87 #344 f1b10e9 FULL-STACK + Mig 58 AddPeMultiWinnerAndLevelFinalize backfill-winners=13 · S86 #343 BE-only notify-exclude ship-proof=Infra.dll-mtime · S85 #342 DOCS+SQL-no-op INVERTED-PASS-FROZEN [#73 TZ+#69] · S83 #333 BE-only guard-V2 hash-FROZEN-shipped · S78 #330 enum-attach · S76 #318 Mig56-ProInitial · S74-bis #315 Mig55 · S73 #313 Mig54-401-not-404 · S69b #307 Office-seed-16/16 → all PASS, full verbatim → git + 2026-06.md via _INDEX)
• Older runs (S83 #325 FE-budget-indent ← S77 #329 banner ← S75 #301 ← S62 #286 ← … → S29 #232) full verbatim → git d2f52ba + archive/2026-06.md (incl #291 06-16 FAIL forensic [gotcha #65]); pre-S38 → archive/2026-05-{runs,q2,q3,q4}.md.

---

🔄 Curate trigger

• ~30KB → archive recent runs → L2 archive/<period>.md (byte-exact append) + _INDEX.md substring pointer. Dup failure patterns → merge. Stale >3mo → remove.

• Last curate: 2026-07-02 S98 (em-main, garble-pressure #53 structural — L1 19.9KB > self-target 17.1KB) (19.9→17.06KB): moved 2 huge S97 entries (S96+S97 FINAL-SESSION-REVIEW + #484 data-fix) content-exact → archive/2026-07.md (NEW) + _INDEX.md +2 substring pointers (count=1 verified; V3 reviewer PASS, byte-arithmetic đóng vòng 2858B). KEPT 🔑 pattern-keys trong 2 stub L1.
• Prev curate: 2026-06-20 S80 (em-main, archive-gate keep-floor-hit → manual) (38.8→~16KB): moved 3 huge run-entries #330/#318/#325 (lines 73/90/96 byte-exact via sed) → archive/2026-06.md (100.6→116.7KB) + _INDEX.md +3 substring pointers (verified count=1). KEPT foundation (bug-patterns/5-stage/10-point gold) + compressed essentials + run-stubs; UPDATED stale (130→354 test Stage-3 + 263→354 essentials, Mig 55→57, bundle→#330) + dropped verbose per-run bundle-history (S86/S85/S84 → archived run-entries).
• Prev curate: 2026-06-17 S70 Harness-9 (65.2→23.2KB dark-matter recovery, built _INDEX/gist gen:1). Earlier: S66 · S40.