# CI/CD Monitor Agent — Persistent Memory

> **Persistent diary cross-session.** Auto-injected first ~200 lines at spawn (L1 HOT).
> Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 `archive/` on-demand · L3 RAG `search_memory` just-in-time. Keep entry ≤ 1.5K chars (gotcha #53).
> Full verbatim run history pre-S40 → git `d2f52ba` + `archive/2026-05-{runs,q2,q3,q4}.md` + `archive/2026-06.md` + `archive/2026-07.md`. 🗺️ **Lookup map: `archive/_INDEX.md`** — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + `2026-0{5,6}.gist.md` theo nhu cầu. See 🔄 Curate-trigger log for archive-move history.

---

## 🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: `iis-deploy-runbook` + `dependency-audit-erp` + `ef-core-migration`. Spawn ~150K — trade-off catch fail tự động.

---

## ⚠️ cicd-lane return-garble tracker (#53)

cicd-spawn đôi khi trả memory-curation slice ("Compressing S112/S111") thay verdict (×5/6 qua S114; cumulative ×18 all-agents → [[feedback_agent_return_garble_recover]], root-cause = curate-at-return). Recover: em-main-solo curl-verify. **S115 (#385) cicd-lane CLEAN-return no-garble — verdict delivered OK.**

---

## 🚨 Recurring CI/CD bug patterns (catch priority)

- **#39 act_runner github.com TCP timeout** — run hang "Set up job" 21s. Log `dial tcp github.com:443 i/o timeout`. Fix: manual checkout bypass hardcoded `.gitea/workflows/deploy.yml`. KHÔNG revert.
- **#40 npm cache `tsc not found`** — `build_fe_admin` fail post `cache: npm`. DISABLED rolled back `a21790d`. KHÔNG re-enable.
- **#41 paths-ignore docs-only skip** — code commit không trigger CI? Check `git diff --name-only HEAD~1 HEAD` vs `paths-ignore` **7-entry từ D10 S111 `8aa3869` (#495): `['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']`**. Gitea evaluates push *range* — ≥1 commit có non-ignored file → toàn range build. ⚠️ **AS-6 BASELINE SHIFT: từ #495, memory-push chứa `memory-budget.json`/agent-memory-file/run-trace HẾT trigger CI (pattern #490/#492 DEAD).**
- **#25 IIS WebSocket** — `notification-hub/negotiate` 401/404 prod. Fix: WebSocket module enable `web.config` site api (skill `iis-deploy-runbook`).
- **#48 SQLite tie-break** — `OrderByDescending(CreatedAt).First()` pick wrong khi 2+ `.Add()` cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy.
- **Bundle hash unchanged = ship FAIL** — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (`curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'`), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).
  - **🔴 #69 (S72 #312) — FE bundle hash NON-DETERMINISTIC per rebuild.** `deploy.yml` `Remove-Item fe-*\* -Exclude web.config`+`Copy-Item dist\*` runs UNCONDITIONALLY every run (path-filter gates whole-workflow, NOT per-step). ⟹ "BE/governance ⟹ frozen" can be FALSE; rotation possible every deploy. **To detect REAL FE change, diff `fe-*/src` in commit/range, NOT hash-delta.**
  - **⚠️ SPA-fallback 200 trap (S72):** server rewrites `/*`→`/index.html` so GET `/assets/index-<ANYTHING>.js` returns **200** even for fake hash (control `ZZdoesnotexist0.js`→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html `<script src>`, GET that exact path + check `size_download` LARGE (real ~1.6MB vs fake ~919b) + `Last-Modified` in deploy window + 2nd-fetch byte-stable (no mid-deploy transient).
  - **🟢 #69-REFINE (S83 #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change.** #69 "rotates EVERY deploy" NOT absolute: identical Vite source ⟹ byte-identical output ⟹ same content-hash. **MOST RELIABLE ship-proof = `Last-Modified` on real bundle IN deploy-window + real-size, NOT hash-delta** (hash-delta false-NEG when src unchanged, false-POS risk per #69). 0-FE-change ⟹ hash EITHER same (#333/#485-487/#495/#385-S115) OR rotated (#312/#489) — both benign; ship-proof = fresh LM-in-window + real-size + marker.
- **Migration drift prod vs repo** — compare `ls .../Persistence/Migrations/*.cs` vs `sqlcmd __EFMigrationsHistory`. Fix: check `Program.cs`/DbInitializer `app.MigrateDatabase()` + app pool recycle. Mig-applied proof = `__EFMigrationsHistory` top==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.

---

## 📋 5-stage checklist (EVERY run)

- **Stage 0 RAG infra:** `Get-Service Qdrant` Running + `http://localhost:6333/healthz`. Collection `proj_solution_erp`.
- **Stage 1 Push+filter:** `git log -1 --format='%H %s'` + `git log origin/main..HEAD` empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
- **Stage 2 Gitea poll** (max 10 iter × 60s): API `.../actions/tasks?limit=5` (NOT `/runs` 404). Match `head_sha` (NOT run_number — cancelled runs shift numbering). ⚠️ `updated_at` stale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter.
- **Stage 3 Test gate:** baseline **486 PASS** (45 Domain + 441 Infra; S114 #384 `d436af2` — 477+9 PE multi-NCC tests [PeMultiWinner/PeSelectWinnerClear/PeListWinnerNamesProjection/CreateContractFromEvaluationMultiWinner +3 modified]; prior 477 [S113 #383], 458, 440, 436, 434, 421). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (`tasks` terminal=`status:success`, `conclusion` NOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK.
- **Stage 4 Post-deploy** (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=`accessToken` route `/api/auth/login`) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns **411 IIS Length-Required pre-auth NOT 404**; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.
  - **Stage 4.6 (S29 CRITICAL):** sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
  - Discovery: ASP.NET 10 record enum needs numeric input unless `JsonStringEnumConverter` (SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth `\\\\SQLEXPRESS` 4-backslash. INFRASTRUCTURE seed MUST run (NOT inside `if(!demoSeedDisabled)`); DEMO seed gated → gotcha #51.
- **Stage 5 Report** PASS/FAIL + evidence + MEMORY update.

---

## ⚠️ Anti-patterns (DO NOT)
1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

---

## 🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)

- **Gitea:** `git.baocaogiaoduc.vn/vietreport-admin/solution-erp` · workflow `.gitea/workflows/deploy.yml` · paths-ignore 7-entry (D10 S111 #495): `['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']`. Anon API works (public repo) when GITEA_TOKEN absent.
- **Prod:** api/admin/eoffice `.solutions.com.vn` · SSH `ssh vietreport-vps` (Administrator, id_ed25519) · IIS phys paths: API `C:\inetpub\solution-erp\api` · admin `\fe-admin` · user `\fe-user`. DB `.\SQLEXPRESS`/`SolutionErp`/`vrapp` SQL-auth. **Conn key = `ConnectionStrings.Default` (NOT `DefaultConnection`)**. DB pw = `buKL3TGBkD0wDDbYVw65QeX9` (User Id=`vrapp`; ⚠️ Password field has NO `vrapp/` prefix — S89 login-failed when prefix included; read verbatim from prod `appsettings.Production.json` ConnectionStrings.Default when `$env:PROD_DB_PASSWORD` empty). ⚠️ skill-doc path `C:\inetpub\apps\SolutionErp\Api` STALE.
  - **SSH→PS quoting (S42):** nested bash→ssh→powershell mangles `$var`/`\"` → use `iconv UTF-16LE | base64` → `powershell -EncodedCommand $B64`; OR write ps1-file + `powershell -File $(cygpath -w)`. sqlcmd over SSH direct `-W -h -1` pw-inline. sys-catalog string-concat → `COLLATE DATABASE_DEFAULT` (collation conflict).
- **Tests baseline:** **486 PASS** (45 Domain + 441 Infra · 0 fail/skip; S114 #384 `d436af2`; prior 477/458/440/436/434/421). Phase 9 UAT skip OK. 🔑 **Concrete count via anon log `.../actions/runs/{run_number}/jobs/0/logs` (~18KB) → per-project `Passed! - Failed: 0, Passed: N, Skipped: 0` (NOT just status=success; `/api/v1/.../tasks/{id}/logs` variant 404s — use web-UI runs path).**
- **Mig latest repo: Mig 65 `BackfillPeQuoteIsSelectedAndWinnerInvariant`** (S114 `d436af2`; DATA-ONLY no-schema 2-dir backfill, INVARIANT `Supplier.IsWinner ⟺ Any(Quote.IsSelected)`; Down=no-op, NO ModelSnapshot change [2 mig-file = correct data-only], SQL-Server-only [SQLite EnsureCreated skips]; VERIFIED-APPLIED-PROD history-top + violations=0 + sys.tables=89). Prev Mig 64 `AddSupplierPublishState` / 63 / 62 `ExpandSupplierFields` (see docs/CLAUDE.md migration table). Path `src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/`. Prod check `sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC`. ⚠️ **`sys.tables`(is_ms_shipped=0)=89** — AddColumn/data-only don't change count; commit touching no new table ⟹ 89 correct, don't FAIL. enum-additive = int NO-mig (history NOT-advance + empty `git diff -- '*Migrations*'`).
- **Bundle hash live:** current ship-baseline = S114/S115 (admin js `D51OYyGV`/css `CfunQHbd` · user js `BVdssm5S`/css `DGqqHQXS`; size 1,655,105/1,561,453b) — see S114/S115 recent-run entries for LM+markers. ⚠️ id≠run_number (head_sha-keyed). PER #69 TWO-MODE hash is SNAPSHOT-per-run NOT stable-baseline (rotate on real-src OR benign-0-src; freeze on 0-src) — ALWAYS re-snapshot LIVE post-success; ship-proof = LM-in-window + real-size (vs 919b SPA-trap) + marker-grep, NOT hash-delta. js-rotate/css-frozen = FE-logic-no-style fingerprint.
- **Bearer:** admin `admin@solutions.com.vn/Admin@123456` (full) · UAT `nv.test@solutions.com.vn/TestUser@123456` (Drafter CCM, gotcha #44 check).

## 🔑 Critical config (flag commit nếu tái xuất)
Node CI `20.x` (`feedback_node_cicd`) · MediatR `12.4.1` (gotcha #1, flag `Version="14`) · Swashbuckle `6.9.0` (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag `cache: npm`)

---

## 🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)
Cross-ref `feedback_per_nv_permission_scope`. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF `HasDefaultValue(false)` · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 **Lookup discrimination** (`FirstOrDefault` ADD `ApproverUserId==actorId` + admin fallback) · 10 **Controller body record count == Command record count**. Bug latency 2-3 days prod silent khi miss 9-10. Scan `grep -n "FirstOrDefault.*Order.*==" *.cs` after OR-of-N refactor.

## 📊 Run stats baseline
BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · **total ~3min code / 0s docs-only**. >5min → escalate. Recent runs all ~4m40-5m.

---

## 📅 Recent runs (compressed — verbatim → archive via `_INDEX.md`)

- **🟢 S115 #385 (rn 385, id 499) `81b59f0` GOVERNANCE/TOOLING-ONLY GO 5m23s (12:25:31→12:30:54+07 2026-07-13, harness adopt presence-not-age + hmw.js STOP-HARD; 20-file: 0 src/fe/mig/test — non-ignored trigger = `.claude/workflows/hmw.js` + `scripts/memory-archive-gate.ps1` [`.ps1`∉`scripts/**.md` ⟹ CI-run ĐÚNG #41; rest .md/.json under docs/agent-memory/workflows-runs/broadcasts IGNORED]):** status=success · **gate CONCRETE `runs/385/jobs/0/logs`: Domain 45 + Infra 441 = 486/0-fail/0-skip = S114 baseline EXACT** (0 src/tests diff → count frozen ✓) · **FE bundle FROZEN ×4** admin js `D51OYyGV`/css `CfunQHbd` · user js `BVdssm5S`/css `DGqqHQXS` = S114 verbatim (NO rotate) — deploy physically re-copied (LM 05:29:32/05:30:26 GMT = 12:29:32/12:30:26+07 IN-window) BUT byte-identical (size 1,655,105/1,561,453b = S114 EXACT) ⟹ #69-REFINE 0-fe-src=identical-Vite=same-hash+size+fresh-LM · SPA-trap ctrl fake-hash 919b text/html · Mig FROZEN 65 (0 migration-file diff; API health/ready 200=startup OK) · smoke 4/4 200 (api ready+live, admin, eoffice). 🔑 GOVERNANCE-ONLY PASS: non-.md-in-.claude (hmw.js/.ps1) STILL trigger CI (#41 LIVE post-#495 D10); FROZEN-hash+fresh-LM+identical-size = no-accidental-rotation proof. cicd-lane clean-return no-garble.
- **🟢 S114 #384 (rn 384, id 498) `d436af2` GO 5m17s — PE multi-NCC per hạng mục + Mig 65 `BackfillPeQuoteIsSelectedAndWinnerInvariant` APPLIED-PROD (data-only 2-dir backfill, invariant-violations=0 [IsWinner⟺Any(IsSelected)], sys.tables 89) · gate 486 (45D+441I) · FE js-ROTATE×2/css-frozen admin `acprMqfB`→`D51OYyGV`/user `Ck3BTDdA`→`BVdssm5S` (= CURRENT ship-baseline, size 1,655,105/1,561,453b) · byte-verify `trúng thầu theo từng hạng mục` · select-winner WIRED 401 · smoke ALL 200 · 🔑 js-rotate+css-frozen = FE-logic-no-style fingerprint — verbatim → `archive/2026-07.md` via `_INDEX`.**
- _(**S111 #495 `8aa3869` H22-WAL-adopt** [deploy.yml SELF-change +2 paths-ignore, run parsed+ran-full = workflow-self-change PASS; bundle FROZEN ×6-streak; 🔑 **AS-6 baseline-shift**: post-#495 memory-push (budget.json/agent-memory/run-trace) HẾT trigger CI, phân-loại giờ soi file ngoài 7-glob] **+ S112 #496 `e100ef0`** PE-workflow+Supplier-import-Phase-B+Mig63 `AddSupplierImportSourceFields` [gate 458, 🔑 411-refine: bodyless-POST→411 ANY-route → re-probe WITH-body → 401/415-matched vs 404-missing] **+ S113 #383 `5fa11b5`** Supplier-import-v2+Mig64 `AddSupplierPublishState`-APPLIED-PROD [IsPublic 37/37, gate 477] — all GO ~5m, FE real-ship, Mig-applied-prod, smoke ALL 200 · full verbatim → `archive/2026-07.md` via `_INDEX`)_
- _(**S107-closeout→S110-spot #490-#494** GOVERNANCE/MEMORY-push GO ×5 [`d74f67f`·`69957fd`·`f5a5c3e`·`6ee60cd`·`ac3f122` — trigger #41 non-.md-in-push: budget.json ×2 + hmw.js ×3, pattern DEAD từ D10 #495] — bundle FROZEN-streak ×5 sau ROTATE #489 [#69 TWO-MODE; LM-in-window + real-size 1,644,931/1,551,280b = signal duy nhất] · Mig 62 + sys.tables 89 frozen · gate 440 · smoke 3×200 ALL · 🔑 #494 spot-detection: LM-prod-newer-than-last-verified-window = missed-run detector · full verbatim → `archive/2026-07.md` via `_INDEX`)_
- _(**S107 #489 `21387cd` GOVERNANCE-ONLY (+office-document 12th sub) GO 5m33s** — JS ROTATED-benign `ChSNAJBK`→`BVyH-FY7` / `Bn2w1znH`→`EDsZebvn` CSS-frozen, 🔑 #69 TWO-MODE CONFIRMED (0-fe-src ROTATE #312/#489 hoặc FROZEN #333/#485-487 — ship-proof = LM-in-window + real-size, hash-delta phụ) · **S104 #488 `778fc98` FULL-STACK Supplier 9→27 + Mig 62 `ExpandSupplierFields` APPLIED-PROD** — AddColumn×18-exact + sys.tables-89-invariant + un-cram 4-NCC Note=NULL/Pkg/Bank/LegalRep-populated-EXACT + DTO-18-new-keys-live + JS-rotate-real-src + 🔑 FULL-STACK-SCHEMA-ADD+DATA-UN-CRAM PASS-pattern — full verbatim → `archive/2026-07.md` via `_INDEX`)_
- _(**S103 #487 + S100 #486 + S98 #485** GOVERNANCE/SCRIPTS-ONLY GO ×3 [.ps1/.json/.md-mix non-ignored → #41] — bundle FROZEN #69-REFINE ×4, GOVERNANCE-PASS stable [#476/#485-487] · 🔧 python `time.sleep` foreground-OK (chỉ bash-sleep-block) · full verbatim → `archive/2026-07.md` (+#485 git `c112be9`) via `_INDEX`)_
- _(**S96+S97 FINAL-REVIEW #478-#484** 11-commit ALL-HEALTHY whole-session-GO [backfill A/005=1/A/008=1/A/010=0-exact + 88-excl/89-incl reconcile; #69 rotation≠ship-signal] · **S97 #484 `fb2dfa9` DATA-FIX Mig61** 🔑 DATA-FIX-BACKFILL PASS = CI + Mig-top==HEAD + target-rows-flip-EXACT + control-untouched + derived-API-field-FLIP-live — verbatim → `archive/2026-07.md` via `_INDEX`)_
- _(S96-S97 endsBeforeCeo-saga + PE-fixes #479-#484 all PASS 440 — key: #482 Mig60 CROSS-STACK [🔴 EF-TERNARY-SQL-translate-LIVE list-200-NOT-500 · #77 field-PRESENT] · #481/#483 FE-only [#77 new-str-PRESENT/old-ABSENT] · #480 baseline-440 +PeSuggestedPriceSetterAuthzTests · JS-rotate/CSS-frozen · verbatim → git + `2026-06.md` via `_INDEX`)_
- _(S90→S96 #465·#470·#472·#473·#476·#477 all PASS — key: #477 🔑 **DATA-SEED-PASS = pre/post-DB-row-EXACT-delta + DLL-mtime + w3wp-recycle + Mig-frozen + API-payload-contains-row** [⚠️ sqlcmd `-W` diacritic-LIKE false-NEG → `_`-wildcard; base64-EncodedCmd-FAIL → `.ps1`-File] · #472 ⚠️ **PROD Identity tables NON-prefixed `Roles` NOT `AspNetRoles`** · #470 GO-LIVE 🔑 no-resurrect = pre+post-DB-count-diff + dll-in-window + w3wp≥run-window [integrated-auth `-E`] · #476 GOVERNANCE-PASS baseline · verbatim → git + `2026-06.md` via `_INDEX`)_
- _(S69b→S89 runs #307·#313·#315·#318·#330·#333·#342-#348 all PASS — key: Mig54-59 applied-prod · #345 PATCH-401+7-authz · #344 backfill-winners=13 · #343 BE-only ship-proof=Infra.dll-mtime · #342 INVERTED-PASS-FROZEN [#73 TZ] · #346 FE-only js+css-BOTH-rotate · #348 411→401 endpoint-wired · full verbatim → git + `2026-06.md` via `_INDEX`)_
- **Older runs (S83 #325 FE-budget-indent ← S77 #329 banner ← S75 #301 ← S62 #286 ← … → S29 #232) full verbatim → git `d2f52ba` + `archive/2026-06.md`** (incl #291 06-16 FAIL forensic [gotcha #65]); pre-S38 → `archive/2026-05-{runs,q2,q3,q4}.md`.

---

## 🔄 Curate trigger
- >~30KB → archive recent runs → L2 `archive/<period>.md` (byte-exact append) + `_INDEX.md` substring pointer. Dup failure patterns → merge. Stale >3mo → remove.
- **Last curate: 2026-07-13 S115 (self, PostToolUse 21.6→<17KB)**: #384/#383/#496/#495 (S114/S113/S112/S111) verbatim → `archive/2026-07.md` + `_INDEX` +4 pointers → S114 compact-stub + S111-113 merged-stub; top S114 garble-block → 1-line tracker.
- **Prev: S111 07-12 self 20.5→17 (#490-#494→archive) · S107 07-10 self 20.4→16.9 (#489/#488→archive) · S104 07-06 self 20.5→17.9 · S98 07-02 em-main 19.9→17.06 (`2026-07.md` NEW) · S80 06-20 em-main 38.8→16 · S70 06-17 Harness-9 65.2→23.2 · earlier S66 · S40.**
