# CI/CD Monitor Agent — Persistent Memory

> **Persistent diary cross-session.** Auto-injected first ~200 lines at spawn (L1 HOT).
> Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 `archive/` on-demand · L3 RAG `search_memory` just-in-time. Keep entry ≤ 1.5K chars (gotcha #53).
> Full verbatim run history pre-S40 → git `d2f52ba` + `archive/2026-05-{runs,q2,q3,q4}.md` + `archive/2026-06.md`. 🗺️ **Lookup map: `archive/_INDEX.md`** — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + `2026-0{5,6}.gist.md` theo nhu cầu. **S80 curate: moved 3 huge run-entries (#330/#318/#325) → archive.**

---

## 🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: `iis-deploy-runbook` + `dependency-audit-erp` + `ef-core-migration`. Spawn ~150K — trade-off catch fail tự động.

---

## 🚨 Recurring CI/CD bug patterns (catch priority)

- **#39 act_runner github.com TCP timeout** — run hang "Set up job" 21s. Log `dial tcp github.com:443 i/o timeout`. Fix: manual checkout bypass hardcoded `.gitea/workflows/deploy.yml`. KHÔNG revert.
- **#40 npm cache `tsc not found`** — `build_fe_admin` fail post `cache: npm`. DISABLED rolled back `a21790d`. KHÔNG re-enable.
- **#41 paths-ignore docs-only skip** — code commit không trigger CI? Check `git diff --name-only HEAD~1 HEAD` vs `paths-ignore: ['docs/**','**/*.md','.claude/skills/**']`. Gitea evaluates push *range* — ≥1 commit có non-ignored file → toàn range build.
- **#25 IIS WebSocket** — `notification-hub/negotiate` 401/404 prod. Fix: WebSocket module enable `web.config` site api (skill `iis-deploy-runbook`).
- **#48 SQLite tie-break** — `OrderByDescending(CreatedAt).First()` pick wrong khi 2+ `.Add()` cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy.
- **Bundle hash unchanged = ship FAIL** — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (`curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'`), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).
  - **🔴 #69 (S72 Run #312) — FE bundle hash NON-DETERMINISTIC per rebuild.** `deploy.yml` `Remove-Item fe-*\* -Exclude web.config` + `Copy-Item dist\*` runs UNCONDITIONALLY every run (path-filter gates whole-workflow trigger, NOT per-step). Identical FE source ⟹ DIFFERENT hash each deploy (Vite/rolldown non-reproducible chunk-id). PROVEN: governance-only `18fced6` (0 files in fe-*/src) rotated BOTH bundles. ⟹ "BE-only/governance ⟹ bundle frozen" is FALSE; rotation EXPECTED every deploy. **To detect REAL FE change, diff `fe-*/src` in commit/range, NOT hash delta.**
  - **⚠️ SPA-fallback 200 trap (S72):** server rewrites `/*`→`/index.html` so GET `/assets/index-<ANYTHING>.js` returns **200** even for fake hash (control `ZZdoesnotexist0.js`→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html `<script src>`, GET that exact path + check `size_download` LARGE (real ~1.6MB vs fake ~919b) + `Last-Modified` in deploy window + 2nd-fetch byte-stable (no mid-deploy transient).
  - **🟢 #69-REFINE (S83 Run #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change.** #69 said "rotates EVERY deploy" — NOT absolute. Run #333 (BE+tests+governance, ZERO `fe-*/src` diff) shipped FE (Last-Modified 17:10:35/17:11:27 BOTH in run-window 17:06→17:11) but hash UNCHANGED (admin `CsJetgZH`/user `BVS0ApIm` = #330 baseline). ⟹ identical Vite source ⟹ byte-identical output ⟹ same content-hash. Rolldown chunk-id apparently MORE deterministic than #69 feared (or only rotates on real src delta). **MOST RELIABLE ship-proof = `Last-Modified` header on the real bundle file IN deploy-window, NOT hash-delta** (hash-delta has false-NEGATIVE when src unchanged + false-POSITIVE risk noted in #69). Always cross-ref `git diff fe-*/src` scope: 0-FE-change ⟹ expect SAME hash + fresh Last-Modified.
- **Migration drift prod vs repo** — compare `ls .../Persistence/Migrations/*.cs` vs `sqlcmd __EFMigrationsHistory`. Fix: check `Program.cs`/DbInitializer `app.MigrateDatabase()` + app pool recycle. Mig-applied proof = `__EFMigrationsHistory` top==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.

---

## 📋 5-stage checklist (EVERY run)

- **Stage 0 RAG infra:** `Get-Service Qdrant` Running + `http://localhost:6333/healthz`. Collection `proj_solution_erp`.
- **Stage 1 Push+filter:** `git log -1 --format='%H %s'` + `git log origin/main..HEAD` empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
- **Stage 2 Gitea poll** (max 10 iter × 60s): API `.../actions/tasks?limit=5` (NOT `/runs` 404). Match `head_sha` (NOT run_number — cancelled runs shift numbering). ⚠️ `updated_at` stale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter.
- **Stage 3 Test gate:** baseline **440 PASS** (45 Domain + 395 Infra; S96 #480 `0c2f9f6` — 436 + new `PeSuggestedPriceSetterAuthzTests` [price-lock-after-submit authz]; prior S96 #479 436, S96/S95 434, S90 #465 421). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (`tasks` terminal=`status:success`, `conclusion` NOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK.
- **Stage 4 Post-deploy** (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=`accessToken` route `/api/auth/login`) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns **411 IIS Length-Required pre-auth NOT 404**; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.
  - **Stage 4.6 (S29 CRITICAL):** sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
  - Discovery: ASP.NET 10 record enum needs numeric input unless `JsonStringEnumConverter` (SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth `\\\\SQLEXPRESS` 4-backslash. INFRASTRUCTURE seed MUST run (NOT inside `if(!demoSeedDisabled)`); DEMO seed gated → gotcha #51.
- **Stage 5 Report** PASS/FAIL + evidence + MEMORY update.

---

## ⚠️ Anti-patterns (DO NOT)
1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

---

## 🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)

- **Gitea:** `git.baocaogiaoduc.vn/vietreport-admin/solution-erp` · workflow `.gitea/workflows/deploy.yml` · paths-ignore `['docs/**','**/*.md','.claude/skills/**']`. Anon API works (public repo) when GITEA_TOKEN absent.
- **Prod:** api/admin/eoffice `.solutions.com.vn` · SSH `ssh vietreport-vps` (Administrator, id_ed25519) · IIS phys paths: API `C:\inetpub\solution-erp\api` · admin `\fe-admin` · user `\fe-user`. DB `.\SQLEXPRESS`/`SolutionErp`/`vrapp` SQL-auth. **Conn key = `ConnectionStrings.Default` (NOT `DefaultConnection`)**. DB pw = `buKL3TGBkD0wDDbYVw65QeX9` (User Id=`vrapp`; ⚠️ Password field has NO `vrapp/` prefix — S89 login-failed when prefix included; read verbatim from prod `appsettings.Production.json` ConnectionStrings.Default when `$env:PROD_DB_PASSWORD` empty). ⚠️ skill-doc path `C:\inetpub\apps\SolutionErp\Api` STALE.
  - **SSH→PS quoting (S42):** nested bash→ssh→powershell mangles `$var`/`\"` → use `iconv UTF-16LE | base64` → `powershell -EncodedCommand $B64`; OR write ps1-file + `powershell -File $(cygpath -w)`. sqlcmd over SSH direct `-W -h -1` pw-inline. sys-catalog string-concat → `COLLATE DATABASE_DEFAULT` (collation conflict).
- **Tests baseline:** **440 PASS** (45 Domain + 395 Infra · 0 fail/skip; S96 Run #480 sha 0c2f9f6 — 436 + new `PeSuggestedPriceSetterAuthzTests`; prior S96 #479 436, S96/S95 434, S90 #465 421). Phase 9 UAT skip per chunk OK.
- **Mig latest repo:** **Mig 59 `20260624084608_AddCcmBudgetPeriodToPurchaseEvaluation`** (S88 #345; PE +1 AddColumn `CcmBudgetPeriodAmount` decimal(18,2) NULL no-backfill — VERIFIED-APPLIED-PROD history-HEAD + col-exists 0-nonnull). Prev Mig 58 `AddPeMultiWinnerAndLevelFinalize` (S87 #344). Path `src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/`. Prod check `sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5`. ⚠️ **`sys.tables` (is_ms_shipped=0) = 88** (S61 Budget-replace DROPPED 93→88 — narrative-93 STALE; when commit touches no schema, 88 correct, don't FAIL on 88. Always cross-ref COMMIT scope vs ambient count). enum-additive (e.g. `ApprovalAttachment=5`) = int-stored NO-mig → expect history NOT-advance + empty `git diff -- '*Migrations*'`.
- **Bundle hash live (S96 Run #480/id=480 — JS-rotated real-FE, CSS-frozen):** admin js **`COYAX-yQ`**/css `Bbtesa6B` · user js **`0MU-kb4N`**/css `BmuPz_Dy` (JS rotated from S96 `kL1-N1Zq`/`BRCrr80A` = real PeWorkflowPanel+PeDetailTabs FE change; CSS IDENTICAL S96/S96 = logic+gate-only ⟹ #69-REFINE byte-identical-CSS⟹same-hash). ⚠️ **id ≠ run_number(365); head_sha-keyed.** ⚠️ **PER #69: hash MAY rotate per deploy (real src-diff) OR stay frozen if 0 fe-src diff (byte-identical) — SNAPSHOT for THIS run, NOT a frozen baseline; ALWAYS re-snapshot LIVE pre-deploy (brief prev-hash=hint NOT truth).** Real FE ship = diff `fe-*/src` in commit/range + real-size-vs-fake (1.6/1.5MB vs ~919b SPA-trap) + Last-Modified-in-window, NOT hash delta alone.
- **Bearer:** admin `admin@solutions.com.vn/Admin@123456` (full) · UAT `nv.test@solutions.com.vn/TestUser@123456` (Drafter CCM, gotcha #44 check).

## 🔑 Critical config (flag commit nếu tái xuất)
Node CI `20.x` (`feedback_node_cicd`) · MediatR `12.4.1` (gotcha #1, flag `Version="14`) · Swashbuckle `6.9.0` (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag `cache: npm`)

---

## 🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)
Cross-ref `feedback_per_nv_permission_scope`. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF `HasDefaultValue(false)` · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 **Lookup discrimination** (`FirstOrDefault` ADD `ApproverUserId==actorId` + admin fallback) · 10 **Controller body record count == Command record count**. Bug latency 2-3 days prod silent khi miss 9-10. Scan `grep -n "FirstOrDefault.*Order.*==" *.cs` after OR-of-N refactor.

## 📊 Run stats baseline
BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · **total ~3min code / 0s docs-only**. >5min → escalate. Recent runs all ~4m40-5m.

---

## 📅 Recent runs (compressed — full verbatim → `archive/2026-06.md` via `archive/_INDEX.md`)

- **🟢 S96 #480 (id=480, run_number 366) `0c2f9f6` GO ~5m25s (PASS) — cross-stack PE 3-commit range (`ba4b8e0`→HEAD): (a) `e657faa` opt-out KẾT-THÚC checkbox [BE+FE `PeWorkflowPanel` 2-app] + (b) `36cbf8b` mark-read stale-noti [BE-only] + (c) `0c2f9f6`=HEAD khóa-sửa-giá-đề-xuất-PRO/CCM-sau-gửi-duyệt [BE `PeSuggestedPriceFeatures` phase-guard + FE `PeDetailTabs.tsx` 2-app gate + test `PeSuggestedPriceSetterAuthzTests`], NO migration (cả 3)** — 3 runs sequential id=478 `e657faa`✅@15:46 → id=479 `36cbf8b`✅@15:52 → **id=480 `0c2f9f6`✅@16:12:59=HEAD=canonical=ship-tất**. Poll running→success ~5m25s (start 16:07:34→16:12:59; busy-wait-spin, `updated_at` fresh-advancing=alive-NOT-hung; foreground-sleep-BLOCKED Windows). status=success ⟹ test-gate green (baseline **440** = 45D+395I, +new `PeSuggestedPriceSetterAuthzTests`; tasks-API terminal-success trust-NOT-log-numeric). 🎯 **FE JS ROTATED (real FE proof — e657faa PeWorkflowPanel + 0c2f9f6 PeDetailTabs 2-app)**: admin `0T6vVJST`(S96)→**`COYAX-yQ`** + user `BGXEVZO6`(S96)→**`0MU-kb4N`** ✅. **CSS FROZEN** admin `Bbtesa6B`/user `BmuPz_Dy` = IDENTICAL S96/S96 (logic+gate-only-FE, 0-style-diff ⟹ #69-REFINE byte-identical-CSS-source⟹same-hash; matches S93/S95/S96 JS-rotate-CSS-frozen; NOT-fail). Real-size-verify (guard SPA-trap): admin JS **1,636,639B** + user **1,542,960B** (control fake-hash `ZZfakehash0.js`→919B text/html=SPA-fallback ⟹ real-bundles-genuine-not-trap); Last-Modified admin 09:11:38GMT=16:11:38+07 / user 09:12:34GMT=16:12:34+07 (=disk-mtime EXACT-match, in-window). **BE ship-proof**: Infrastructure.dll `16:08:35`+Application.dll `16:08:10`+Api.dll `16:10:20` all-in-window · w3wp fresh PID=3184@`16:12:39` + PID=11104@`16:12:21` AT-END-window (pool recycled ⟹ new DLL; 6 w3wp total=3-site pools, 2-newest bracket deploy-end). **Mig prod=59 `AddCcmBudgetPeriodToPurchaseEvaluation` UNCHANGED** (=repo-HEAD; 0-mig-file range `ba4b8e0..0c2f9f6`⟹correct-frozen). Smoke: api ready/live+admin/eoffice-root ALL 200, **auth accessToken 468ch, GET /api/purchase-evaluations 200 items[]-valid (PE-not-broken by price-lock-guard) + /approval-workflows-v2 + /notifications 200**. ⚠️ UI checkbox/chuông/khóa-giá NOT curl-verifiable (login+phiếu-thật → anh Kiệt UAT); scope=deploy-ship+test-gate+bundle-rotate+smoke. 🔑 **REAL-FE-CHANGE PASS = JS-hash-rotate(vs prev-snapshot) + real-size-vs-919b-trap + Last-Modified-in-window + CSS-frozen-OK-if-logic-only(#69-REFINE) + BE-DLL-mtime-in-window + w3wp-recycle + Mig-frozen + smoke-200-payload-valid**. base64-EncodedCommand-over-ssh WORKED this run (CLIXML-wrapper = PS-remoting-noise, `grep -v`; contradicts S96-note base64-FAIL — both methods situational, try base64 first fallback `.ps1`-File S42).
- _(S96 #479 `36cbf8b` GO ~5m40s PASS — PE 2-commit e657faa opt-out-checkbox+36cbf8b mark-read-stale-noti NO-mig · baseline-436 · JS admin`0T6vVJST`→`kL1-N1Zq`/user`BGXEVZO6`→`BRCrr80A` rotated **CSS FROZEN** Bbtesa6B/BmuPz_Dy · real-size 1636619/1542940B · Mig59-frozen · w3wp PID3024@15:52:19 · full verbatim → git `36cbf8b` + archive `2026-06.md` via `_INDEX`)_
- _(S96 #477 `ba4b8e0` Infra data-only seed import NCC `DONGDUONG` Excel anh-Kiệt NO-mig ungated-idempotent · baseline-434 · 🎯 DONGDUONG-LANDED pre=3→post=4 suppliers total=21 EXACT+1-idempotent Type=2-NhaThauPhu · **FE bundle FROZEN** admin `0T6vVJST`/user `BGXEVZO6` 0-fe-src-diff(#69-REFINE) · Mig59-frozen · w3wp PID13660@14:54:28 recycle+DLL-in-window · 🔑 **DATA-SEED-PASS = pre-vs-post-DB-row-EXACT-delta + DLL-mtime + w3wp-recycle + Mig-frozen + API-payload-contains-row** · ⚠️ sqlcmd `-W` diacritic-LIKE false-NEG use `_`-wildcard; base64-EncodedCommand-FAIL→`.ps1`-File-method reliable — full verbatim → git `ba4b8e0` + archive `2026-06.md` via `_INDEX`)_
- _(S95 #476 `18a17f5` GOVERNANCE-ONLY adopt Harness-17 — 2 `.ps1` scripts/ trigger CI despite 0-prod-code · baseline-434 · JS→`0T6vVJST`/`BGXEVZO6` rotated **CSS FROZEN** · Mig59 frozen · brief-prev-hash STALE→re-snapshot-LIVE · 🔑 GOVERNANCE-PASS=CI-green+JS-rotate/CSS-frozen(#69)+3×200+Mig-frozen — full verbatim → git `18a17f5` + archive `2026-06.md` via `_INDEX`)_
- _(S93 #473 `8ee8f3e` FULL-STACK PE tree-node show-CODE DTO+3-proj NO-MIG · S92 #472 `1408d4e` BE-only permission-revoke +5-menu-group VERIFY-PROD-EFFECT [hidden-key non-Admin CanRead=0, Admin-retained, Pe_* untouched, leak=0; ⚠️ **PROD Identity tables NON-prefixed `Roles` NOT `AspNetRoles`**] · S91 #470 `55494ad` GO-LIVE no-resurrect demo-NCC seed-gate [pre+post-restart demo-code=0; **PATTERN go-live-no-resurrect = pre+post-DB-count-diff + dll-in-window + w3wp≥run-window**; integrated-auth `-E`] · S90 #465 `a71753b` FULL-STACK PE draft-private js-rotate-css-FROZEN → all PASS baseline-422→434, full verbatim → git + archive `2026-06.md` via `_INDEX`)_
- _(S89 #348 `1aa3bdb` PE create-contract-1→N multi-winner endpoint-411→401 · S88-quad #347 `73cce1f` PE D1 Block-B re-key+CEO-notify · S88-bis #346 `0d3c0cf` FE-only PE-layout js+css-BOTH-rotate · **S88 #345 `9b1ee0c` = Mig 59 `AddCcmBudgetPeriodToPurchaseEvaluation` AddColumn-no-backfill VERIFIED-PROD** setter PATCH-401 + 7 authz-test → all PASS, full verbatim → git + archive `2026-06.md` via `_INDEX`)_
- _(S87 #344 `f1b10e9` FULL-STACK PE CCM 3-fix + **Mig 58 `AddPeMultiWinnerAndLevelFinalize`** backfill-winners=13 VERIFIED-PROD select-winner-411→401 · S86 #343 `ae4f106` BE-only notify-exclude-drafter ship-proof=Infrastructure.dll-Last-Modified · S85 #342 `d4f6021` DOCS+SQL-no-op INVERTED-PASS-when-FROZEN [gotcha #73 TZ + #69-REFINE] · S83 #333 `fc1f19d` BE-only guard-V2-required hash-FROZEN-but-shipped · S78 #330 `7886fd0` enum-attach-on-approve · S76 #318 `e33481e` Mig56-ProInitial/Adjust · S74-bis #315 Mig55 · S73 #313 Mig54-endpoint-401-not-404-probe · S69b #307 Office-golive-seed-16/16 → all PASS, full verbatim → git + archive `2026-06.md` via `_INDEX`)_
- **Older runs (S83 #325 FE-budget-indent ← S77 #329 banner ← S75 #301 ← S62 #286 ← … → S29 #232) full verbatim → git `d2f52ba` + `archive/2026-06.md`** (incl #291 06-16 FAIL forensic [gotcha #65]); pre-S38 → `archive/2026-05-{runs,q2,q3,q4}.md`.

---

## 🔄 Curate trigger
- >~30KB → archive recent runs → L2 `archive/<period>.md` (byte-exact append) + `_INDEX.md` substring pointer. Dup failure patterns → merge. Stale >3mo → remove.
- **Last curate: 2026-06-20 S80 (em-main, archive-gate keep-floor-hit → manual)** (38.8→~16KB): moved 3 huge run-entries #330/#318/#325 (lines 73/90/96 byte-exact via `sed`) → `archive/2026-06.md` (100.6→116.7KB) + `_INDEX.md` +3 substring pointers (verified count=1). KEPT foundation (bug-patterns/5-stage/10-point gold) + compressed essentials + run-stubs; UPDATED stale (130→354 test Stage-3 + 263→354 essentials, Mig 55→57, bundle→#330) + dropped verbose per-run bundle-history (S86/S85/S84 → archived run-entries).
- **Prev curate: 2026-06-17 S70 Harness-9** (65.2→23.2KB dark-matter recovery, built _INDEX/gist gen:1). Earlier: S66 · S40.
