# CI/CD Monitor Agent — Persistent Memory > **Persistent diary cross-session.** Auto-injected first ~200 lines at spawn (L1 HOT). > Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 `archive/` on-demand · L3 RAG `search_memory` just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). > Full verbatim run history pre-S40 → git `d2f52ba` + `archive/2026-05-{runs,q2,q3,q4}.md`. --- ## 🎯 Role baseline Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: `iis-deploy-runbook` + `dependency-audit-erp` + `ef-core-migration`. Spawn ~150K — trade-off catch fail tự động. --- ## 🚨 Recurring CI/CD bug patterns (catch priority) - **#39 act_runner github.com TCP timeout** — run hang "Set up job" 21s. Log `dial tcp github.com:443 i/o timeout`. Fix: manual checkout bypass hardcoded `.gitea/workflows/deploy.yml` (pass #110). KHÔNG revert. - **#40 npm cache `tsc not found`** — `build_fe_admin` fail post `cache: npm`. DISABLED rolled back `a21790d`. KHÔNG re-enable. - **#41 paths-ignore docs-only skip** — code commit không trigger CI? Check `git diff --name-only HEAD~1 HEAD` vs `paths-ignore: ['docs/**','**/*.md','.claude/skills/**']`. Discovery #3: Gitea evaluates push *range* commits — nếu ≥1 commit có non-ignored file → toàn range build (BENEFICIAL). - **#25 IIS WebSocket** — `notification-hub/negotiate` 401/404 prod. Fix: WebSocket module enable `web.config` site api (skill `iis-deploy-runbook`). - **#48 SQLite tie-break** — `OrderByDescending(CreatedAt).First()` pick wrong khi 2+ `.Add()` cùng frozen-clock. Fix: discriminator filter `.Where(Summary.Contains("Chuyển phase"))` BEFORE OrderBy. - **Bundle hash unchanged = ship FAIL** — push+action success nhưng prod không đổi. Verify `curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[a-z0-9]+\.js'`. Fix: SSH `Restart-WebAppPool`. ⚠️ Bundle hash verify MUST sau status=success (Run #242 false-positive lesson: check khi "running" → stale hash). - **Migration drift prod vs repo** — compare `ls .../Persistence/Migrations/*.cs` vs `sqlcmd __EFMigrationsHistory`. Fix: check `Program.cs` `app.MigrateDatabase()` + app pool recycle. --- ## 📋 5-stage checklist (EVERY run) - **Stage 0 RAG infra:** `Get-Service Qdrant` Running + `http://localhost:6333/healthz`. Collection `proj_solution_erp` (prefix `proj_*` 7 project — Discovery #8). - **Stage 1 Push+filter:** `git log -1 --format='%H %s'` + `git log origin/main..HEAD` empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return). - **Stage 2 Gitea poll** (max 10 iter × 60s): API `.../actions/tasks?limit=5` (NOT `/runs` 404). Match `head_sha`. ⚠️ task table `updated_at` stale ~2min (gotcha #46) → cross-check VPS mtime. - **Stage 3 Test gate:** baseline **130 PASS** (58 Domain + 72 Infra). Phase 9 UAT exception lower OK (`feedback_uat_skip_verify`). - **Stage 4 Post-deploy** (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=`accessToken` route `/api/auth/login`) → 3-5 endpoint smoke 2XX (incl new) → FE bundle hash 2 app changed → SignalR negotiate (gotcha #25 if relevant) → EF mig prod==repo. - **Stage 4.6 (S29 CRITICAL):** sqlcmd seed sample verify post-deploy (NOT chỉ schema). `sqlcmd -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'"` → 0 rows = seed GATE BLOCKED → gotcha #51. - Discovery #4: ASP.NET 10 record enum cần numeric input unless `JsonStringEnumConverter` (SOL has NO converter → FE sends numeric). #5: sqlcmd ssh Windows-auth cần `\\\\SQLEXPRESS` 4-backslash. #6: INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST run, NOT inside `if(!demoSeedDisabled)`; DEMO seed (DemoUsers/Contracts/PE) OK gated → gotcha #51. - **Stage 5 Report** PASS/FAIL + evidence + MEMORY update. --- ## ⚠️ Anti-patterns (DO NOT) 1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay) --- ## 🧠 SOLUTION_ERP CI/CD essentials (S40 verified) - **Gitea:** `git.baocaogiaoduc.vn/vietreport-admin/solution-erp` · workflow `.gitea/workflows/deploy.yml` · paths-ignore `['docs/**','**/*.md','.claude/skills/**']` - **Prod:** api/admin/eoffice `.solutions.com.vn` · SSH `ssh vietreport-vps` (Administrator, id_ed25519) · IIS site phys paths (S42 verified): API `C:\inetpub\solution-erp\api` · admin `\fe-admin` · user `\fe-user` (3 sites Started). DB `.\SQLEXPRESS`/`SolutionErp`/`vrapp` SQL-auth. **Conn string key = `ConnectionStrings.Default` (NOT `DefaultConnection`!)** — read pw from prod appsettings.Production.json when `$env:PROD_DB_PASSWORD` empty. - **SSH→PS quoting (S42 lesson):** nested bash→ssh→powershell mangles `$var`/`\"`. Use `iconv UTF-16LE | base64` → `powershell -EncodedCommand $B64`. Single-quote literal paths. - **Tests baseline:** **263 PASS** (S62 Run #286 sha 7926c21 spec; 45 Domain + 218 Infra — em-main supplied; supersedes prev 228/240/256). CI gate runs both test projects BEFORE build/deploy → status=success ⟹ test gate passed (`tasks` endpoint reports terminal as `status:success`, `conclusion` field NOT populated). Local grep undercounts (Theory/InlineData) — trust CI conclusion. Phase 9 UAT mode skip per chunk OK. - **Mig latest repo:** **Mig 52 `20260616035929_AddHoSoLinkToPurchaseEvaluation`** (S65; PE +HoSoLink hyperlink-NAS, AddColumn-only no new table). Prev Mig 51 `AddDepartmentParentId` (S65 Department.ParentId loose-Guid org-tree, AddColumn-only) + Mig 50 `ReplaceBudgetModuleWithPeWorkItemBudgets` (S61 Budget→PeWorkItemBudgets net-reduce). Path `src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/` (52 mig .cs non-designer total). Prod check `sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5`. ⚠️ Table-count: `sys.tables` (is_ms_shipped=0, excl mighist) = **88** (S62 Run #286 verified — S61 Budget-replace DROPPED tables 93→88). Narrative-93 is STALE pre-S61 — when commit touches no schema, 88 is correct, don't FAIL on 88↔93. Always cross-ref COMMIT scope vs ambient count. - **Bearer:** admin `admin@solutions.com.vn/Admin@123456` (full) · UAT `nv.test@solutions.com.vn/TestUser@123456` (Drafter CCM, gotcha #44 check) - **Bundle hash live S77:** admin `D532XZKG` (Run #303 sha 6983609 — ROTATED from `CcrZqfht`; EmployeesListPage employee-detail banner text-polish) · user `CuFaBoWt` (Run #303 sha 6983609 — ROTATED from `DniDFUB_`; same page both apps SHA256-identical `F013B748`). Prev live admin `CcrZqfht`/user `DniDFUB_` (Run #302 S76). ⚠️ ASYMMETRIC-deploy lesson (S66): FE-one-app commit → that app's bundle MUST rotate + OTHER app MUST stay frozen; admin-rotate-when-only-fe-user-changed = anomaly → flag. S50 mid-deploy transient lesson: pre-success snapshot can show intermediate FE copy in-flight — re-confirm hash AFTER status=success ALWAYS (anti-pattern #3). FROZEN-expectation runs (BE-only or other-app): hash MUST stay = live pre-deploy value; rotate w/o relevant FE change = anomaly. - **DB pw (S42, when `$PROD_DB_PASSWORD` empty):** `vrapp/buKL3TGBkD0wDDbYVw65QeX9` read from `C:\inetpub\solution-erp\api\appsettings.Production.json`→`ConnectionStrings.Default`. ⚠️ Skill-doc path `C:\inetpub\apps\SolutionErp\Api` is STALE → real path `C:\inetpub\solution-erp\api`. sqlcmd over SSH works direct (no UTF-16 encode needed). ⚠️ sys-catalog string-concat queries hit collation conflict (`Latin1_General_CI_AS_KS_WS` vs `SQL_Latin1_General_CP1_CI_AS`) → add `COLLATE DATABASE_DEFAULT` per concatenated column. ## 🔑 Critical config (flag commit nếu tái xuất) Node CI `20.x` (`feedback_node_cicd`) · MediatR `12.4.1` (gotcha #1, flag `Version="14`) · Swashbuckle `6.9.0` (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag `cache: npm`) --- ## 🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23) Cross-ref `feedback_per_nv_permission_scope`. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF `HasDefaultValue(false)` · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 **Lookup discrimination** (`FirstOrDefault` ADD `ApproverUserId==actorId` + admin fallback) · 10 **Controller body record count == Command record count**. Bug latency 2-3 days prod silent khi miss 9-10. Scan `grep -n "FirstOrDefault.*Order.*==" *.cs` after OR-of-N refactor. ## 📊 Run stats baseline BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · **total ~3min code / 0s docs-only**. >5min → escalate. --- ## 📅 Recent runs (FIFO — older → archive/git) - **2026-06-17 S69 Run #305 (run_number 305, id419) sha=`a8bbdae` PASS ~4-5m (FE BOTH-APP foundation "Văn phòng số" + index.css sync + BE menu-seed NO-mig: 3 shared component mới PageHeader/KpiCard/WidgetCard + OfficeDashboardPage landing route `/office/dashboard` 4-place-wire BOTH apps + `fe-admin/src/index.css` SYNCED (Hồ sơ NS accent tokens, rotate admin mạnh) + BE menu key `Off_Dashboard` (MenuKeys.cs L100 + DbInitializer.cs L1825 seed parent=`Off` Order0 LayoutDashboard); deploy 9/9 session after #297–#304 all PASS):** Push `764fe70..a8bbdae` (1 commit, 20 files). Diff: FE 14 files (2× {PageHeader,KpiCard,WidgetCard,OfficeDashboardPage,App.tsx,Layout.tsx,menuKeys.ts} + fe-admin index.css) + BE 2 (MenuKeys.cs + DbInitializer.cs) + 4 agent-memory `.md`. `.tsx`/`.cs`/`.css` non-ignored → full pipeline RAN (the 4 `**/*.md` agent-memory match paths-ignore but co-mixed w/ code → Discovery#3 range any-non-ignored ⟹ whole build). GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (path `C:\inetpub\solution-erp\api`, uid `vrapp`). Run IN-PROGRESS first 4 polls (running 09:26→09:29) — correctly did NOT verify-bundle-mid-flight (anti#3); pre-deploy baseline captured BEFORE poll-loop: admin `CNUv1jxY` (S78 #304 live) + user `CpOskeS1` (S78 #304 live) — both == spec S68 baseline (still live, deploy not yet shipped). Polled iter5 status=success (started ~09:25 → success 09:29:50 ≈4-5m). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **286** baseline (45D+241I) passed; `conclusion` empty — `tasks` endpoint terminal=`status:success` doesn't populate `conclusion`, trust success; 286 INFERRED from gate-passes-pre-build invariant NOT log-confirmed numerically). **★ BUNDLE BOTH ROTATE (FE-both-app + index.css sync ⟹ both MUST rotate; verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no-transient — anti#3): admin ROTATE `CNUv1jxY→Bl2o_kUq`** ✓ (Văn phòng số + index.css shipped) **+ user ROTATE `CpOskeS1→BImrKQNn`** ✓ (foundation shipped). BOTH required → BOTH did. ⚠️ **Prod CI hashes (`Bl2o_kUq`/`BImrKQNn`) ≠ spec local-build (`TbkadgKd`/`DrxDysO7`) — EXPECTED divergence (CI rebuilds w/ different Node/npm/dep-resolution → different content-hash; only matters NEW≠baseline, BOTH rotated confirmed ship). Spec assumption "prod khớp local nếu source-identical" holds ONLY if byte-identical build-env — it is NOT (S77 same lesson).** Smoke **4×200**: api `/health/ready`+`/health/live` + admin root + eoffice root. **NO migration** — prod `__EFMigrationsHistory` top = `20260616035929_AddHoSoLinkToPurchaseEvaluation` (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (`git diff --name-only 764fe70 a8bbdae -- '*Migrations*'` = EMPTY; menu-seed is runtime-idempotent DbInitializer NOT EF-migration → top did NOT advance). **sys.tables=88 verified** (sqlcmd COUNT excl mighist — unchanged, menu-seed inserts rows not tables). **★ MENU-SEED VERIFIED (NEW check this run — DbInitializer seed ungated reaches prod by design): sqlcmd `SELECT [Key],ParentKey,Label,IsVisible FROM MenuItems WHERE [Key]='Off_Dashboard'` → 1 row Key=`Off_Dashboard` Parent=`Off` Label="Bảng điều khiển Văn phòng số" IsVisible=1 ✓.** **★ OFFICE-HIDDEN CONFIRMED (RevokeTemporarilyHiddenModulesAsync StartsWith("Off") scope — DbInitializer.cs L2172): sqlcmd `SELECT r.Name FROM Permissions p JOIN Roles r ON r.Id=p.RoleId WHERE p.MenuKey='Off_Dashboard'` → ONLY `Admin` (CanRead=1), 1 row / 13 roles total ✓ → non-admin NO Off_Dashboard perm → Office stays hidden (revoke executed, admin auto via All[]).** 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "Dashboard landing render / Hồ sơ NS CSS" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88+menu-seed+office-hidden. **LESSON: FE-both-app + index.css-sync + BE-menu-seed (NO-EF-mig) verify = both bundles MUST ROTATE + Mig-top + sys.tables MUST stay prev (menu-seed = DbInitializer runtime row-insert, NOT a migration → does NOT advance __EFMigrationsHistory NOR sys.tables; verify seed via direct MenuItems SELECT not via mig-count). Office-hidden = query Permissions-by-role: temporarily-hidden module has perm row ONLY for Admin (All[] auto-grant) after RevokeTemporarilyHidden runs. ⚠️ Prod CI bundle-hash ≠ local-build-hash is NORMAL — never FAIL on hash-mismatch-to-local, only FAIL if NOT-rotated-from-baseline. TOOLING (re-confirmed S78): Bash=POSIX (`$var`/`$env:`/`''` literal-quote mangle through bash→ssh→PS→sqlcmd layers) → write PS to `.ps1` + `powershell -File "ABS/FWD/SLASH.ps1"`; Gitea `tasks` via `Invoke-RestMethod` (match `head_sha -eq sha`, `?limit=N` honored URI-direct); SSH→PS base64 `-EncodedCommand` (UTF-16LE iconv) for BOTH appsettings-read AND sqlcmd; ⚠️ **sqlcmd string-LITERAL in query: doubled `''x''` BREAKS (PS single-quote-string closes early) → build literal via `[char]39 + "x" + [char]39` concat (NOT `''`)**; this sqlcmd build supports `-U/-P` direct (no -ConnectionString flag); table name `Roles` NOT `AspNetRoles`, `Permissions(RoleId,MenuKey,CanRead)`; CLIXML/Progress stdout-noise grep-filter out. NEVER fixed code (READ-only).** Tag `[s69, run305, pass, fe-both-app-vanphongso-foundation, shared-PageHeader-KpiCard-WidgetCard, index-css-sync-admin, bundle-BOTH-rotate-Bl2o_kUq-BImrKQNn, prod-ci-hash-NE-local-EXPECTED, be-menu-seed-Off_Dashboard-NO-mig, no-mig-top-stays-mig52, tables88-verified, menu-seed-verified-sqlcmd, office-hidden-confirmed-admin-only-1of13, sqlcmd-char39-literal-not-doubled-quote, deploy9of9, no-regression, test286-inferred]`. - **2026-06-16 S78 Run #304 (run_number 304, id418) sha=`37752eb` PASS ~4m (FE BOTH-APP 1-line CSS-precedence fix: "Hồ sơ NS" employee-detail banner NAME đen→trắng — `