# CI/CD Monitor Agent — Persistent Memory > **Persistent diary cross-session.** Auto-injected first ~200 lines at spawn (L1 HOT). > Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 `archive/` on-demand · L3 RAG `search_memory` just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). > Full verbatim run history pre-S40 → git `d2f52ba` + `archive/2026-05-{runs,q2,q3,q4}.md`. --- ## 🎯 Role baseline Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: `iis-deploy-runbook` + `dependency-audit-erp` + `ef-core-migration`. Spawn ~150K — trade-off catch fail tự động. --- ## 🚨 Recurring CI/CD bug patterns (catch priority) - **#39 act_runner github.com TCP timeout** — run hang "Set up job" 21s. Log `dial tcp github.com:443 i/o timeout`. Fix: manual checkout bypass hardcoded `.gitea/workflows/deploy.yml` (pass #110). KHÔNG revert. - **#40 npm cache `tsc not found`** — `build_fe_admin` fail post `cache: npm`. DISABLED rolled back `a21790d`. KHÔNG re-enable. - **#41 paths-ignore docs-only skip** — code commit không trigger CI? Check `git diff --name-only HEAD~1 HEAD` vs `paths-ignore: ['docs/**','**/*.md','.claude/skills/**']`. Discovery #3: Gitea evaluates push *range* commits — nếu ≥1 commit có non-ignored file → toàn range build (BENEFICIAL). - **#25 IIS WebSocket** — `notification-hub/negotiate` 401/404 prod. Fix: WebSocket module enable `web.config` site api (skill `iis-deploy-runbook`). - **#48 SQLite tie-break** — `OrderByDescending(CreatedAt).First()` pick wrong khi 2+ `.Add()` cùng frozen-clock. Fix: discriminator filter `.Where(Summary.Contains("Chuyển phase"))` BEFORE OrderBy. - **Bundle hash unchanged = ship FAIL** — push+action success nhưng prod không đổi. Verify `curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[a-z0-9]+\.js'`. Fix: SSH `Restart-WebAppPool`. ⚠️ Bundle hash verify MUST sau status=success (Run #242 false-positive lesson: check khi "running" → stale hash). - **Migration drift prod vs repo** — compare `ls .../Persistence/Migrations/*.cs` vs `sqlcmd __EFMigrationsHistory`. Fix: check `Program.cs` `app.MigrateDatabase()` + app pool recycle. --- ## 📋 5-stage checklist (EVERY run) - **Stage 0 RAG infra:** `Get-Service Qdrant` Running + `http://localhost:6333/healthz`. Collection `proj_solution_erp` (prefix `proj_*` 7 project — Discovery #8). - **Stage 1 Push+filter:** `git log -1 --format='%H %s'` + `git log origin/main..HEAD` empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return). - **Stage 2 Gitea poll** (max 10 iter × 60s): API `.../actions/tasks?limit=5` (NOT `/runs` 404). Match `head_sha`. ⚠️ task table `updated_at` stale ~2min (gotcha #46) → cross-check VPS mtime. - **Stage 3 Test gate:** baseline **130 PASS** (58 Domain + 72 Infra). Phase 9 UAT exception lower OK (`feedback_uat_skip_verify`). - **Stage 4 Post-deploy** (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=`accessToken` route `/api/auth/login`) → 3-5 endpoint smoke 2XX (incl new) → FE bundle hash 2 app changed → SignalR negotiate (gotcha #25 if relevant) → EF mig prod==repo. - **Stage 4.6 (S29 CRITICAL):** sqlcmd seed sample verify post-deploy (NOT chỉ schema). `sqlcmd -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'"` → 0 rows = seed GATE BLOCKED → gotcha #51. - Discovery #4: ASP.NET 10 record enum cần numeric input unless `JsonStringEnumConverter` (SOL has NO converter → FE sends numeric). #5: sqlcmd ssh Windows-auth cần `\\\\SQLEXPRESS` 4-backslash. #6: INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST run, NOT inside `if(!demoSeedDisabled)`; DEMO seed (DemoUsers/Contracts/PE) OK gated → gotcha #51. - **Stage 5 Report** PASS/FAIL + evidence + MEMORY update. --- ## ⚠️ Anti-patterns (DO NOT) 1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay) --- ## 🧠 SOLUTION_ERP CI/CD essentials (S40 verified) - **Gitea:** `git.baocaogiaoduc.vn/vietreport-admin/solution-erp` · workflow `.gitea/workflows/deploy.yml` · paths-ignore `['docs/**','**/*.md','.claude/skills/**']` - **Prod:** api/admin/eoffice `.solutions.com.vn` · SSH `ssh vietreport-vps` (Administrator, id_ed25519) · IIS site phys paths (S42 verified): API `C:\inetpub\solution-erp\api` · admin `\fe-admin` · user `\fe-user` (3 sites Started). DB `.\SQLEXPRESS`/`SolutionErp`/`vrapp` SQL-auth. **Conn string key = `ConnectionStrings.Default` (NOT `DefaultConnection`!)** — read pw from prod appsettings.Production.json when `$env:PROD_DB_PASSWORD` empty. - **SSH→PS quoting (S42 lesson):** nested bash→ssh→powershell mangles `$var`/`\"`. Use `iconv UTF-16LE | base64` → `powershell -EncodedCommand $B64`. Single-quote literal paths. - **Tests baseline:** **263 PASS** (S62 Run #286 sha 7926c21 spec; 45 Domain + 218 Infra — em-main supplied; supersedes prev 228/240/256). CI gate runs both test projects BEFORE build/deploy → status=success ⟹ test gate passed (`tasks` endpoint reports terminal as `status:success`, `conclusion` field NOT populated). Local grep undercounts (Theory/InlineData) — trust CI conclusion. Phase 9 UAT mode skip per chunk OK. - **Mig latest repo:** **Mig 52 `20260616035929_AddHoSoLinkToPurchaseEvaluation`** (S65; PE +HoSoLink hyperlink-NAS, AddColumn-only no new table). Prev Mig 51 `AddDepartmentParentId` (S65 Department.ParentId loose-Guid org-tree, AddColumn-only) + Mig 50 `ReplaceBudgetModuleWithPeWorkItemBudgets` (S61 Budget→PeWorkItemBudgets net-reduce). Path `src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/` (52 mig .cs non-designer total). Prod check `sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5`. ⚠️ Table-count: `sys.tables` (is_ms_shipped=0, excl mighist) = **88** (S62 Run #286 verified — S61 Budget-replace DROPPED tables 93→88). Narrative-93 is STALE pre-S61 — when commit touches no schema, 88 is correct, don't FAIL on 88↔93. Always cross-ref COMMIT scope vs ambient count. - **Bearer:** admin `admin@solutions.com.vn/Admin@123456` (full) · UAT `nv.test@solutions.com.vn/TestUser@123456` (Drafter CCM, gotcha #44 check) - **Bundle hash live S77:** admin `D532XZKG` (Run #303 sha 6983609 — ROTATED from `CcrZqfht`; EmployeesListPage employee-detail banner text-polish) · user `CuFaBoWt` (Run #303 sha 6983609 — ROTATED from `DniDFUB_`; same page both apps SHA256-identical `F013B748`). Prev live admin `CcrZqfht`/user `DniDFUB_` (Run #302 S76). ⚠️ ASYMMETRIC-deploy lesson (S66): FE-one-app commit → that app's bundle MUST rotate + OTHER app MUST stay frozen; admin-rotate-when-only-fe-user-changed = anomaly → flag. S50 mid-deploy transient lesson: pre-success snapshot can show intermediate FE copy in-flight — re-confirm hash AFTER status=success ALWAYS (anti-pattern #3). FROZEN-expectation runs (BE-only or other-app): hash MUST stay = live pre-deploy value; rotate w/o relevant FE change = anomaly. - **DB pw (S42, when `$PROD_DB_PASSWORD` empty):** `vrapp/buKL3TGBkD0wDDbYVw65QeX9` read from `C:\inetpub\solution-erp\api\appsettings.Production.json`→`ConnectionStrings.Default`. ⚠️ Skill-doc path `C:\inetpub\apps\SolutionErp\Api` is STALE → real path `C:\inetpub\solution-erp\api`. sqlcmd over SSH works direct (no UTF-16 encode needed). ⚠️ sys-catalog string-concat queries hit collation conflict (`Latin1_General_CI_AS_KS_WS` vs `SQL_Latin1_General_CP1_CI_AS`) → add `COLLATE DATABASE_DEFAULT` per concatenated column. ## 🔑 Critical config (flag commit nếu tái xuất) Node CI `20.x` (`feedback_node_cicd`) · MediatR `12.4.1` (gotcha #1, flag `Version="14`) · Swashbuckle `6.9.0` (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag `cache: npm`) --- ## 🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23) Cross-ref `feedback_per_nv_permission_scope`. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF `HasDefaultValue(false)` · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 **Lookup discrimination** (`FirstOrDefault` ADD `ApproverUserId==actorId` + admin fallback) · 10 **Controller body record count == Command record count**. Bug latency 2-3 days prod silent khi miss 9-10. Scan `grep -n "FirstOrDefault.*Order.*==" *.cs` after OR-of-N refactor. ## 📊 Run stats baseline BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · **total ~3min code / 0s docs-only**. >5min → escalate. --- ## 📅 Recent runs (FIFO — older → archive/git) - **2026-06-16 S78 Run #304 (run_number 304, id418) sha=`37752eb` PASS ~4m (FE BOTH-APP 1-line CSS-precedence fix: "Hồ sơ NS" employee-detail banner NAME đen→trắng — `