solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md

CI/CD Monitor Agent — Persistent Memory

Persistent diary cross-session. Auto-injected first 200 lines / 25KB at spawn. Update BEFORE every stop. Curate when > 25KB.

---

🎯 Role baseline

Read-only CI/CD pipeline + post-deploy verifier for SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch. Output: PASS/FAIL verdict + evidence under 500 words. Spawn cost ~150K tokens — trade-off để catch fail tự động không phụ thuộc em main nhớ verify.

---

🚨 Recurring CI/CD bug patterns (catch with priority)

Gotcha #39 — act_runner github.com TCP timeout

• Symptom: CI run hang ở "Set up job" → timeout 21s, run stays "queued" forever
• Verify: log line Error: dial tcp ... github.com:443 ... i/o timeout
• Fix: manual checkout bypass đã hardcode trong .gitea/workflows/deploy.yml (run #108/#109), pass at #110. KHÔNG revert. Nếu pattern returns → escalate em main check VPS network

Gotcha #40 — npm cache tsc not found

• Symptom: build_fe_admin fail sau khi enable cache: npm ở actions/setup-node@v4
• Verify: log line sh: tsc: command not found hoặc npm error code ETIMEDOUT
• Fix: DISABLED npm cache rolled back ở a21790d. KHÔNG re-enable. Build time chấp nhận ~3 min thay vì optimize

Gotcha #41 — paths-ignore docs-only skip

• Symptom: Commit code thật mà CI không trigger (run list không có entry mới)
• Verify: git diff --name-only HEAD~1 HEAD vs paths-ignore: ['docs/**', '**/*.md', '.claude/skills/**']
• Fix: Nếu commit có code thật bị skip nhầm → check pattern conflict. Nếu commit chỉ docs → expected behavior (saving ~9 min deploy / commit MD-only). Discovery #3 cumulative 4× S23 t3-t6: Gitea evaluates push range commits (not just tip) when at least 1 commit has non-ignored files — anomaly BENEFICIAL catches verify gate.

Gotcha #25 — IIS WebSocket / module exclusion

• Symptom: notification-hub/negotiate returns 401 hoặc 404 prod (FE SignalR connect fail)
• Verify: curl -X POST https://api.solutions.com.vn/notification-hub/negotiate → non-200
• Fix: IIS WebSocket module enable trong web.config của site api.solutions.com.vn (skill iis-deploy-runbook)

Gotcha #48 — Multi-Changelog.Add SQLite tie-break (S25 lesson)

• Symptom: Tests using OrderByDescending(CreatedAt).First() pick wrong audit row khi same SaveChangesAsync transaction có 2+ Changelog.Add() entries cùng CreatedAt (SQLite frozen-clock).
• Verify: Test fail message Expected changelog.ContextNote not to be <null> post BE refactor adding new audit Add() alongside existing LogTransitionAsync chain.
• Fix: Discriminator filter: .Where(c => c.Summary!.Contains("Chuyển phase")) OR .Where(c => c.EntityType == X) BEFORE OrderByDescending. Pattern verified Plan AB Chunk A2 (Run #216 PASS).
• Side benefit: CI test gate catches BEFORE prod deploy → bro UAT spared broken audit. UAT mode skip-test pattern STILL RISKY khi refactor > 100 LOC touches existing test paths.

Deploy ship verification — bundle hash unchanged

• Symptom: commit push success + Gitea action success + status PASS, nhưng prod không có thay đổi visible (user UAT báo "đã deploy mà không thấy")
• Root cause candidates:
  • IIS app pool chưa recycle → giữ assembly cũ trong memory
  • NSSM service script không copy file đúng folder
  • Browser cache (rare nếu Vite hash chuẩn)
• Verify: curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[a-z0-9]+\.js' — hash giữ nguyên = ship fail
• Fix: SSH vietreport-vps "Restart-WebAppPool admin.solutions.com.vn" + recheck bundle hash

Migration drift prod vs repo

• Symptom: Latest mig trong repo (vd Mig 27) nhưng prod chưa có (DbInitializer startup fail)
• Verify: Compare ls Migrations/*.cs vs sqlcmd ... __EFMigrationsHistory
• Fix: Check Program.cs startup hook app.MigrateDatabase() còn không + app pool recycle. Hoặc manual dotnet ef database update --connection prod qua SSH

---

📋 5-stage checklist (apply EVERY run)

Stage 0: Local RAG infrastructure status (S27 NEW — post NSSM Service mode)

• Get-Service Qdrant → must show Status=Running, StartType=Automatic (NSSM Windows Service)
• Nếu Stopped/Paused → Start-Service Qdrant (Admin) hoặc & "D:\.claude-rag\scripts\fix-service-start.ps1" elevated
• Invoke-RestMethod http://localhost:6333/healthz -TimeoutSec 3 verify HTTP

Stage 1: Push happened + filter check

• git log -1 --format='%H %s' — latest commit
• git log origin/main..HEAD — must be empty (synced)
• git diff --name-only HEAD~1 HEAD vs paths-ignore — nếu chỉ docs → SKIPPED-DOCS

Stage 2: Gitea Actions poll (max 10 iter × 60s)

• API: https://git.baocaogiaoduc.vn/api/v1/repos/vietreport-admin/solution-erp/actions/tasks?limit=5 (NOT /runs — 404)
• Match head_sha == $commitSha → get runId
• Status: queued / in_progress / completed
• Conclusion (when completed): success / failure / cancelled / timed_out
• Discovery S21 t5: Gitea API task table caches updated_at stale (~2 min behind reality) — cross-check VPS file mtime nếu time-sensitive.

Stage 3: Test gate verify (Domain 58 + Infra 53 baseline = 111 PASS)

• Logs grep: Passed: line per stage
• Phase 9 UAT exception: test count may be lower nếu em main skip per chunk (memory feedback_uat_skip_verify) — NOT a failure
• Delta from baseline → report

Stage 4: Post-deploy live verify (if SUCCESS)

• Auth login → bearer (admin + nv.test for non-admin gotcha #44 check). Token field = accessToken (NOT token). Route = /api/auth/login (NOT /api/v1/auth/login 404)
• 3-5 endpoint smoke 2XX expected (include endpoint mới trong commit)
• FE bundle hash 2 app changed (compare pre vs post)
• SignalR negotiate (gotcha #25 — if commit relates notification)
• EF migration latest prod == latest repo
• NEW Stage 4.6 (S29 Plan B CRITICAL): sqlcmd seed sample data verify post-deploy — KHÔNG chỉ check schema/Mig table. Phải verify seed data populated khi feature cần sample (V2 workflow Drafter dropdown, etc.). Pattern: sqlcmd ... -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'" → 0 rows post-deploy = seed GATE BLOCKED → escalate gotcha #51 INFRASTRUCTURE vs DEMO seed分类.
• Discovery #4 (S23 t6 Plan P): ASP.NET Core 10 record types với enum fields cần numeric input unless JsonStringEnumConverter registered. SOLUTION_ERP API has NO converter — FE × 2 correctly sends numeric (WorkflowReturnMode = { OneLevel: 1, OneStep: 2, Assignee: 3, Drafter: 4 }).
• Discovery #5 (S25 t3 Plan AC): sqlcmd Windows-auth via ssh requires \\\\SQLEXPRESS 4-backslash escape; \\SQLEXPRESS produces 0 output silently.
• Discovery #6 NEW (S29 Plan B CICD): INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST always run — NOT inside if (!demoSeedDisabled). DEMO seed (DemoUsers/DemoContracts/DemoPE) OK gated. Anti-pattern: Implementer mirror PE V2 seed pattern (which IS gated) for Contract V2 → V2 path BLOCKED prod khi DemoSeed:Disabled=true. Fix: hoist SeedSampleContractWorkflowV2Async out of gate (Run #232 PASS).

Stage 5: Report PASS/FAIL with evidence + MEMORY.md update

---

⚠️ Anti-patterns observed (DO NOT)

1. ❌ Push fix code — READ only, escalate to em main
2. ❌ Speculate fail cause without log evidence
3. ❌ Skip post-deploy live verify khi SUCCESS — bundle hash là biggest catch
4. ❌ Skip MEMORY.md update
5. ❌ Poll forever (max 10 iter ~10 min timeout)
6. ❌ Auto-rollback — escalate với recommendation, KHÔNG tự chạy
7. ❌ Verify khi commit docs-only — SKIPPED-DOCS + return ngay (per Discovery #3 anomaly note)

---

🧠 SOLUTION_ERP CI/CD essentials

• Gitea: https://git.baocaogiaoduc.vn/vietreport-admin/solution-erp
• Workflow: .gitea/workflows/deploy.yml (test gate 2 step + build BE + build FE × 2 + deploy)
• Path filter: paths-ignore: ['docs/**', '**/*.md', '.claude/skills/**'] (gotcha #41 + Discovery #3 anomaly)
• Prod URLs: api / admin / eoffice .solutions.com.vn
• SSH VPS: ssh vietreport-vps (user=Administrator, key=id_ed25519)
• DB prod: .\SQLEXPRESS / SolutionErp / vrapp user. Connection string fallback C:\inetpub\solution-erp\api\appsettings.Production.json khi $env:PROD_DB_PASSWORD empty local (S21 t5 discovery).
• Tests baseline: 120/120 PASS (58 Domain + 62 Infra = baseline 53 + 9 BW1-BW7 Plan C Contract V2 ApproveV2 BW1 happy path + BW2 terminal + BW3 skipToFinal F2 + BW4 outsider Forbidden + BW5 wrong ApplicableType + BW6abc UPSERT Cascade + BW7 V1 fallback). S33 Run #350 PASS confirmed delta +9. Pre-S33: 111/111 unchanged S25→S32.
• Mig latest repo: Mig 34 20260526110207_AddEmployeeProfiles (S33 Plan B G-H1 Phase 10.1 — 7 Hrm entity tables + EmployeeCodeSequences atomic NV/{YYYY}/{D4}). Prev Mig 33 AddContractLevelOpinions (S29 Plan B). Prev Mig 31 RefactorSkipToFinalToApproverLevel (S23 t1 Plan K) — F2 swap Users → ApprovalWorkflowLevels per-Approver-slot, preserved.
• Mig latest prod: sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5
• Bearer test:
  • Admin: admin@solutions.com.vn / Admin@123456 (full)
  • UAT non-admin: nv.test@solutions.com.vn / TestUser@123456 (Drafter CCM — verify gotcha #44 silent 403 patterns)

---

🔑 Critical config (gotcha cross-ref)

• Node CI pin: 20.x (memory feedback_node_cicd — bài học NamGroup)
• MediatR pin: 12.4.1 (gotcha #1)
• Swashbuckle pin: 6.9.0 (gotcha #2)
• act_runner: manual checkout bypass github.com (gotcha #39)
• npm cache: DISABLED (gotcha #40 — KHÔNG re-enable)

Flag commit nếu thấy <PackageReference Include="MediatR" Version="14... hoặc cache: npm tái xuất hiện.

---

🎯 Per-NV admin opt-in wire surface — 10-point checklist (cumulative S22+5 → S23 t6)

Cross-ref memory feedback_per_nv_permission_scope.md. Future per-NV/per-Level refactor MUST verify:

1. Domain entity field
2. EF config HasDefaultValue(false)
3. Migration 3-file rule (Up + Designer + Snapshot)
4. Service handler read field
5. Domain DTO + Application DTO mirror
6. Designer FE checkbox inline (admin)
7. Admin overview AwLevelDto record + ToDto ctor
8. CreateAwLevelInput record + Update mutation handler
9. Lookup discrimination in handler (FirstOrDefault ADD ApproverUserId == actorId filter + admin fallback)
10. Controller body record mirror count check ([FromBody] record param count = Command record param count)

Bug latency observed when miss points 9-10: 2-3 days prod silent (Mig 28-29 deploy → S23 t4-t6 catch). Prophylactic codebase scan recommended: grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N schema refactor.

---

📊 Run stats baseline (cumulative)

• Build time BE (test_domain + test_infra + build_be): ~90s baseline
• Build time FE × 2 app: ~60s baseline mỗi app
• Deploy NSSM + IIS recycle: ~30s
• Total CI run time: ~3 min code commit / 0s docs-only commit
• Trend trigger: nếu run time > 5 min → escalate (cluster network slow hoặc dependency bloat)
• Bundle size baseline: fe-admin ~800KB gz / fe-user ~750KB gz (Vite production build)

---

📅 Recent runs (FIFO — slim post-curate 2026-05-22)

• 2026-05-28 15:06-15:10 Run #359 (task 359) sha=f45090b VERDICT=PASS ~3m55s (S36 Plan G-O2 Phòng họp Mig 36 + BE CRUD + FE 2 app): Push range 8afdc1e..f45090b 1 commit (scope Domain+App+Infra+Api+FE-Admin+FE-User full-stack). Stage results ALL PASS (task #359 status=success, dur 3m55s computed 15:06:37→15:10:32, slightly over baseline 3m30s due to Mig 36 + BE CRUD + FE × 2 cookie-cutter): test gate 120/120 baseline preserved + build_be (MeetingRoom/MeetingBooking entity + Mig 36 + Application Features compile OK) + build_fe_admin + build_fe_user (BOTH rotate). Post-deploy verify ALL PASS Stage 4: (a) 3 prod domain smoke ALL 200 OK: api/health/ready 200/0.33s + admin 200/0.24s + eoffice 200/0.27s. (b) Bundle hash BOTH rotated: fe-admin Bl6e54yi→C9kzTTmq + fe-user DHmW2tUF→CC4DQ-Tr (Plan G-O2 ship confirmed FE × 2 app cookie-cutter). (c) Mig 36 prod TOP 5 DESC head MATCHES repo: 20260528074125_AddMeetingRooms (Mig 36 head) → AddHrmConfigs (35) → AddEmployeeProfiles (34) → AddContractLevelOpinions (33) → AddApprovalWorkflowToContract (32). (d) 2 NEW endpoint smoke ALL 200 OK: GET /api/meeting-rooms 200/0.26s → 4 rows (ONL-1 Trực tuyến Zoom cap=50 + PH-A Phòng họp lớn cap=20 + PH-B Phòng họp nhỏ cap=8 + PHG-501 Phòng Giám đốc cap=6) + GET /api/meeting-bookings?roomId=<first>&startDate=2026-05-28&endDate=2026-06-04 200/0.29s → 0 rows (no booking yet, expected). (e) MeetingRooms seed verify EXACT MATCH spec: COUNT=4 + Code list ONL-1/PH-A/PH-B/PHG-501 all present (DbInitializer SeedMeetingRoomsAsync auto-applied on startup). (f) Menu seed verify EXACT MATCH spec: 4 row Off_PhongHop% (root Off_PhongHop Order=2 ParentKey=Off + 3 leaf Off_PhongHop_View Order=1 + Off_PhongHop_Manage Order=2 + Off_PhongHop_Book Order=3). 0 prod regression observed Run #359. Polling pattern WORKED: until-loop 7 iter × 30s = 3m30s wait from running → success transition (15:06:37 created → 15:10:32 updated, smooth deploy). Token cost ~11K (Read MEMORY + git log + Gitea API task poll × 8 + 3 smoke + 2 endpoint + 3 sqlcmd + 1 bundle hash + 1 auth login + tmp/mr.json parse). Tag: [s36, run359, pass, mig-36, meeting-rooms-bookings, full-stack]. Pattern 12-bis cross-module mirror PE V2 → Meeting Room CRUD confirmed strong: BE+Mig+FE×2 1 commit cohesive, 4 sample seed clean, menu hierarchy 4 row exact match.

• 2026-05-28 S36 startup VERIFY (em main spawn read-only post-S35 wrap) sha=8afdc1e (HEAD docs only): S35 wrap complete — 5 commits push range 63dd9ec..8afdc1e. Gitea API last 5 tasks (limit=10) all status=success: task #358 sha=021674a (Run #244 — S35 Plan G-H2 Task 4 declarative HrmConfigsPage, 2026-05-28 10:01-10:05, dur=218s = 3m38s) + task #357 sha=909655c (Run #243 — S35 Plan G-H2 Task 3 BE CRUD 16 endpoint, 09:51-09:55, 223s = 3m43s) + task #356 sha=c3cd343 (Run #242 — S35 FE inline forms 5 satellite, 09:39-09:43, 232s = 3m52s) + task #355 sha=07b3f3b (Run #241 — S34 Plan 4 G-H2 Mig 35 foundation, 2026-05-27 15:04-15:08, 210s = 3m30s) + task #354 sha=e506cd8 (Run #240 — S34 Plan 3 Item 3 BE satellite CRUD, 14:56-15:00, 216s = 3m36s). Token API unauth quirk: TokenLen=0 reported but Total=244 workflow_runs returned (anonymous read OK for public repo metadata — gotcha confirmed). Final 2 docs commits 4d50b34 + 8afdc1e correctly SKIPPED CI per gotcha #41 path filter **/*.md (no task #359 in list — verified). VERDICT=PASS Stage 4 cumulative S35 healthy: (a) 3 prod domain smoke ALL 200 OK: api.solutions.com.vn/health/ready 200/0.37s + admin.solutions.com.vn 200/0.50s + eoffice.solutions.com.vn 200/0.31s. Note /api/health (per spec line) returns 404 — canonical path is /health/ready (gotcha S32 already documented). (b) Bundle hash live prod EXACT MATCH S35 STATUS expected: fe-admin index-Bl6e54yi.js ✓ + fe-user index-DHmW2tUF.js ✓ (Plan G-H2 Task 4 declarative HrmConfigsPage Run #244 ship confirmed end-to-end). (c) Qdrant RAG infrastructure live: Service Qdrant Status=Running StartType=Automatic (NSSM Windows Service per S27 fix preserved) + http://localhost:6333/healthz 200 "healthz check passed" + collection name CORRECTED proj_solution_erp (NOT solution_erp — naming convention proj_* cross-project: ashico_erp/benhvien_aau/dh_y_duoc/namgroup_main/solution_erp/vipix_ai_infra/vipix_multisite total 7 collections registered) POINTS=3076 STATUS=green optimizer_status=ok (S31 baseline 2988 → +88 chunks S32-S35 cumulative content growth — Plan G-H1 EmployeeProfiles + Plan G-H2 HrmConfig + Pattern 12-bis/12-ter MEMORY docs + S33-S35 STATUS/HANDOFF updates). (d) TLS cert not yet probed this spawn (openssl s_client not invoked — defer per S32 baseline notAfter 2026-07-23 ~56 days remaining + auto-renew win-acme 30d window ~2026-06-23). 0 prod regression observed S35 cumulative end-to-end (3 deploy Run all PASS, all post-deploy smoke green, bundle hash match expected, RAG infrastructure live). DISCOVERY #8 NEW collection naming: RAG collection prefix proj_* standardized across 7 projects on this VPS — future skill cicd-monitor Stage 0 should reference proj_solution_erp not solution_erp. Update Stage 0 spec in agent system prompt. STANDBY STATE ANNOUNCED: Awaiting S36 push trigger. Token cost S36 startup ~14K (Read MEMORY × 2 + 4 Invoke-RestMethod Gitea API + 4 Invoke-WebRequest smoke + 3 Qdrant probe + 2 bundle hash grep + cache fallback tasks.json). Tag: [s36, startup, standby, discovery-8-rag-naming].

• 2026-05-28 09:51-09:55 Run #243 (task 357) sha=909655c VERDICT=PASS ~3m43s (S35 Plan G-H2 Task 3 — HrmConfig BE CRUD 16 endpoint cookie-cutter): Push range c3cd343..909655c 1 commit 2 BE file NEW (+576 LOC, 0 mod): HrmConfigFeatures.cs (439 LOC Application/Hrm CQRS — 16 Command+Query handler 4 catalog × 4 verb: LeaveType/Holiday/Shift/OtPolicy × Create/Update/Delete/GetList) + HrmConfigsController.cs (137 LOC Api 16 endpoint route /api/hrm-configs/*). Stage results ALL PASS (Gitea task API status=success, conclusion=None UI bug per Run #237/#238/#350 pattern, duration computed 09:51:26→09:55:09 = 3m43s): test_domain 58 + test_infra 62 baseline 120/120 UNCHANGED (no test add per Phase 9 UAT mode per feedback_uat_skip_verify) + build_be (HrmConfigFeatures + Controller compile OK — Application/Hrm namespace existed from S33 EmployeeFeatures, MediatR auto-discovery WORK no gotcha #1) + build_fe_admin + build_fe_user (BOTH no rebuild — 0 FE file in push, bundle hash preserved expected). Post-deploy verify ALL PASS Stage 4: (a) Bundle hash BOTH UNCHANGED: fe-admin BhR0MmLN (== Run #242 baseline) + fe-user DIdNaB6W (== Run #242 baseline) — CRITICAL anti-pattern verify PASS: BE-only diff means FE bundle untouched, ship correct (no spurious FE rebuild). (b) 3 prod domain smoke ALL 200 OK: api/health/ready 200 0.16s + admin 200 0.11s + eoffice 200 0.11s. (c) 4 NEW HrmConfig GET endpoint smoke ALL 200 + EXACT row count match S34 seed: /api/hrm-configs/leave-types 200 → 5 rows (ANNUAL/COMPASSIONATE/MATERNITY/SICK/UNPAID — match LeaveTypes=5 S34 Mig 35 seed) + /api/hrm-configs/holidays?year=2026 200 → 10 rows (Tết DL Jan 1 + 5 Tết NĐ Feb 16-20 + Giỗ Tổ Apr 14 + 30/4 + 1/5 + Quốc Khánh Sep 2 — match Holidays=10 S34 VN 2026 seed) + /api/hrm-configs/shifts 200 → 3 rows (CA1/CA2/HC — match ShiftPatterns=3 S34) + /api/hrm-configs/ot-policies 200 → 1 row (STANDARD — match OtPolicies=1 S34 Luật LĐ VN 2019). Total wire BE 4 GET endpoint × 19 row prod live = wire BE claim verify ✓ — 16 endpoint route [Authorize] class-level WORK (admin bearer 200, no silent 403 gotcha #44). (d) Mig 35 prod TOP 3 DESC UNCHANGED: AddHrmConfigs (Mig 35 head) → AddEmployeeProfiles (Mig 34) → AddContractLevelOpinions (Mig 33) — no new mig push (BE CRUD only, expected). (e) Polling pattern Run #242 lesson APPLIED: at +3m24s push (curl probe 09:54:53) status="running" → waited via Python parse loop to verify status=success BEFORE bundle hash + endpoint smoke. Run #242 false-positive trap avoided. 0 prod regression observed Run #243. Pattern 12-bis cookie-cutter cross-module mirror PE V2 ApproveV2 → Hrm CRUD strong: 4 catalog × 4 verb pattern uniform clean. Cumulative S35 deploy: 2× Run (#242 FE inline forms + #243 BE CRUD 16 endpoint). Expect more S35 kicks if FE 2 app page wire (list/create/edit dialog 4 catalog) added per Plan G-H2 Task 4. Token cost ~11K (Read MEMORY + git log + git diff + Gitea API task × 2 + 1 ssh sqlcmd + 7 curl smoke + parse Python). Tag: [s35, run243, pass, be-only, hrm-configs-crud, 16-endpoint]. Anti-pattern verify ROI: BE-only diff bundle UNCHANGED check catches spurious FE rebuild scenarios (e.g. shared Vite config touch, dependency cache bust). Run #243 confirmed clean — no spurious rebuild.

• 2026-05-28 09:39-09:42 Run #242 sha=c3cd343 VERDICT=PASS ~3m30s (S35 Phase 1.5 Item 3-FE inline forms 5 satellite × 2 app cookie-cutter): Push range 63dd9ec..c3cd343 1 commit 4 FE files (+1560/-198 LOC): fe-admin EmployeesListPage.tsx 573→1200 +627 + fe-user mirror SHA256 IDENTICAL + types/employee.ts × 2 app +53 LOC each. Stage results ALL PASS (Gitea Actions list page green-check icon "Success", duration ~3m30s): test_domain 58 + test_infra 62 baseline 120/120 UNCHANGED (no test add — FE-only Phase 9 UAT mode per feedback_uat_skip_verify) + build_be (unchanged BE — only FE diff) + build_fe_admin + build_fe_user (BOTH bundle rotate confirmed) + deploy NSSM IIS recycle. Post-deploy verify ALL PASS Stage 4 + VPS mtime cross-check: (a) Bundle hash BOTH rotated: fe-admin RNTX6Fvo→BhR0MmLN + fe-user 29A1LuMm→DIdNaB6W (Item 3-FE inline forms 5 satellite ship confirmed FE × 2 app cookie-cutter). CSS hash also rotated (admin DQSzWczW→Dlk3cj6D + user D-PasqRK→25W039gj). (b) VPS mtime cross-check: C:\inetpub\solution-erp\fe-admin\ assets folder = 2026-05-28 09:42:32 AM + index.html = 09:41:37 AM + fe-user\ assets = 09:42:32 + index.html = 09:42:27 — match Run #242 deploy 09:42, NOT stale Run #241 (was 2026-05-27 15:07:28). NSSM service Restart-WebAppPool + xcopy ship verified end-to-end. (c) 3 prod domain smoke ALL 200 OK: api/health/ready 200 "Healthy" 0.32s + admin 200 0.32s + eoffice 200 0.31s. (d) 3 endpoint smoke 200 OK via admin bearer: /api/contracts 200 (7 demo HĐ rows w/ maHopDong + phase) + /api/employees?page=1&pageSize=5 200 (5 NV/2026/0001-0033 sample w/ employeeCode + departmentName populated — confirm Mig 34 EmployeeProfile seed 33 rows still applied) + /api/menus 200 (84 menu items including 5 Hrm_Config* + 2 Off* + new Hrm_HoSo). Token = accessToken (NOT token — gotcha preserved). (e) Mig 35 prod TOP 1 UNCHANGED = no BE Mig in push (FE-only diff). Plan 4 Phase 1.5 Item 3-FE inline forms + Thêm button right side header satellite cookie-cutter mirror pattern FE × 2 app WORK end-to-end + Phase 9 UAT defer BE wire smoke 1 week per em main decision. Polling pattern observed: at +2 min push (09:41) Run #242 status="Running" yellow spinner — premature curl at that point showed bundle hash STALE (RNTX6Fvo/29A1LuMm baseline pre-rotate) which would have been false-positive FAIL ship had I stopped polling. Until-loop pattern via Monitor tool WORK — waited ~2 additional min for green check before final bundle hash re-fetch. Lesson: bundle hash verify MUST happen AFTER status=Success confirmed, NOT immediately after push. 0 prod regression observed Run #242. Pattern 12-bis cross-module mirror PE → Hrm cookie-cutter strong (S33 Hrm EmployeeProfile module 7 entity → S35 Hrm inline forms 5 satellite × 2 app + 4 file diff only). Cumulative S35 deploy: 1× Run (#242), expect more S35 kicks if BE wire endpoints CRUD satellite added (Phase 9 UAT 1 week defer). Token cost ~12K (Read MEMORY + git log + 1 ssh + 1 ssh mtime cross-check + 1 Atom feed + 4 curl smoke + 1 Monitor until-loop). Tag: [s35, run242, pass, fe-only, hrm-inline-forms]. Smart-friend pre-commit clean 7× confirmed (Implementer Reviewer gate Phase 9 UAT mode — test gate baseline 120/120 unchanged, no new test added FE-only diff).

• 2026-05-28 S35 warm-up VERIFY (em main spawn read-only post-S34 wrap) sha=63dd9ec (HEAD docs only, deploy commit 07b3f3b Run #241): S34 wrap complete — 7 commits push range edba4ae..63dd9ec, 4 CI Run all SUCCESS (#238 sha=ea440da Plan 2 G-O1 Danh bạ + #239 sha=61e9ce5 Plan 3 batch 4 item + #240 sha=e506cd8 Plan 3 Item 3 BE satellite CRUD + #241 sha=07b3f3b 2026-05-27 15:04-15:07 ~3m30s Plan 4 G-H2 Mig 35 schema foundation deploy). S34 final 2 docs commits 1849197 + 63dd9ec correctly SKIPPED CI per gotcha #41 path filter **/*.md. VERDICT=PASS — Run #241 deploy fully shipped + verified post-deploy: (a) Run #241 status=success duration 3m30s (Gitea API task workflow_runs query — created 15:04:19, updated 15:07:49 +07:00). (b) 3 prod domain smoke ALL 200 OK: api.solutions.com.vn/health/ready returns "Healthy" 200/0.47s + admin.solutions.com.vn 200/0.33s + eoffice.solutions.com.vn 200/0.22s. (c) Mig 35 prod TOP 5 DESC sqlcmd via ssh vietreport-vps 'sqlcmd -S ".\SQLEXPRESS" -d SolutionErp -E -h -1 -Q "..."' pattern: 20260527075940_AddHrmConfigs (Mig 35 head MATCHES repo) → AddEmployeeProfiles (Mig 34) → AddContractLevelOpinions (Mig 33) → AddApprovalWorkflowToContract (Mig 32) → RefactorSkipToFinalToApproverLevel (Mig 31). (d) 4 HRM catalog table verify: LeaveTypes=5 + Holidays=10 + ShiftPatterns=3 + OtPolicies=1 = 19 row total EXACT MATCH spec (ANNUAL/SICK/MATERNITY/COMPASSIONATE/UNPAID + VN 2026 holidays + HC/CA1/CA2 shifts + STANDARD OT policy Luật LĐ VN 2019). (e) Menu seed verify GOOD — Plan 4 spec wrote HrmConfig% but actual key uses underscore Hrm_Config%: query SELECT [Key], Label, [Order], ParentKey FROM MenuItems WHERE [Key] LIKE 'Hrm%' returns 7 row: root Hrm (Order=28 ParentKey=NULL) + Hrm_HoSo (Order=1 ParentKey=Hrm) + Hrm_Config (sub-group Order=2 ParentKey=Hrm) + 4 leaf under Hrm_Config Order 1-4 (Hrm_Config_LeaveTypes "Loại phép" + Hrm_Config_Holidays "Ngày lễ" + Hrm_Config_Shifts "Ca làm việc" + Hrm_Config_OtPolicies "Chính sách OT"). 5 G-H2 menu rows present (sub-group + 4 leaf) — Pattern 16-bis MenuKeys.cs sync OK BE+FE. Note key naming discrepancy: S35 task spec said 'HrmConfig%' (no underscore) but BE seeded 'Hrm_Config%' matching MenuKeys.cs constants — verified by reading commit 07b3f3b message body. Recommend STATUS.md S35 task description fix key prefix. (f) Bundle hash S35 BASELINE snapshot for next push compare: fe-admin = /assets/index-RNTX6Fvo.js + fe-user = /assets/index-29A1LuMm.js. Plan 4 G-H2 was BE-only (4 entity + Mig 35 + DbInitializer seed + MenuKeys both apps mirror) — FE 2 app bundle Plan 4 commit 07b3f3b rotated FROM ChA9_vP5/DCpX7akt (Run #238 baseline) THROUGH #239 #240 to current RNTX6Fvo/29A1LuMm. (g) Defer S35 Implementer Case 2 cookie-cutter task pending: Task 2 BE CQRS 4 catalog CRUD (16 endpoint) + Task 4 FE 2 app 4 catalog page (list/create/edit dialog). Em main spawn cicd-monitor next when first S35 G-H2 wire push hits. 0 prod regression observed S34 cumulative end-to-end (4 deploy Run all PASS, all post-deploy smoke green). STANDBY STATE ANNOUNCED: Awaiting S35 push trigger Plan 4 Task 2+4 wire (mirror pattern Plan 2 G-O1 BE Controller + Application Features + FE 2 app cookie-cutter — bundle rotate expected ×2 + 4 new endpoint smoke /api/leave-types + /api/holidays + /api/shifts + /api/ot-policies GET list verify). Token cost S35 warm-up ~15K (Read MEMORY + git log + Gitea API task + 3 curl smoke + 4 sqlcmd via ssh + 1 grep bundle hash + git show commit detail). Tag: [s35, warm-up, standby, mig-35].

• Archived 7 verbose Run entries → archive/2026-05-q4.md 2026-05-28 S36 curate (em main proxy): Run #238 (G-O1 Danh bạ S34) + Run #237 (Plan B Phase 2 S33) + Run #350 (Plan B G-H1 Mig 34 S33) + S33 startup + S32 wrap + S32 startup + S26 Run #222-#227 polish. KEY takeaways absorbed forward in baseline section: test 120→130, Mig 34→35, bundle hash baselines + Discovery #7 path filter eval/** missing + Discovery #8 collection naming proj_* + gotcha #51 INFRASTRUCTURE seed lesson preserved in Stage 4.6 + 10-surface-point checklist intact.

• 2026-05-22 (S29 wrap — Run #229-#232 verify PASS + Plan B CICD CRITICAL DemoSeed gate catch — gotcha #51 NEW INFRASTRUCTURE vs DEMO): Run #229 PASS Plan CA + Run #230 PASS Hotfix 1 resolvePath staticMap + Run #231 PASS Plan B Contract V2 Mig 32+33 + Run #232 sha=38f1c4d PASS Hotfix CICD SeedSampleContractWorkflowV2 out of DemoSeed gate — CRITICAL CATCH agentId a2ea2e3a5dbe271b5 ~90K: nested inside if (!demoSeedDisabled) DbInitializer.cs:105-111, prod DemoSeed:Disabled=true → seed SKIP → QT-HD-V2-001 KHÔNG tồn tại prod → Drafter Workspace dropdown V2 EMPTY → V2 contract path BLOCKED end-to-end UAT. Patterns proven NEW: Discovery #6 INFRASTRUCTURE vs DEMO seed phân biệt → gotcha #51. Smart Friend ROI 4× cumulative: S22 #44 + S25 #48 + S29 Plan B Reviewer ApplicableType + S29 Plan B CICD DemoSeed gate.

• Archived to archive/2026-05-q3.md 2026-05-27 S34 curate (em main proxy): Run #215+#216 (gotcha #48 SQLite tie-break catch+fix pair, S25 t1-t2) — full detail preserved in archive. KEY lesson cumulative trong foundation gotcha #48 line 40-45 (test discriminator filter Summary contains "Chuyển phase"). 2026-05-13 S22 chốt cuối verify (Run #214 baseline + Discovery #3 docs-only skip). 2026-05-12 setup baseline.

---

🔄 Curate trigger

• Memory size > 25KB → archive recent runs to archive/<period>.md
• Duplicate failure patterns → merge into single entry (vd act_runner timeout x3 → 1 entry)
• Stale > 3 months → remove

Last curate: 2026-05-27 S34 em main proxy curate (post-S33 wrap, sequence 1/4) — archived Run #215+#216 verbose detail (gotcha #48 catch+fix pair, KEY lesson preserved in foundation line 40-45) + 2026-05-13 S22 verify + 2026-05-12 setup → archive/2026-05-q3.md. KEEP in MEMORY: Run #237 + Run #350 (S33 prod deploy), S33 startup health, S32 wrap + S32 startup, S29 wrap, S28 wrap, S26 Run #222-#227, S22 curate. Foundation 10-surface-point per-NV checklist + Stage 0-5 checklist + Stage 4.6 sqlcmd seed verify + Discovery #6 INFRASTRUCTURE vs DEMO seed + gotchas #39-#48 detail all preserved untouched. MEMORY size before: 32.9 KB → after: target ~22-24 KB. Previous curate: 2026-05-26 S32 — Run #231 PARTIAL detail → archive/2026-05-q2.md. Previous curate: 2026-05-22 — Run #186-#221 → archive/2026-05-runs.md. Next trigger: > 25KB OR Plan G-O1 Danh bạ kick off.