solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md

# CI/CD Monitor Agent — Persistent Memory

> **Persistent diary cross-session.** Auto-injected first ~200 lines at spawn (L1 HOT).
> Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 `archive/` on-demand · L3 RAG `search_memory` just-in-time. Keep entry ≤ 1.5K chars (gotcha #53).
> Full verbatim run history pre-S40 → git `d2f52ba` + `archive/2026-05-{runs,q2,q3,q4}.md`.

---

## 🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: `iis-deploy-runbook` + `dependency-audit-erp` + `ef-core-migration`. Spawn ~150K — trade-off catch fail tự động.

---

## 🚨 Recurring CI/CD bug patterns (catch priority)

- **#39 act_runner github.com TCP timeout** — run hang "Set up job" 21s. Log `dial tcp github.com:443 i/o timeout`. Fix: manual checkout bypass hardcoded `.gitea/workflows/deploy.yml` (pass #110). KHÔNG revert.
- **#40 npm cache `tsc not found`** — `build_fe_admin` fail post `cache: npm`. DISABLED rolled back `a21790d`. KHÔNG re-enable.
- **#41 paths-ignore docs-only skip** — code commit không trigger CI? Check `git diff --name-only HEAD~1 HEAD` vs `paths-ignore: ['docs/**','**/*.md','.claude/skills/**']`. Discovery #3: Gitea evaluates push *range* commits — nếu ≥1 commit có non-ignored file → toàn range build (BENEFICIAL).
- **#25 IIS WebSocket** — `notification-hub/negotiate` 401/404 prod. Fix: WebSocket module enable `web.config` site api (skill `iis-deploy-runbook`).
- **#48 SQLite tie-break** — `OrderByDescending(CreatedAt).First()` pick wrong khi 2+ `.Add()` cùng frozen-clock. Fix: discriminator filter `.Where(Summary.Contains("Chuyển phase"))` BEFORE OrderBy.
- **Bundle hash unchanged = ship FAIL** — push+action success nhưng prod không đổi. Verify `curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[a-z0-9]+\.js'`. Fix: SSH `Restart-WebAppPool`. ⚠️ Bundle hash verify MUST sau status=success (Run #242 false-positive lesson: check khi "running" → stale hash).
- **Migration drift prod vs repo** — compare `ls .../Persistence/Migrations/*.cs` vs `sqlcmd __EFMigrationsHistory`. Fix: check `Program.cs` `app.MigrateDatabase()` + app pool recycle.

---

## 📋 5-stage checklist (EVERY run)

- **Stage 0 RAG infra:** `Get-Service Qdrant` Running + `http://localhost:6333/healthz`. Collection `proj_solution_erp` (prefix `proj_*` 7 project — Discovery #8).
- **Stage 1 Push+filter:** `git log -1 --format='%H %s'` + `git log origin/main..HEAD` empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
- **Stage 2 Gitea poll** (max 10 iter × 60s): API `.../actions/tasks?limit=5` (NOT `/runs` 404). Match `head_sha`. ⚠️ task table `updated_at` stale ~2min (gotcha #46) → cross-check VPS mtime.
- **Stage 3 Test gate:** baseline **130 PASS** (58 Domain + 72 Infra). Phase 9 UAT exception lower OK (`feedback_uat_skip_verify`).
- **Stage 4 Post-deploy** (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=`accessToken` route `/api/auth/login`) → 3-5 endpoint smoke 2XX (incl new) → FE bundle hash 2 app changed → SignalR negotiate (gotcha #25 if relevant) → EF mig prod==repo.
  - **Stage 4.6 (S29 CRITICAL):** sqlcmd seed sample verify post-deploy (NOT chỉ schema). `sqlcmd -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'"` → 0 rows = seed GATE BLOCKED → gotcha #51.
  - Discovery #4: ASP.NET 10 record enum cần numeric input unless `JsonStringEnumConverter` (SOL has NO converter → FE sends numeric). #5: sqlcmd ssh Windows-auth cần `\\\\SQLEXPRESS` 4-backslash. #6: INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST run, NOT inside `if(!demoSeedDisabled)`; DEMO seed (DemoUsers/Contracts/PE) OK gated → gotcha #51.
- **Stage 5 Report** PASS/FAIL + evidence + MEMORY update.

---

## ⚠️ Anti-patterns (DO NOT)
1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

---

## 🧠 SOLUTION_ERP CI/CD essentials (S40 verified)

- **Gitea:** `git.baocaogiaoduc.vn/vietreport-admin/solution-erp` · workflow `.gitea/workflows/deploy.yml` · paths-ignore `['docs/**','**/*.md','.claude/skills/**']`
- **Prod:** api/admin/eoffice `.solutions.com.vn` · SSH `ssh vietreport-vps` (Administrator, id_ed25519) · IIS site phys paths (S42 verified): API `C:\inetpub\solution-erp\api` · admin `\fe-admin` · user `\fe-user` (3 sites Started). DB `.\SQLEXPRESS`/`SolutionErp`/`vrapp` SQL-auth. **Conn string key = `ConnectionStrings.Default` (NOT `DefaultConnection`!)** — read pw from prod appsettings.Production.json when `$env:PROD_DB_PASSWORD` empty.
  - **SSH→PS quoting (S42 lesson):** nested bash→ssh→powershell mangles `$var`/`\"`. Use `iconv UTF-16LE | base64` → `powershell -EncodedCommand $B64`. Single-quote literal paths.
- **Tests baseline:** **263 PASS** (S62 Run #286 sha 7926c21 spec; 45 Domain + 218 Infra — em-main supplied; supersedes prev 228/240/256). CI gate runs both test projects BEFORE build/deploy → status=success ⟹ test gate passed (`tasks` endpoint reports terminal as `status:success`, `conclusion` field NOT populated). Local grep undercounts (Theory/InlineData) — trust CI conclusion. Phase 9 UAT mode skip per chunk OK.
- **Mig latest repo:** **Mig 52 `20260616035929_AddHoSoLinkToPurchaseEvaluation`** (S65; PE +HoSoLink hyperlink-NAS, AddColumn-only no new table). Prev Mig 51 `AddDepartmentParentId` (S65 Department.ParentId loose-Guid org-tree, AddColumn-only) + Mig 50 `ReplaceBudgetModuleWithPeWorkItemBudgets` (S61 Budget→PeWorkItemBudgets net-reduce). Path `src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/` (52 mig .cs non-designer total). Prod check `sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5`. ⚠️ Table-count: `sys.tables` (is_ms_shipped=0, excl mighist) = **88** (S62 Run #286 verified — S61 Budget-replace DROPPED tables 93→88). Narrative-93 is STALE pre-S61 — when commit touches no schema, 88 is correct, don't FAIL on 88↔93. Always cross-ref COMMIT scope vs ambient count.
- **Bearer:** admin `admin@solutions.com.vn/Admin@123456` (full) · UAT `nv.test@solutions.com.vn/TestUser@123456` (Drafter CCM, gotcha #44 check)
- **Bundle hash live S70:** admin `BDwV5d0X` (FROZEN since S68 Run #293 — fe-admin untouched S69-S70) · user `DbVv6rsf` (Run #295 sha 456c7a7 — ROTATED from `CZfo_PFZ`, FE-user-only EmployeesListPage layout-2col + tô-màu 1 file). ⚠️ ASYMMETRIC-deploy lesson (S66): FE-one-app commit → that app's bundle MUST rotate + OTHER app MUST stay frozen; admin-rotate-when-only-fe-user-changed = anomaly → flag. S50 mid-deploy transient lesson: pre-success snapshot can show intermediate FE copy in-flight — re-confirm hash AFTER status=success ALWAYS (anti-pattern #3). FROZEN-expectation runs (BE-only or other-app): hash MUST stay = live pre-deploy value; rotate w/o relevant FE change = anomaly.
- **DB pw (S42, when `$PROD_DB_PASSWORD` empty):** `vrapp/buKL3TGBkD0wDDbYVw65QeX9` read from `C:\inetpub\solution-erp\api\appsettings.Production.json`→`ConnectionStrings.Default`. ⚠️ Skill-doc path `C:\inetpub\apps\SolutionErp\Api` is STALE → real path `C:\inetpub\solution-erp\api`. sqlcmd over SSH works direct (no UTF-16 encode needed). ⚠️ sys-catalog string-concat queries hit collation conflict (`Latin1_General_CI_AS_KS_WS` vs `SQL_Latin1_General_CP1_CI_AS`) → add `COLLATE DATABASE_DEFAULT` per concatenated column.

## 🔑 Critical config (flag commit nếu tái xuất)
Node CI `20.x` (`feedback_node_cicd`) · MediatR `12.4.1` (gotcha #1, flag `Version="14`) · Swashbuckle `6.9.0` (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag `cache: npm`)

---

## 🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)
Cross-ref `feedback_per_nv_permission_scope`. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF `HasDefaultValue(false)` · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 **Lookup discrimination** (`FirstOrDefault` ADD `ApproverUserId==actorId` + admin fallback) · 10 **Controller body record count == Command record count**. Bug latency 2-3 days prod silent khi miss 9-10. Scan `grep -n "FirstOrDefault.*Order.*==" *.cs` after OR-of-N refactor.

## 📊 Run stats baseline
BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · **total ~3min code / 0s docs-only**. >5min → escalate.

---

## 📅 Recent runs (FIFO — older → archive/git)

- **2026-06-16 S70 Run #295 (run_number 295, id409) sha=`456c7a7` PASS ~4m21s (FE-User "Hồ sơ Nhân sự" layout 2-cột: cây tổ chức + list chồng cột trái, detail cột phải + tô màu panel chi tiết — FE-USER-ONLY 1 file, NO BE, NO migration; follow-up S69 Run #294 same page):** Push `ec517f7..456c7a7` 1 file `fe-user/src/pages/hrm/EmployeesListPage.tsx` (309+/262−, 571-line layout rewrite). `.tsx` → full pipeline RAN. Both tokens SET this run (GITEA_TOKEN+PROD_DB_PW present — unlike S65-S69 empty; DB pw still works via prod `appsettings.Production.json`→`ConnectionStrings.Default` `buKL3...` len24). Run IN-PROGRESS first poll (running 11:36) — correctly did NOT verify-bundle-mid-flight (anti#3), polled iter6 status=success (started ~11:36 → success 11:40:46 ≈4m21s). Pre-deploy baseline snapshot iter0: user `CZfo_PFZ` (S69 baseline) + admin `BDwV5d0X` — captured BEFORE poll, correct timing. CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **263** baseline (45D+218I; FE-only ⟹ 0 BE call-site risk) passed; `conclusion` empty — trust success). **★ BUNDLE ASYMMETRIC (the change-point — FE-one-app verify) ALL PASS, verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient (anti#3): user ROTATE `CZfo_PFZ→DbVv6rsf`** ✓ (EmployeesListPage layout-2col shipped — re-rotated from S69's CZfo_PFZ, consecutive FE-user deploys on SAME page = new content-hash each, normal) **+ admin FROZEN `BDwV5d0X`==baseline** ✓ (fe-admin 0 files touched → MUST NOT rotate = correct; rotate-when-untouched = anomaly→flag). Health api live+ready **200/200** + admin/eoffice root 200. **NO migration** — prod `__EFMigrationsHistory` top = `20260616035929_AddHoSoLinkToPurchaseEvaluation` (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (commit 0 mig files — `git diff --name-only | grep Migrations/` = NONE; FE-only cannot alter schema → top did NOT advance past S68's Mig 52; prev-2 = `AddDepartmentParentId` Mig51 + `ReplaceBudgetModuleWithPeWorkItemBudgets` Mig50 chain intact). sys.tables stays 88 by construction (no schema touch — skipped explicit count, FE-only). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "2-cột đẹp/tô-màu" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged. **LESSON: pure-FE-one-app verify (3rd consecutive same-page FE-user S68→S69→S70) = (a) status=success ⟹ deploy ran; (b) target-app bundle ROTATE + sibling-app FROZEN (asymmetric — re-rotate of already-rotated page across sessions is NORMAL, each deploy=new hash; rotate-sibling-when-untouched = anomaly); (c) migration top MUST stay = prev (FE cannot alter schema — advancing top on FE-only commit = bug); (d) health 200×3. No BE call-site/DTO/endpoint smoke needed (no API surface). Tokens-present vs empty: both paths work — when GITEA_TOKEN set use `token` header, but anon also fine for public repo. NEVER fixed code (READ-only).** Tag `[s70, run295, pass, fe-user-hoso-2col-layout, fe-only-1file, bundle-asymmetric-user-rotate-admin-frozen, no-mig-top-stays-mig52, tables88, no-regression, test263, tokens-present]`.
- **2026-06-16 S69 Run #294 (run_number 294, id408) sha=`ec517f7` PASS ~4m18s (FE-User cây tổ chức gốc "SOLUTION COMPANY" toả xuống phòng ban — gộp nút "Tất cả"+list-phẳng → 1 node gốc công ty; FE-USER-ONLY 1 file, NO BE, NO migration):** Push `5a0aaa4..ec517f7` 1 file `fe-user/src/pages/hrm/EmployeesListPage.tsx`. `.tsx` → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (`buKL3...` len24). Run IN-PROGRESS first poll (running 11:24:05) — correctly did NOT verify-bundle-mid-flight (anti#3), polled iter5 status=success (started 11:24:05 → success ~11:28:23 ≈4m18s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **263** baseline (45D+218I; FE-only ⟹ 0 BE call-site risk) passed; `conclusion` empty — trust success). **★ BUNDLE ASYMMETRIC (the change-point — FE-one-app verify) ALL PASS, verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient (anti#3): user ROTATE `DXkyUjtQ→CZfo_PFZ`** ✓ (EmployeesListPage shipped — was rotated S68, now re-rotated) **+ admin FROZEN `BDwV5d0X`==baseline** ✓ (fe-admin 0 files touched → MUST NOT rotate = correct; rotate-when-untouched = anomaly→flag). Health api live+ready **200/200** + admin/eoffice root 200. **NO migration** — prod `__EFMigrationsHistory` top = `20260616035929_AddHoSoLinkToPurchaseEvaluation` (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (commit 0 mig files; FE-only cannot alter schema — top did NOT advance past S68's Mig 52; prev-2 = `AddDepartmentParentId` Mig51 + `ReplaceBudgetModuleWithPeWorkItemBudgets` Mig50 confirm chain intact). sys.tables stays 88 by construction (no schema touch). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "đẹp/cây-gốc-công-ty" NOT verified (anh xem mắt) — only ship+rotate+health. **LESSON: pure-FE-one-app verify = (a) status=success ⟹ deploy ran; (b) target-app bundle ROTATE + sibling-app FROZEN (asymmetric — rotate-sibling-when-untouched = anomaly); (c) migration top MUST stay = prev (FE cannot alter schema — advancing top on a FE-only commit would be a bug); (d) health 200×3. No BE call-site risk, no DTO/endpoint smoke needed (no API surface changed). Re-rotate of an already-rotated app across consecutive sessions is normal (each FE deploy = new content-hash). Tokens empty: anon Gitea + prod-appsettings DB pw works (Bash tool POSIX — read pw via ssh→appsettings ConnectionStrings.Default len24).** Tag `[s69, run294, pass, fe-user-org-tree-root-company, fe-only-1file, bundle-asymmetric-user-rotate-admin-frozen, no-mig-top-stays-mig52, tables88, no-regression, test263]`.
- **2026-06-16 S68 Run #293 (run_number 293, id407) sha=`5a0aaa4` PASS ~5m00s (FE-User "Hồ sơ Nhân sự" 3-panel master-detail 5-tab rewrite + PE "Link hồ sơ" hyperlink NAS + rename "Dự trù PRO"→"Ngân sách PRO" — CROSS-STACK BE Mig 52 + FE×2; 2-commit `318860a`(FE-User EmployeesListPage) + `5a0aaa4`(PE cross-stack)):** Push `6ce5803..5a0aaa4` 15 files: BE `PurchaseEvaluation.cs`(+HoSoLink) + `PurchaseEvaluationConfiguration.cs` + `PurchaseEvaluationFeatures.cs`(Create/Update cmd +trailing optional `HoSoLink=null`) + `PurchaseEvaluationDtos.cs`(Detail DTO +hoSoLink) + Mig 52 `20260616035929_AddHoSoLinkToPurchaseEvaluation`(3-file) + FE `EmployeesListPage.tsx`(fe-user, 3-panel rewrite cây-tổ-chức+list+5-tab) + PE×2 (`PeDetailTabs.tsx`+`PeWorkspaceCreateView.tsx`+`types/purchaseEvaluation.ts` both apps, mục E hyperlink + label rename). `.cs`+`.tsx`+Mig → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (`buKL3...`). Run IN-PROGRESS first poll (running 11:15) — correctly did NOT verify-bundle-mid-flight (anti#3), polled iter5 status=success (started 11:13:57 → success 11:18:57 ≈5m). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **263** baseline (45D+218I; BE Create/Update +trailing optional param=null ⟹ call-sites UNBROKEN) passed; `conclusion` empty — trust success). **★ MIGRATION APPLIED (the KEY): prod `__EFMigrationsHistory` top = `20260616035929_AddHoSoLinkToPurchaseEvaluation`** ✓ (Mig 52; ADVANCED from prev top Mig 51 `AddDepartmentParentId`; DbInitializer auto-migrate on startup ran — api ready=200 confirms boot OK post-mig). Mig content = AddColumn `HoSoLink nvarchar(1000) maxLength=1000 NULL` (AddColumn-only, NO new table → **sys.tables stays 88** verified). DB-level: `sys.columns WHERE object_id=OBJECT_ID('PurchaseEvaluations') AND name='HoSoLink'` = 1 row ✓ (column physically present prod). **★ BUNDLE BOTH ROTATE (verified AFTER status=success +re-confirm STABLE 2nd-fetch no transient — anti#3): admin `Df06fmpq→BDwV5d0X`** ✓ (PE files shipped — was FROZEN since S67, now correctly rotated) **+ user `DxK3fCfh→DXkyUjtQ`** ✓ (EmployeesListPage + PE files shipped). Both required to rotate per spec → BOTH did. Health api live+ready **200/200** + admin/eoffice root 200. **★ NEW-FIELD live smoke (admin bearer, route `/api/auth/login` field `accessToken` len468):** `GET /api/purchase-evaluations/{48154149...}` (phiếu thật `PE/2026/A/015`) = **200** ✓ JSON detail carries `"hoSoLink":null` (key PRESENT, null cho phiếu cũ = backward-compat ✓ — Detail DTO wired AND Create/Update signature-change did NOT break GET path) · `GET /api/departments/tree` = **200** ✓ (Phase B dependency from Mig 51, still live). 0 regression. NO prod-data mutation (read-only GETs + sqlcmd SELECT-only). **LESSON: nullable-AddColumn cross-stack verify = (a) `__EFMigrationsHistory` top ADVANCED to new Mig (Mig 52) + api ready=200 (auto-migrate boot OK); (b) `sys.tables`=88 unchanged (AddColumn ≠ new table) + `sys.columns` confirms col present; (c) Detail DTO field-presence via authed GET on a REAL phiếu — `"hoSoLink":null` for old rows proves both DTO-wiring AND backward-compat (trailing-optional Create/Update param won't break GET — confirmed 200); (d) cross-stack BE+FE×2 = BOTH bundles rotate (admin un-froze from S67). Tokens empty: anon Gitea + prod-appsettings DB pw works (Bash tool is POSIX bash NOT PS despite shell=PowerShell env — `$env:`/`jq`/`python3` unavailable; parse via `tr ','|grep` + read pw via `ssh→appsettings`).** Tag `[s68, run293, pass, pe-hosolink-mig52-applied, fe-user-hoso-3panel-rewrite, pe-link-hoso-hyperlink, rename-dutru-to-ngansach, bundle-both-rotate-admin-unfroze, hosolink-null-in-dto-backward-compat, no-new-table-88, test263]`.
- **2026-06-16 S67 Run #292 (run_number 292, id406) sha=`6ce5803` PASS ~4m06s (phân cấp phòng ban Department.ParentId + /tree + FE-admin "Phòng cha" picker — CROSS-STACK BE+FE-admin+Mig 51, RETRY của Run #291 FAIL):** Push 3-commit chain `0f44d97`(BE Department.ParentId + DepartmentFeatures + /tree + Config + Mig 51) → `8c8179c`(FE-admin DepartmentsPage picker + types/master.ts) → `6ce5803`(Tests fix `MasterCatalogFilteredUniqueTests.cs` +ParentId 5th arg CreateDepartmentCommand call-site). **★ Run #291 (id405) sha=`8c8179c` = `failure` @10:42:18 (test-gate CS7036 — CreateDepartmentCommand thêm ParentId param nhưng test call-site chưa update → build_test FAIL → deploy GATED → Mig NOT applied).** `6ce5803` = the fix → Run #292 status=success @10:50:16→10:54:22. Files full-range (8 src incl `.cs`+`.tsx`+Mig 3-file) → full pipeline RAN. Tokens empty → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (`buKL3...`). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **263** (45D+218I, +1 fixed call-site) passed; `conclusion` empty — trust success). **★ MIGRATION APPLIED (the KEY — last time gated, this time landed): prod `__EFMigrationsHistory` top = `20260616032402_AddDepartmentParentId`** ✓ (Mig 51; prev top Mig 50 `ReplaceBudgetModuleWithPeWorkItemBudgets`; DbInitializer auto-migrate on startup ran — api ready=200 confirms boot OK post-mig). Mig content = AddColumn `ParentId uniqueidentifier NULL` + `IX_Departments_ParentId` (loose-Guid NO physical FK — convention Master; AddColumn-only, NO new table → sys.tables stays 88). **★ BUNDLE ASYMMETRIC (verified AFTER status=success, terminal not mid-flight): admin ROTATE `DRob3iVl→Df06fmpq`** ✓ (fe-admin picker shipped — was FROZEN since S65, now rotated correctly) **+ user FROZEN `DxK3fCfh`==baseline** ✓ (fe-user 0 files touched → MUST NOT rotate = correct). Health api live+ready **200/200** + admin/eoffice root 200. **★ NEW-ENDPOINT live smoke (admin bearer, route `/api/auth/login` field `accessToken` len468):** `GET /api/departments/tree` = **200** ✓ JSON array nodes each w/ `parentId`+`directEmployeeCount`+`totalEmployeeCount`+`children[]` (e.g. CCM direct=10/total=10, BOD 1/1, ACT 0/0; all `parentId:null` currently — no parent assigned yet in prod, but col+endpoint live) · `GET /api/departments?page=1&pageSize=5` list = **200** ✓ DTO includes `parentId` (IT/PM both null). 0 regression. **LESSON: FAIL→retry chain — when commit N fails test-gate (CS7036 call-site mismatch from a signature change) the deploy is GATED so prod Mig stays at N-1; the FIX commit N+1 must re-trigger a NEW run (here #292) whose success ⟹ Mig finally applies. KEY verify on retry = confirm `__EFMigrationsHistory` top ADVANCED to the new Mig (not stuck at prev) + api ready=200 (no startup crash from the auto-migrate). Cross-stack BE+FE-one-app = asymmetric bundle: changed-app rotates (admin un-froze from S65), untouched-app stays frozen. New endpoint = curl 200 + structural-key presence proof; FK-less ParentId is convention (PE/Master loose-Guid) not a defect. Tokens empty: anon Gitea + prod-appsettings DB pw works (PS `$env:` syntax in spec N/A — Bash tool is POSIX, read pw via ssh→appsettings).** Tag `[s67, run292, pass, dept-parentid-hierarchy, mig51-applied-after-291-fail, retry-chain-cs7036-fix, bundle-asymmetric-admin-rotate-user-frozen, tree-endpoint-200, no-new-table-88, test263]`.
- **2026-06-16 S66 Run #290 (run_number 290, id404) sha=`c98030f` PASS ~4m24s (redesign foundation màu fe-user "nâng màu giữ brand" — FE-USER-ONLY 14 files, NO BE, NO migration):** Push `4004481..c98030f` 14 files all `fe-user/src/**` (index.css + 6 ui primitives Button/Dialog/Input/Label/Select/Textarea + 4 shell DataTable/Layout/PageHeader/TopBar + UserDashboardPage + 2 color-map types contracts/purchaseEvaluation). `.tsx`+`.css` → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + (DB pw read deferred, FE-only no DB touch). Run IN-PROGRESS first poll (running 10:15) — correctly did NOT verify-bundle-mid-flight (anti#3), polled iter3 status=success (10:13:04→10:17:28 ≈4m24s). CI gate (both test proj pre-deploy ⟹ status=success ⟹ test gate **263** baseline (45 Domain + 218 Infra) passed; `conclusion` empty — trust success). **★ BUNDLE ASYMMETRIC (the change-point — one-app-FE verify) ALL PASS, verified AFTER status=success +re-confirm STABLE no transient: user ROTATE `Cpion_xQ→DxK3fCfh`** ✓ (fe-user 14 files shipped) **+ admin FROZEN `DRob3iVl`==baseline** ✓ (fe-admin 0 files touched → MUST NOT rotate = correct). Health api live+ready **200/200** + admin/eoffice root 200. **★ CSS-REDESIGN-SHIPPED PROOF (live-vs-source diff, defeats stale-bundle false-pass):** fetch live eoffice CSS `index-DZ-56qoN.css` (71740 B) → grep brand `#1F7DC1` PRESENT + NEW accent tokens `--color-teal-{50..800}` + `app-gradient-brand` + `card-accent` (×5) + `--color-accent-{500,600}` ALL PRESENT ✓. Cross-check: these tokens = **0 occurrences in HEAD~1 index.css**, **added in c98030f** (git show source: teal×6, app-gradient-brand×1, card-accent×5, 1F7DC1×1) → live CSS IS new redesign NOT stale. **NO migration** — repo HEAD top = Mig 50 `ReplaceBudgetModuleWithPeWorkItemBudgets` GIỮ ✓ (commit 0 mig files; FE-only cannot alter schema → prod `__EFMigrationsHistory` top stays Mig 50 by construction, fe-user site static no .NET runtime — DB check skipped per FE-only scope). 0 regression. **LESSON: ASYMMETRIC FE-one-app verify = (a) status=success ⟹ deploy ran; (b) target-app bundle ROTATE + sibling-app FROZEN (rotate-sibling-when-untouched = anomaly→flag); (c) for visual/CSS redesign — fetch live CSS asset + grep claimed tokens, AND diff-prove tokens were 0-in-HEAD~1 / added-in-commit (live-token-present alone could be coincidental stale; the 0→N delta defeats stale-bundle false-pass). Visual "đẹp" NOT verified (anh xem mắt) — only ship+rotate+CSS-token-present. Tokens empty: anon Gitea works; FE-only ⟹ no DB pw needed.** Tag `[s66, run290, pass, fe-user-redesign-foundation, fe-only-14files, bundle-asymmetric-user-rotate-admin-frozen, css-tokens-shipped-0to-N-delta, teal-accent-gradient-brand, no-mig, test263]`.

- **2026-06-16 S65 Run #289 (run_number 289, id403) sha=`4004481` PASS ~4m41s (mở quyền XEM "Hồ sơ Nhân sự" read-only cho MỌI role eoffice — BE-ONLY 1 file DbInitializer SEED, NO migration, NO FE):** Push `cfed3d0..4004481` 1 file `DbInitializer.cs` (+66, new `SeedAllRolesHrmProfileReadPermissionsAsync` grant CanRead `Hrm`+`Hrm_HoSo` 13 role, upgrade-only mirror PE :2107, chạy SAU `RevokeTemporarilyHiddenModulesAsync` S58 để THẮNG revoke; Create/Update/Delete giữ false). `.cs` → full pipeline RAN. **GITEA_TOKEN+PROD_DB_PW empty** → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (`buKL3...`). Run IN-PROGRESS first poll (running 09:53) — correctly did NOT FAIL/verify-bundle-mid-flight (anti#3), polled iter5 status=success (09:52:03→09:56:44 ≈4m41s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **263** baseline (45 Domain + 218 Infra) passed; `conclusion` empty — trust success). **Bundle FROZEN (BE-only, NO FE change → MUST NOT rotate = correct): admin `DRob3iVl` + user `Cpion_xQ`** ✓ both unchanged pre==post (live advanced from S62 `0xKYGhhf`/`C81ZdG9G` via intervening S63/S64 FE deploys — narrative-Run#286 hash now 3 deploys stale; current live = baseline). Health live+ready **200/200** + admin/eoffice root 200. **NO migration** — prod `__EFMigrationsHistory` top = `20260612173224_ReplaceBudgetModuleWithPeWorkItemBudgets` (Mig 50) == repo HEAD GIỮ ✓ (commit 0 mig files). **★ SEED-EFFECT 7-CHECK reviewer ALL PASS (the change-point — asymmetric read-open/write-locked, non-admin `nv.test`=Drafter + admin):** (1) menus/me non-admin tree = exactly `['Hrm','Hrm_HoSo']` present ✓ + (4) `Hrm_Dashboard`+`Hrm_Config*` ABSENT ✓ (revoke still che); (2) non-admin GET /api/employees?page=1&pageSize=20 = **200** ✓ (was 403 — MẤU CHỐT, seed áp đúng + `Hrm_HoSo.Read` resolve qua permission matrix); (3) non-admin GET /api/employees/{id} = **200** ✓; (5) non-admin POST = **403** ✓; (6) non-admin PUT+DELETE = **403/403** ✓ (read-only giữ); (7) admin GET = **200** ✓ no regression. **DB-level ground-truth (sqlcmd via SSH):** `Permissions WHERE MenuKey IN ('Hrm','Hrm_HoSo')` = 13 rows EACH, all CanRead=1, AnyWrite=1 (=Admin only, full CRUD legit — upgrade-only KHÔNG hạ write existing) ✓. EmployeeProfiles=33 INTACT + ZZCICD-PROBE=0 (403 POST tạo 0 rows, no leak) ✓. ⚠️ **CRED NOTE: reviewer spec ghi non-admin pw `User@1234567` SAI** → 401 "Email/mật khẩu không đúng"; pw THẬT = memory-baseline `TestUser@123456` ✓. 0 regression. **LESSON: SEED-change verify (DbInitializer, no-mig) = (a) status=success ⟹ app restarted ⟹ seed ran; (b) prove EFFECT not schema — curl asymmetric probe (read 200 / write 403 / admin 200) + sqlcmd Permission-row count==role-count w/ CanRead=1 & write-row=Admin-only; (c) "seed NO-OP" failure-class (S58) caught iff check#2 still 403 post-restart — here 200 ⟹ revoke-then-grant ordering correct. BE-only ⟹ bundle FROZEN = pass-signal (rotate w/o FE = anomaly). Tokens empty: anon Gitea + prod-appsettings DB pw works. Spec creds may be illustrative — fall back to memory-baseline on 401.** Tag `[s65, run289, pass, hrm-hoso-public-readonly, seed-only-no-mig, bundle-frozen, asymmetric-read200-write403, permrows-13-canread, no-leak, spec-cred-wrong-fallback-baseline]`.
- **Older runs (S62 #286 06-13 <- ... -> S29 #232) full verbatim archived S66 -> `archive/2026-06.md`** (incl #291 06-16 FAIL forensic [lesson=gotcha #65 + #292-inline] + #383 ex-VITRILAC); pre-S38 -> `archive/2026-05-{runs,q2,q3,q4}.md` + git `d2f52ba`.

---

## 🔄 Curate trigger
- >~30KB → archive recent runs → L2 `archive/<period>.md`. Dup failure patterns → merge. Stale >3mo → remove.
- **Last curate: 2026-06-16 S66 em main** (86.8→29.2KB — 2.9× over-cap fix): byte-exact `sed` move Run #286→#232 (S62→S29, incl #291 forensic [lesson=gotcha #65 + #292-inline] + #383 ex-VỊ-TRÍ-LẠC) → `archive/2026-06.md`; refreshed essentials Mig 50→52 + bundle-live S69→S70 #295. Kept baseline (gotcha patterns + Stage 0-5 + Stage 4.6 + 10-point + Discovery #4-8) + 6 current 06-16 runs #289-#295.
- **Prev curate: 2026-05-29 S40 em main proxy** (35.3→~21KB): archived Run #359/#243/#242/#241/#240 + S35/S36 startup → q4 + git d2f52ba; refreshed stale 120→130 test + Mig 34→40. Prev: S34 q3 · S32 q2 · S22 runs.