# CI/CD Monitor Agent — Persistent Memory > **Persistent diary cross-session.** Auto-injected first ~200 lines at spawn (L1 HOT). > Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 `archive/` on-demand · L3 RAG `search_memory` just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). > Full verbatim run history pre-S40 → git `d2f52ba` + `archive/2026-05-{runs,q2,q3,q4}.md`. --- ## 🎯 Role baseline Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: `iis-deploy-runbook` + `dependency-audit-erp` + `ef-core-migration`. Spawn ~150K — trade-off catch fail tự động. --- ## 🚨 Recurring CI/CD bug patterns (catch priority) - **#39 act_runner github.com TCP timeout** — run hang "Set up job" 21s. Log `dial tcp github.com:443 i/o timeout`. Fix: manual checkout bypass hardcoded `.gitea/workflows/deploy.yml` (pass #110). KHÔNG revert. - **#40 npm cache `tsc not found`** — `build_fe_admin` fail post `cache: npm`. DISABLED rolled back `a21790d`. KHÔNG re-enable. - **#41 paths-ignore docs-only skip** — code commit không trigger CI? Check `git diff --name-only HEAD~1 HEAD` vs `paths-ignore: ['docs/**','**/*.md','.claude/skills/**']`. Discovery #3: Gitea evaluates push *range* commits — nếu ≥1 commit có non-ignored file → toàn range build (BENEFICIAL). - **#25 IIS WebSocket** — `notification-hub/negotiate` 401/404 prod. Fix: WebSocket module enable `web.config` site api (skill `iis-deploy-runbook`). - **#48 SQLite tie-break** — `OrderByDescending(CreatedAt).First()` pick wrong khi 2+ `.Add()` cùng frozen-clock. Fix: discriminator filter `.Where(Summary.Contains("Chuyển phase"))` BEFORE OrderBy. - **Bundle hash unchanged = ship FAIL** — push+action success nhưng prod không đổi. Verify `curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[a-z0-9]+\.js'`. Fix: SSH `Restart-WebAppPool`. ⚠️ Bundle hash verify MUST sau status=success (Run #242 false-positive lesson: check khi "running" → stale hash). - **Migration drift prod vs repo** — compare `ls .../Persistence/Migrations/*.cs` vs `sqlcmd __EFMigrationsHistory`. Fix: check `Program.cs` `app.MigrateDatabase()` + app pool recycle. --- ## 📋 5-stage checklist (EVERY run) - **Stage 0 RAG infra:** `Get-Service Qdrant` Running + `http://localhost:6333/healthz`. Collection `proj_solution_erp` (prefix `proj_*` 7 project — Discovery #8). - **Stage 1 Push+filter:** `git log -1 --format='%H %s'` + `git log origin/main..HEAD` empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return). - **Stage 2 Gitea poll** (max 10 iter × 60s): API `.../actions/tasks?limit=5` (NOT `/runs` 404). Match `head_sha`. ⚠️ task table `updated_at` stale ~2min (gotcha #46) → cross-check VPS mtime. - **Stage 3 Test gate:** baseline **130 PASS** (58 Domain + 72 Infra). Phase 9 UAT exception lower OK (`feedback_uat_skip_verify`). - **Stage 4 Post-deploy** (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=`accessToken` route `/api/auth/login`) → 3-5 endpoint smoke 2XX (incl new) → FE bundle hash 2 app changed → SignalR negotiate (gotcha #25 if relevant) → EF mig prod==repo. - **Stage 4.6 (S29 CRITICAL):** sqlcmd seed sample verify post-deploy (NOT chỉ schema). `sqlcmd -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'"` → 0 rows = seed GATE BLOCKED → gotcha #51. - Discovery #4: ASP.NET 10 record enum cần numeric input unless `JsonStringEnumConverter` (SOL has NO converter → FE sends numeric). #5: sqlcmd ssh Windows-auth cần `\\\\SQLEXPRESS` 4-backslash. #6: INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST run, NOT inside `if(!demoSeedDisabled)`; DEMO seed (DemoUsers/Contracts/PE) OK gated → gotcha #51. - **Stage 5 Report** PASS/FAIL + evidence + MEMORY update. --- ## ⚠️ Anti-patterns (DO NOT) 1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay) --- ## 🧠 SOLUTION_ERP CI/CD essentials (S40 verified) - **Gitea:** `git.baocaogiaoduc.vn/vietreport-admin/solution-erp` · workflow `.gitea/workflows/deploy.yml` · paths-ignore `['docs/**','**/*.md','.claude/skills/**']` - **Prod:** api/admin/eoffice `.solutions.com.vn` · SSH `ssh vietreport-vps` (Administrator, id_ed25519) · IIS site phys paths (S42 verified): API `C:\inetpub\solution-erp\api` · admin `\fe-admin` · user `\fe-user` (3 sites Started). DB `.\SQLEXPRESS`/`SolutionErp`/`vrapp` SQL-auth. **Conn string key = `ConnectionStrings.Default` (NOT `DefaultConnection`!)** — read pw from prod appsettings.Production.json when `$env:PROD_DB_PASSWORD` empty. - **SSH→PS quoting (S42 lesson):** nested bash→ssh→powershell mangles `$var`/`\"`. Use `iconv UTF-16LE | base64` → `powershell -EncodedCommand $B64`. Single-quote literal paths. - **Tests baseline:** **263 PASS** (S62 Run #286 sha 7926c21 spec; 45 Domain + 218 Infra — em-main supplied; supersedes prev 228/240/256). CI gate runs both test projects BEFORE build/deploy → status=success ⟹ test gate passed (`tasks` endpoint reports terminal as `status:success`, `conclusion` field NOT populated). Local grep undercounts (Theory/InlineData) — trust CI conclusion. Phase 9 UAT mode skip per chunk OK. - **Mig latest repo:** **Mig 53 `20260617060207_AddPeUrgentAndCeoApprovalThreshold`** (S71 Run#308; PE +IsUrgentByPro/+IsUrgentByCcm bit-default-0 + ApprovalWorkflows +CeoApprovalThreshold decimal(18,2)-nullable, 3 AddColumn no new table — VERIFIED APPLIED PROD). Prev Mig 52 `AddHoSoLinkToPurchaseEvaluation` (S65 PE HoSoLink hyperlink-NAS) + Mig 51 `AddDepartmentParentId` (S65 org-tree loose-Guid) + Mig 50 `ReplaceBudgetModuleWithPeWorkItemBudgets` (S61 net-reduce). Path `src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/` (53 mig .cs non-designer total). Prod check `sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5`. ⚠️ Table-count: `sys.tables` (is_ms_shipped=0, excl mighist) = **88** (S62 Run #286 verified — S61 Budget-replace DROPPED tables 93→88). Narrative-93 is STALE pre-S61 — when commit touches no schema, 88 is correct, don't FAIL on 88↔93. Always cross-ref COMMIT scope vs ambient count. - **Bearer:** admin `admin@solutions.com.vn/Admin@123456` (full) · UAT `nv.test@solutions.com.vn/TestUser@123456` (Drafter CCM, gotcha #44 check) - **Bundle hash live S71:** admin `BgNCjwsG` (Run #308 sha ebd7e1c — ROTATED from `Wt54PHYl`; PE cờ-gấp UI list/detail + designer ngưỡng) · user `CBvh0vtf` (Run #308 sha ebd7e1c — ROTATED from `B99fMU6X`; PE list/detail). BOTH rotate (FE-both-app). Prev live admin `Wt54PHYl`/user `B99fMU6X` (Run #306/#307 S70). [Older S77 #303: admin `D532XZKG`/user `CuFaBoWt`]. ⚠️ ASYMMETRIC-deploy lesson (S66): FE-one-app commit → that app's bundle MUST rotate + OTHER app MUST stay frozen; admin-rotate-when-only-fe-user-changed = anomaly → flag. S50 mid-deploy transient lesson: pre-success snapshot can show intermediate FE copy in-flight — re-confirm hash AFTER status=success ALWAYS (anti-pattern #3). FROZEN-expectation runs (BE-only or other-app): hash MUST stay = live pre-deploy value; rotate w/o relevant FE change = anomaly. - **DB pw (S42, when `$PROD_DB_PASSWORD` empty):** `vrapp/buKL3TGBkD0wDDbYVw65QeX9` read from `C:\inetpub\solution-erp\api\appsettings.Production.json`→`ConnectionStrings.Default`. ⚠️ Skill-doc path `C:\inetpub\apps\SolutionErp\Api` is STALE → real path `C:\inetpub\solution-erp\api`. sqlcmd over SSH works direct (no UTF-16 encode needed). ⚠️ sys-catalog string-concat queries hit collation conflict (`Latin1_General_CI_AS_KS_WS` vs `SQL_Latin1_General_CP1_CI_AS`) → add `COLLATE DATABASE_DEFAULT` per concatenated column. ## 🔑 Critical config (flag commit nếu tái xuất) Node CI `20.x` (`feedback_node_cicd`) · MediatR `12.4.1` (gotcha #1, flag `Version="14`) · Swashbuckle `6.9.0` (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag `cache: npm`) --- ## 🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23) Cross-ref `feedback_per_nv_permission_scope`. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF `HasDefaultValue(false)` · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 **Lookup discrimination** (`FirstOrDefault` ADD `ApproverUserId==actorId` + admin fallback) · 10 **Controller body record count == Command record count**. Bug latency 2-3 days prod silent khi miss 9-10. Scan `grep -n "FirstOrDefault.*Order.*==" *.cs` after OR-of-N refactor. ## 📊 Run stats baseline BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · **total ~3min code / 0s docs-only**. >5min → escalate. --- ## 📅 Recent runs (FIFO — older → archive/git) - **2026-06-17 S71 Run #308 (id422) sha=`ebd7e1c` PASS ~4m41s (FULL-STACK: BE Mig 53 + ApproveV2 CCM-finalize-by-threshold + endpoint PUT /urgent + FE 2-app PE cờ-gấp UI + designer ngưỡng. PE "cờ gấp PRO/CCM + CCM duyệt-final theo ngưỡng giá trị" anh Kiệt FDC. 27 files: BE 9 {Controller +SSetUrgent, App PurchaseEvaluationUrgentFeatures.cs NEW, PurchaseEvaluationFeatures, CreateContractFromEvaluation, AwV2AdminFeatures, Dtos, Domain PE+AW, Config} + 3 Mig-file + Service PurchaseEvaluationWorkflowService + FE 6 {PeDetailTabs+ListPage+types ×2-app + ApprovalWorkflowsV2Page admin-only} + 2 test {PeUrgentToggleAuthzTests, PeCcmThresholdFinalizeTests} + 3 agent-memory .md. deploy 12/12 session after #297–#307 all PASS):** Push parent `1f8947e..ebd7e1c` (1 commit). `git diff --name-only 1f8947e ebd7e1c -- '*Persistence/Migrations*'` = 3 files (Mig 53 .cs/.Designer.cs/ModelSnapshot.cs) — REAL EF migration this time (contrast S69b/S70 which had ZERO). `.cs/.tsx` non-ignored → full pipeline RAN. GITEA_TOKEN present (PS-scope) → authed Gitea API; PROD_DB_PW empty → DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (`vrapp`/`buKL3TGBkD0wDDbYVw65QeX9`, path `C:\inetpub\solution-erp\api`). Run IN-PROGRESS at spawn (running 13:27:58, 1st-seen updated 13:28:54) — pre-deploy NOT snapshotted as baseline (anti#3: prev-live = #306/#307 spec admin `Wt54PHYl`/user `B99fMU6X`). Poll-loop iter6 status=success (created 13:27:58 → success-update 13:32:39 ≈4m41s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test **306** baseline (45D+261I; +14 PeUrgentToggleAuthz+PeCcmThresholdFinalize) passed; `conclusion` empty — `tasks` terminal=`status:success` không populate conclusion, trust success; 306 INFERRED gate-passes-pre-build invariant NOT log-confirmed numeric). **★ BUNDLE BOTH ROTATE (FE-both-app PE-UI ⟹ both MUST rotate; verified AFTER status=success + re-confirm STABLE 2nd-fetch identical no-transient — anti#3): admin ROTATE `Wt54PHYl→BgNCjwsG`** ✓ **+ user ROTATE `B99fMU6X→CBvh0vtf`** ✓. Asset reachable 200: admin js 1,592,331b + user js 1,496,984b (not white). Smoke **4×200** health: api `/health/ready`+`/health/live` + admin root + eoffice root. **★ ENDPOINT /urgent PROBE: `PUT /api/purchase-evaluations/{guid}/urgent` unauth → 401 (NOT 404)** ✓ — route EXISTS + class-level `[Authorize]` any-auth enforced (controller `[HttpPut("{id:guid}/urgent")] SetUrgent → SetPurchaseEvaluationUrgentCommand`; behavioral PRO/CCM-only authz = unit-test PeUrgentToggleAuthzTests, not probed). PE list unauth → 401 ✓ (reachable, gated). **🔑★ MIG 53 VERIFIED APPLIED PROD (sqlcmd over SSH, ground-truth not inference): prod `__EFMigrationsHistory` top5 = `20260617060207_AddPeUrgentAndCeoApprovalThreshold`(53)→`AddHoSoLinkToPurchaseEvaluation`(52)→`AddDepartmentParentId`(51)→`ReplaceBudgetModuleWithPeWorkItemBudgets`(50)→`AddWorkItemToPurchaseEvaluation`(49) == repo HEAD ✓ (DbInitializer auto-migrate-on-boot ran; app pool recycled w/ new binary). 3 NEW COLUMNS confirmed via sys.columns: `PurchaseEvaluations.IsUrgentByCcm`=bit is_nullable=0 ✓ + `PurchaseEvaluations.IsUrgentByPro`=bit is_nullable=0 ✓ + `ApprovalWorkflows.CeoApprovalThreshold`=decimal is_nullable=1 precision=18 scale=2 ✓ — ALL match Mig 53 Up() spec exactly. sys.tables=88 (excl mighist) UNCHANGED — 3 AddColumn no new table ✓.** 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only; migration-apply = expected boot-side-effect of deploy). Behavioral CCM-finalize-by-threshold + cờ-gấp UI render = anh Kiệt UAT (em confirm Mig applied + columns + endpoint route + bundles shipped per spec). **LESSON: full-stack PE commit w/ REAL Mig = verify ALL 4 axes: (1) BE+FE both-bundle ROTATE (FE 2-app), (2) Mig history-top advance + 3 columns sys.columns ground-truth (NOT just "schema looks right" — query each col type/nullable/precision vs Up() spec), (3) sys.tables stays 88 (AddColumn-only ⟹ no table delta; rotate-to-89 would = unexpected CREATE TABLE leak), (4) new endpoint 401-not-404 unauth probe (route-exists proof; authz-logic = unit-test domain not curl). Mig-applied-on-prod proof = `__EFMigrationsHistory` top == repo-HEAD (DbInitializer MigrateAsync boot-hook) — if top stuck at Mig 52 ⟹ app pool didn't recycle / migrate-on-boot failed ⟹ FAIL even if status=success+bundle-rotated (deploy shipped binary but DB-side incomplete). Distinguishes this run from S69b/S70 (zero-migration BE/FE — those had NO history-advance EXPECTED). TOOLING: Bash=POSIX → write PS to `$LOCALAPPDATA/Temp/x.ps1` + `powershell.exe -NoProfile -File "C:\Users\pqhuy\AppData\Local\Temp\x.ps1"` (⚠️ bash→Windows path auto-convert mangles `/tmp/` → use `$LOCALAPPDATA/Temp` explicit Windows path); Invoke-RestMethod for Gitea `tasks` (match `head_sha -eq sha`, poll-loop in PS not bash — bash jq absent/python3 broken); SSH→PS base64 `-EncodedCommand` (UTF-16LE Unicode) for BOTH appsettings-read AND sqlcmd; sqlcmd pw-literal via PS here-string `$P=` var inside encoded payload + query-string built w/ `[char]39` for embedded single-quotes (object-name literals); `-W -h -1` no-header + `-s '|'` pipe-delim; read DB pw from prod appsettings when `$PROD_DB_PASSWORD` empty. NEVER fixed code (READ-only).** Tag `[s71, run308, pass, full-stack-pe-urgent-ccm-threshold, mig53-AddPeUrgentAndCeoApprovalThreshold-VERIFIED-APPLIED-PROD, 3-cols-sys-columns-confirmed-IsUrgentByPro-IsUrgentByCcm-bit0-CeoApprovalThreshold-decimal18-2-null, history-top-advance-53, tables88-unchanged-addcolumn-only, bundle-BOTH-rotate-BgNCjwsG-CBvh0vtf, asset-200-reachable, endpoint-urgent-401-not-404-route-exists, pe-list-401-gated, health-4x200, test306-inferred, deploy12of12, no-regression, behavioral-ccm-finalize-anh-kiet-uat]`. - **2026-06-17 S69b Run #307 (run_number 307, id421) sha=`1f8947e` PASS ~4m33s (BE-ONLY GOLIVE security hướng-ra-prod "Văn phòng số" public Read+Create mọi role — NEW `SeedAllRolesOfficeModulePermissionsAsync` DbInitializer.cs chạy lúc API boot SAU `RevokeTemporarilyHiddenModulesAsync` để THẮNG revoke (mirror S65 HRM/Hồ-sơ-NS pattern), allow-list 16 Office key grant CanRead+CanCreate=true upgrade-only (row tồn tại→nâng, chưa có→tạo R+C=true U/D=false, KHÔNG hạ KHÔNG đụng Update/Delete); +6 test `OfficeModulePermissionSeedTests.cs` (286→292). KHÔNG FE KHÔNG migration. deploy 11/11 session after #297–#306 all PASS):** Push `c556f6c..1f8947e` (1 commit, 5 files: DbInitializer.cs + 1 test + 3 agent-memory `.md` cicd/reviewer/test-spec). Diff `git diff --name-only c556f6c 1f8947e -- '*Migrations*' '*Persistence/Migrations*'` = EMPTY ✓ (DbInitializer.cs ở `Persistence/` NOT `Persistence/Migrations/` — seed runtime-idempotent NOT EF-migration). `.cs` non-ignored → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW env empty trong bash-scope (env:GITEA_TOKEN có ở PS-scope; PROD_DB_PW thật rỗng) → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (path `C:\inetpub\solution-erp\api`, uid `vrapp` len24). Run IN-PROGRESS at spawn (running 10:33:40→10:38) — pre-deploy bundle baseline captured BEFORE poll-loop (anti#3): admin `Wt54PHYl` + user `B99fMU6X` — both == S70 #306 spec baseline (still live, deploy not yet shipped). Polled iter5 status=success (created 10:33:40 → success-update 10:38:13 ≈4m33s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test **292** baseline (45D+247I; +6 OfficeModulePermissionSeedTests) passed; `conclusion` empty — `tasks` endpoint terminal=`status:success` không populate `conclusion`, trust success; 292 INFERRED gate-passes-pre-build invariant NOT log-confirmed numeric). **★ BUNDLE BOTH FROZEN (BE-only ⟹ both MUST stay = pre-deploy live; verified AFTER status=success): admin `Wt54PHYl` UNCHANGED** ✓ **+ user `B99fMU6X` UNCHANGED** ✓ (no FE touch → no content-hash rotation; rotate-w/o-FE-change = anomaly, did NOT happen). Smoke **4×200**: api `/health/ready`+`/health/live` + admin root + eoffice root. **NO migration** — prod `__EFMigrationsHistory` top5 = `AddHoSoLinkToPurchaseEvaluation`(Mig52)→`AddDepartmentParentId`(51)→`ReplaceBudgetModuleWithPeWorkItemBudgets`(50)→`AddWorkItemToPurchaseEvaluation`(49)→`AddProjectMasterFields`(48) == repo HEAD GIỮ NGUYÊN ✓. **sys.tables=88** (sqlcmd COUNT excl mighist — unchanged, seed inserts/updates ROWS not tables). **🔑★ GOLIVE DB VERIFIED THỰC SỰ ÁP PROD (seed chạy lúc boot ⟹ proof = prod Permissions, NOT chỉ binary-shipped): sqlcmd `Roles` (13 total: Accounting/Admin/AuthorizedSigner/CatalogManager/CostControl/DeptManager/Director/Drafter/Equipment/Finance/HrAdmin/Procurement/ProjectManager). (1) ALLOW-LIST 16/16 R=1∧C=1 cho Drafter (non-admin demo-role) — Off,Off_Dashboard,Off_DanhBa,Off_PhongHop(+View+Book),Off_DeXuat(+List+Create+Inbox),Off_DonTu(+Leave+Ot+Travel),Off_DatXe,Off_ItTicket ✓. (2) CROSS-ROLE: MỌI 13 role count=16 R∧C (golive ÁP TOÀN BỘ không Drafter-fluke) ✓. (3) EXCLUDED-3 (Off_PhongHop_Manage/Off_AttendanceReport/Off_ChamCong) Drafter R=0∧C=0 ✓ + LEAK-check: CHỈ Admin có R=1 trên 3 key này (0 non-admin leak) ✓. (4) HRM/Personal Drafter KHÔNG mở: Hrm_Dashboard=0, all 7 Hrm_Config*=0, Personal=0 ✓; Hrm=1 + Hrm_HoSo=1 (S65 public-read GIỮ NGUYÊN không đổi) ✓. (5) Admin KHÔNG hạ: full Office incl 3 excluded all R=1∧C=1 ✓.** ⟹ boot-time seed CHẠY THẬT trên prod (app pool recycled w/ new binary post-deploy; nếu seed CHƯA chạy thì non-admin sẽ vẫn 0 — nhưng 16/16 across 13 roles = đã chạy). Menu-tree non-admin via API NOT tested (DB-query mục 4 đủ proof per spec). 0 regression. NO prod-data mutation ngoài chính golive seed (read-only curls + sqlcmd SELECT-only; seed-write là chủ đích của deploy). Visual menu-render = anh UAT. **LESSON: BE-only GOLIVE security-seed verify = both bundles MUST stay FROZEN (no FE → no rotate; rotate=anomaly) + Mig-top + sys.tables MUST stay prev (DbInitializer seed = runtime row insert/update, NOT EF-migration → KHÔNG advance __EFMigrationsHistory NOR sys.tables) + **CORE proof = prod Permissions DB query**, NOT bundle-frozen alone (frozen chỉ chứng minh FE không đổi, KHÔNG chứng minh API binary mới deploy + seed chạy — phải query Permissions cho non-admin role thấy allow-list grant + cross-role để loại fluke + leak-check excluded chỉ-Admin). upgrade-only seed THẮNG revoke vì chạy SAU trong SeedAsync. golive-state AMBIENT-after-this-deploy (parent commit c556f6c chưa có method này → pre-1f8947e non-admin Office = 0; bằng chứng "đã chạy" = 16/16). TOOLING (re-confirmed S70): Bash=POSIX (`$var`/`$env:` mangle through bash→ssh→PS layers) → write PS to `/tmp/x.ps1` + `powershell.exe -NoProfile -File "C:/.../Temp/x.ps1"` (⚠️ heredoc em-dash/non-ASCII corrupts → ASCII-only in PS source); jq ABSENT + system python3 broken → Invoke-RestMethod cho Gitea `tasks` (match `head_sha -eq sha`, `?limit=N` URI-direct); SSH→PS base64 `-EncodedCommand` (UTF-16LE `[Text.Encoding]::Unicode`) cho BOTH appsettings-read AND sqlcmd; sqlcmd pw-literal qua single-quote in B64-payload (no `$` survives clean), `-U/-P` direct + `-W -s '|'` pipe-delim + `-h -1` no-header; read DB pw from prod appsettings when `$PROD_DB_PASSWORD` empty. NEVER fixed code (READ-only).** Tag `[s69b, run307, pass, be-only-golive-vanphongso-public-read-create-allowlist16, SeedAllRolesOfficeModulePermissionsAsync-boot-after-revoke, bundle-BOTH-frozen-Wt54PHYl-B99fMU6X, no-mig-top-stays-mig52, tables88, GOLIVE-DB-VERIFIED-16of16-across-13-roles, excluded3-canread0-only-admin-leak0, hrm-personal-still-0, hrm-hoso-public-unchanged, admin-not-downgraded, test292-inferred, deploy11of11, seed-not-migration-no-history-advance, no-regression]`. - **2026-06-17 S70 Run #306 (run_number 306, id420) sha=`c556f6c` PASS ~4m42s (FE-ONLY re-skin "Văn phòng số" TOÀN MODULE 10 page PURO layout — PageHeader/KpiCard/WidgetCard apply + CSS Hồ sơ NS; presentation-only "phẫu thuật giữ nguyên logic", byte-identical logic. 20 .tsx: fe-admin 11 {AttendanceReport,InternalDirectory,ItTickets,MeetingCalendar,MeetingRooms,ProposalCreate,ProposalDetail,ProposalsList,WorkflowAppDetail,WorkflowAppsList}Page + fe-user 9 mirror (no AttendanceReport — admin-only) + 3 agent-memory .md (cicd/fe-designer/reviewer). NO BE/NO migration/NO new menu key. deploy 10/10 session after #297–#305 all PASS):** Push `a8bbdae..c556f6c` (1 commit, 23 files). `git diff --name-only a8bbdae c556f6c -- '*Migrations*' '*Persistence*'` = EMPTY ✓. `.tsx` non-ignored → full pipeline RAN (3 `.md` agent-memory match paths-ignore but co-mixed w/ 20 tsx → Discovery#3 range any-non-ignored ⟹ whole build). GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (path `C:\inetpub\solution-erp\api`, uid `vrapp` len24). Run IN-PROGRESS at spawn (status=running 09:57:54→09:58:40) — pre-deploy baseline captured BEFORE poll-loop: admin `Bl2o_kUq` (S69 #305 live) + user `BImrKQNn` (S69 #305 live) — both == spec baseline (still live, deploy not yet shipped, anti#3 honored no mid-flight verify). Polled status=success (started 09:57:54 → upd 10:02:36 ≈4m42s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test **286** baseline (45D+241I; FE-only ⟹ 0 BE call-site risk) passed; `conclusion` empty — `tasks` endpoint terminal=`status:success` không populate `conclusion`, trust success; 286 INFERRED gate-passes-pre-build invariant NOT log-confirmed). **★ BUNDLE BOTH ROTATE (FE-both-app 10-page re-skin ⟹ both MUST rotate; verified AFTER status=success + re-confirm STABLE 2nd-fetch identical no-transient — anti#3): admin ROTATE `Bl2o_kUq→Wt54PHYl` (css `CKzdDktL→BpHtX3vS`)** ✓ **+ user ROTATE `BImrKQNn→B99fMU6X` (css `eoxUcs8v→DXRSCQW7`)** ✓. BOTH required → BOTH did. Asset reachable 200: admin js 1,588,241b + user js 1,493,955b (not white). ⚠️ Prod CI hashes ≠ spec local-build (fe-user `C8-p69Kn`/`DezuRkK9`, fe-admin `yFhLO2Wp`/`Dd2WiO6n`) — EXPECTED CI-rebuild content-hash divergence (only matters NEW≠baseline, BOTH rotated confirmed). Smoke **4×200** health: api `/health/ready`+`/health/live` + admin root + eoffice root. **Office-backed API live (admin bearer): GET /api/proposals 200 · /api/it-tickets 200 · /api/meeting-rooms 200 · /api/employees 200** (data layer healthy, not 500). `/api/workflow-apps`→404 = WRONG-route-guess NOT regression (FE-only can't change BE routing; real route differs — leave-requests/don-tu; FE uses api-client wrapper, route grep didn't land). **SPA deep-link 6 Office routes admin all 200** (/proposals, /it-tickets, /workflow-apps/leave, /directory, /meeting-calendar, /office/dashboard serve index.html — not 404/white). **NO migration** — prod `__EFMigrationsHistory` top = `20260616035929_AddHoSoLinkToPurchaseEvaluation` (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact). **sys.tables=88 verified** (sqlcmd COUNT excl mighist — unchanged). **★ OFFICE-HIDDEN CONFIRMED (non-admin Drafter=nv.test): sqlcmd `Off_Dashboard` perm row ONLY for Admin (1/13 roles) ✓ + Drafter has ALL 17 Off_* keys present but EVERY one CanRead=0 ✓ → menu gates on CanRead=1 ⟹ non-admin sees NO Office menu.** ⚠️ **LESSON-CORRECTION vs S69: my coarse `GROUP BY COUNT(*) WHERE MenuKey LIKE 'Off%'` first showed "all 13 roles ~18 Off perms" — MISLEADING; those are perm-ROWS-with-CanRead=0 (existence ≠ access). Must filter `CanRead=1` OR inspect per-key. MenuItems.IsVisible=1 on all Off keys = admin-side global-visible flag; user-visibility = IsVisible AND role.CanRead. Office-hidden state is AMBIENT (FE-only commit cannot touch Permissions/MenuItems seed — git diff zero BE/DbInitializer) = identical pre-c556f6c.** 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual per-page render = anh user UAT (em chỉ confirm bundle shipped + route không 500/trắng per spec). **TOOLING (re-confirmed + NEW): Bash tool = POSIX bash → ⚠️ system `python3` BROKEN (ZKBioTime Python311 on PATH → `SRE module mismatch` fatal) AND `jq` NOT installed → MUST use `powershell.exe -NoProfile -File "C:/.../tmp/x.ps1"` + `Invoke-RestMethod` for ALL JSON parse (Gitea tasks match `head_sha -eq sha`); poll-loop in pure-bash w/ jq SILENTLY returns empty (no error) — verify parser works before trusting loop. SSH→PS base64 `-EncodedCommand` (UTF-16LE) for appsettings-read AND sqlcmd; sqlcmd string-LITERAL via `[char]39` concat (NOT doubled-quote); `-U/-P` direct (no -ConnectionString flag); `Roles`/`Permissions(RoleId,MenuKey,CanRead)`/`MenuItems([Key],IsVisible)`. NEVER fixed code (READ-only).** Tag `[s70, run306, pass, fe-only-vanphongso-reskin-10page-PURO, fe-both-app-bundle-BOTH-rotate-Wt54PHYl-B99fMU6X, asset-200-reachable, office-api-200-proposals-ittickets-meetingrooms-employees, workflow-apps-404-wrong-route-not-regression, spa-deeplink-6route-200, no-mig-top-stays-mig52, tables88-verified, office-hidden-confirmed-Off_Dashboard-admin-only-drafter-canread0, lesson-canread0-rows-not-access, ambient-not-deploy-caused, jq-absent-python3-broken-use-powershell, deploy10of10, no-regression, test286-inferred]`. - **2026-06-17 S69 Run #305 (run_number 305, id419) sha=`a8bbdae` PASS ~4-5m (FE BOTH-APP foundation "Văn phòng số" + index.css sync + BE menu-seed NO-mig: 3 shared component mới PageHeader/KpiCard/WidgetCard + OfficeDashboardPage landing route `/office/dashboard` 4-place-wire BOTH apps + `fe-admin/src/index.css` SYNCED (Hồ sơ NS accent tokens, rotate admin mạnh) + BE menu key `Off_Dashboard` (MenuKeys.cs L100 + DbInitializer.cs L1825 seed parent=`Off` Order0 LayoutDashboard); deploy 9/9 session after #297–#304 all PASS):** Push `764fe70..a8bbdae` (1 commit, 20 files). Diff: FE 14 files (2× {PageHeader,KpiCard,WidgetCard,OfficeDashboardPage,App.tsx,Layout.tsx,menuKeys.ts} + fe-admin index.css) + BE 2 (MenuKeys.cs + DbInitializer.cs) + 4 agent-memory `.md`. `.tsx`/`.cs`/`.css` non-ignored → full pipeline RAN (the 4 `**/*.md` agent-memory match paths-ignore but co-mixed w/ code → Discovery#3 range any-non-ignored ⟹ whole build). GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod `appsettings.Production.json`→`ConnectionStrings.Default` (path `C:\inetpub\solution-erp\api`, uid `vrapp`). Run IN-PROGRESS first 4 polls (running 09:26→09:29) — correctly did NOT verify-bundle-mid-flight (anti#3); pre-deploy baseline captured BEFORE poll-loop: admin `CNUv1jxY` (S78 #304 live) + user `CpOskeS1` (S78 #304 live) — both == spec S68 baseline (still live, deploy not yet shipped). Polled iter5 status=success (started ~09:25 → success 09:29:50 ≈4-5m). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate **286** baseline (45D+241I) passed; `conclusion` empty — `tasks` endpoint terminal=`status:success` doesn't populate `conclusion`, trust success; 286 INFERRED from gate-passes-pre-build invariant NOT log-confirmed numerically). **★ BUNDLE BOTH ROTATE (FE-both-app + index.css sync ⟹ both MUST rotate; verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no-transient — anti#3): admin ROTATE `CNUv1jxY→Bl2o_kUq`** ✓ (Văn phòng số + index.css shipped) **+ user ROTATE `CpOskeS1→BImrKQNn`** ✓ (foundation shipped). BOTH required → BOTH did. ⚠️ **Prod CI hashes (`Bl2o_kUq`/`BImrKQNn`) ≠ spec local-build (`TbkadgKd`/`DrxDysO7`) — EXPECTED divergence (CI rebuilds w/ different Node/npm/dep-resolution → different content-hash; only matters NEW≠baseline, BOTH rotated confirmed ship). Spec assumption "prod khớp local nếu source-identical" holds ONLY if byte-identical build-env — it is NOT (S77 same lesson).** Smoke **4×200**: api `/health/ready`+`/health/live` + admin root + eoffice root. **NO migration** — prod `__EFMigrationsHistory` top = `20260616035929_AddHoSoLinkToPurchaseEvaluation` (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (`git diff --name-only 764fe70 a8bbdae -- '*Migrations*'` = EMPTY; menu-seed is runtime-idempotent DbInitializer NOT EF-migration → top did NOT advance). **sys.tables=88 verified** (sqlcmd COUNT excl mighist — unchanged, menu-seed inserts rows not tables). **★ MENU-SEED VERIFIED (NEW check this run — DbInitializer seed ungated reaches prod by design): sqlcmd `SELECT [Key],ParentKey,Label,IsVisible FROM MenuItems WHERE [Key]='Off_Dashboard'` → 1 row Key=`Off_Dashboard` Parent=`Off` Label="Bảng điều khiển Văn phòng số" IsVisible=1 ✓.** **★ OFFICE-HIDDEN CONFIRMED (RevokeTemporarilyHiddenModulesAsync StartsWith("Off") scope — DbInitializer.cs L2172): sqlcmd `SELECT r.Name FROM Permissions p JOIN Roles r ON r.Id=p.RoleId WHERE p.MenuKey='Off_Dashboard'` → ONLY `Admin` (CanRead=1), 1 row / 13 roles total ✓ → non-admin NO Off_Dashboard perm → Office stays hidden (revoke executed, admin auto via All[]).** 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "Dashboard landing render / Hồ sơ NS CSS" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88+menu-seed+office-hidden. **LESSON: FE-both-app + index.css-sync + BE-menu-seed (NO-EF-mig) verify = both bundles MUST ROTATE + Mig-top + sys.tables MUST stay prev (menu-seed = DbInitializer runtime row-insert, NOT a migration → does NOT advance __EFMigrationsHistory NOR sys.tables; verify seed via direct MenuItems SELECT not via mig-count). Office-hidden = query Permissions-by-role: temporarily-hidden module has perm row ONLY for Admin (All[] auto-grant) after RevokeTemporarilyHidden runs. ⚠️ Prod CI bundle-hash ≠ local-build-hash is NORMAL — never FAIL on hash-mismatch-to-local, only FAIL if NOT-rotated-from-baseline. TOOLING (re-confirmed S78): Bash=POSIX (`$var`/`$env:`/`''` literal-quote mangle through bash→ssh→PS→sqlcmd layers) → write PS to `.ps1` + `powershell -File "ABS/FWD/SLASH.ps1"`; Gitea `tasks` via `Invoke-RestMethod` (match `head_sha -eq sha`, `?limit=N` honored URI-direct); SSH→PS base64 `-EncodedCommand` (UTF-16LE iconv) for BOTH appsettings-read AND sqlcmd; ⚠️ **sqlcmd string-LITERAL in query: doubled `''x''` BREAKS (PS single-quote-string closes early) → build literal via `[char]39 + "x" + [char]39` concat (NOT `''`)**; this sqlcmd build supports `-U/-P` direct (no -ConnectionString flag); table name `Roles` NOT `AspNetRoles`, `Permissions(RoleId,MenuKey,CanRead)`; CLIXML/Progress stdout-noise grep-filter out. NEVER fixed code (READ-only).** Tag `[s69, run305, pass, fe-both-app-vanphongso-foundation, shared-PageHeader-KpiCard-WidgetCard, index-css-sync-admin, bundle-BOTH-rotate-Bl2o_kUq-BImrKQNn, prod-ci-hash-NE-local-EXPECTED, be-menu-seed-Off_Dashboard-NO-mig, no-mig-top-stays-mig52, tables88-verified, menu-seed-verified-sqlcmd, office-hidden-confirmed-admin-only-1of13, sqlcmd-char39-literal-not-doubled-quote, deploy9of9, no-regression, test286-inferred]`. - **2026-06-16 S78 Run #304 (run_number 304, id418) sha=`37752eb` PASS ~4m (FE BOTH-APP 1-line CSS-precedence fix: "Hồ sơ NS" employee-detail banner NAME đen→trắng — `