vietreport-admin/solution-erp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
231a7b0c36917d13abbed42ef13d02c3297564e7
solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md

39 KiB
Raw Blame History

CI/CD Monitor Agent — Persistent Memory

Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 archive/ on-demand · L3 RAG search_memory just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → git d2f52ba + archive/2026-05-{runs,q2,q3,q4}.md.

🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.

🚨 Recurring CI/CD bug patterns (catch priority)

  • #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded .gitea/workflows/deploy.yml (pass #110). KHÔNG revert.
  • #40 npm cache tsc not found — build_fe_admin fail post cache: npm. DISABLED rolled back a21790d. KHÔNG re-enable.
  • #41 paths-ignore docs-only skip — code commit không trigger CI? Check git diff --name-only HEAD~1 HEAD vs paths-ignore: ['docs/**','**/*.md','.claude/skills/**']. Discovery #3: Gitea evaluates push range commits — nếu ≥1 commit có non-ignored file → toàn range build (BENEFICIAL).
  • #25 IIS WebSocket — notification-hub/negotiate 401/404 prod. Fix: WebSocket module enable web.config site api (skill iis-deploy-runbook).
  • #48 SQLite tie-break — OrderByDescending(CreatedAt).First() pick wrong khi 2+ .Add() cùng frozen-clock. Fix: discriminator filter .Where(Summary.Contains("Chuyển phase")) BEFORE OrderBy.
  • Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[a-z0-9]+\.js'. Fix: SSH Restart-WebAppPool. ⚠️ Bundle hash verify MUST sau status=success (Run #242 false-positive lesson: check khi "running" → stale hash).
  • Migration drift prod vs repo — compare ls .../Persistence/Migrations/*.cs vs sqlcmd __EFMigrationsHistory. Fix: check Program.cs app.MigrateDatabase() + app pool recycle.

📋 5-stage checklist (EVERY run)

  • Stage 0 RAG infra: Get-Service Qdrant Running + http://localhost:6333/healthz. Collection proj_solution_erp (prefix proj_* 7 project — Discovery #8).
  • Stage 1 Push+filter: git log -1 --format='%H %s' + git log origin/main..HEAD empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
  • Stage 2 Gitea poll (max 10 iter × 60s): API .../actions/tasks?limit=5 (NOT /runs 404). Match head_sha. ⚠️ task table updated_at stale ~2min (gotcha #46) → cross-check VPS mtime.
  • Stage 3 Test gate: baseline 130 PASS (58 Domain + 72 Infra). Phase 9 UAT exception lower OK (feedback_uat_skip_verify).
  • Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=accessToken route /api/auth/login) → 3-5 endpoint smoke 2XX (incl new) → FE bundle hash 2 app changed → SignalR negotiate (gotcha #25 if relevant) → EF mig prod==repo.
    • Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). sqlcmd -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'" → 0 rows = seed GATE BLOCKED → gotcha #51.
    • Discovery #4: ASP.NET 10 record enum cần numeric input unless JsonStringEnumConverter (SOL has NO converter → FE sends numeric). #5: sqlcmd ssh Windows-auth cần \\\\SQLEXPRESS 4-backslash. #6: INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST run, NOT inside if(!demoSeedDisabled); DEMO seed (DemoUsers/Contracts/PE) OK gated → gotcha #51.
  • Stage 5 Report PASS/FAIL + evidence + MEMORY update.

⚠️ Anti-patterns (DO NOT)

  1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

🧠 SOLUTION_ERP CI/CD essentials (S40 verified)

  • Gitea: git.baocaogiaoduc.vn/vietreport-admin/solution-erp · workflow .gitea/workflows/deploy.yml · paths-ignore ['docs/**','**/*.md','.claude/skills/**']
  • Prod: api/admin/eoffice .solutions.com.vn · SSH ssh vietreport-vps (Administrator, id_ed25519) · IIS site phys paths (S42 verified): API C:\inetpub\solution-erp\api · admin \fe-admin · user \fe-user (3 sites Started). DB .\SQLEXPRESS/SolutionErp/vrapp SQL-auth. Conn string key = ConnectionStrings.Default (NOT DefaultConnection!) — read pw from prod appsettings.Production.json when $env:PROD_DB_PASSWORD empty.
    • SSH→PS quoting (S42 lesson): nested bash→ssh→powershell mangles $var/\". Use iconv UTF-16LE | base64 → powershell -EncodedCommand $B64. Single-quote literal paths.
  • Tests baseline: 263 PASS (S62 Run #286 sha 7926c21 spec; 45 Domain + 218 Infra — em-main supplied; supersedes prev 228/240/256). CI gate runs both test projects BEFORE build/deploy → status=success ⟹ test gate passed (tasks endpoint reports terminal as status:success, conclusion field NOT populated). Local grep undercounts (Theory/InlineData) — trust CI conclusion. Phase 9 UAT mode skip per chunk OK.
  • Mig latest repo: Mig 52 20260616035929_AddHoSoLinkToPurchaseEvaluation (S65; PE +HoSoLink hyperlink-NAS, AddColumn-only no new table). Prev Mig 51 AddDepartmentParentId (S65 Department.ParentId loose-Guid org-tree, AddColumn-only) + Mig 50 ReplaceBudgetModuleWithPeWorkItemBudgets (S61 Budget→PeWorkItemBudgets net-reduce). Path src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/ (52 mig .cs non-designer total). Prod check sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5. ⚠️ Table-count: sys.tables (is_ms_shipped=0, excl mighist) = 88 (S62 Run #286 verified — S61 Budget-replace DROPPED tables 93→88). Narrative-93 is STALE pre-S61 — when commit touches no schema, 88 is correct, don't FAIL on 88↔93. Always cross-ref COMMIT scope vs ambient count.
  • Bearer: admin admin@solutions.com.vn/Admin@123456 (full) · UAT nv.test@solutions.com.vn/TestUser@123456 (Drafter CCM, gotcha #44 check)
  • Bundle hash live S72: admin xkSz9BfE (Run #298 sha 292d64d — ROTATED from BDwV5d0X, broke S68→S71 frozen-streak; FE-admin mirror EmployeesListPage + index.css accent 2 files) · user BumgrwCJ (Run #297 sha ab4e681 S71 — FROZEN through S72 deploy-2 since fe-user untouched). ⚠️ ASYMMETRIC-deploy lesson (S66): FE-one-app commit → that app's bundle MUST rotate + OTHER app MUST stay frozen; admin-rotate-when-only-fe-user-changed = anomaly → flag. S50 mid-deploy transient lesson: pre-success snapshot can show intermediate FE copy in-flight — re-confirm hash AFTER status=success ALWAYS (anti-pattern #3). FROZEN-expectation runs (BE-only or other-app): hash MUST stay = live pre-deploy value; rotate w/o relevant FE change = anomaly.
  • DB pw (S42, when $PROD_DB_PASSWORD empty): vrapp/buKL3TGBkD0wDDbYVw65QeX9 read from C:\inetpub\solution-erp\api\appsettings.Production.json→ConnectionStrings.Default. ⚠️ Skill-doc path C:\inetpub\apps\SolutionErp\Api is STALE → real path C:\inetpub\solution-erp\api. sqlcmd over SSH works direct (no UTF-16 encode needed). ⚠️ sys-catalog string-concat queries hit collation conflict (Latin1_General_CI_AS_KS_WS vs SQL_Latin1_General_CP1_CI_AS) → add COLLATE DATABASE_DEFAULT per concatenated column.

🔑 Critical config (flag commit nếu tái xuất)

Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)

🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)

Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.

📊 Run stats baseline

BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate.

📅 Recent runs (FIFO — older → archive/git)

  • 2026-06-16 S76 Run #302 (run_number 302, id416) sha=536dd6b PASS ~4m (FE BOTH-APP additive: PE "Link hồ sơ ổ mạng" render upgrade — đường-dẫn-ổ-mạng từ chữ+Copy → <a href=file://…> bấm-thử mở Explorer + GIỮ nút Copy dự phòng. 2 file PeDetailTabs.tsx fe-user+fe-admin SHA256-IDENTICAL b415023b…, +46/−14 mỗi file, NO BE/mig/index.css/Employee-page; deploy 6/6 session after #297/#298/#299/#300/#301 all PASS): Push 2 files {fe-admin,fe-user}/src/components/pe/PeDetailTabs.tsx. .tsx → not in paths-ignore → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default. Run IN-PROGRESS first 6 polls (running 14:54:20→14:56:57) — correctly did NOT verify-bundle-mid-flight (anti#3); pre-deploy baseline captured BEFORE poll-loop: admin I1fpLeYw (S75 #301 live) + user DrQYkzh0 (S75 #301 live) — both == spec baseline. Polled iter7 status=success (started ~14:53 → success 14:57:13 ≈4m). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 286 baseline (45D+241I; FE-only ⟹ 0 BE call-site risk) passed; conclusion empty — tasks endpoint terminal=status:success doesn't populate conclusion, trust success). ★ BUNDLE BOTH ROTATE (the change-point — FE-BOTH-app, both PeDetailTabs changed ⟹ both bundles MUST rotate; verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient — anti#3): admin ROTATE I1fpLeYw→CcrZqfht ✓ (file:// link shipped) + user ROTATE DrQYkzh0→DniDFUB_ ✓ (same component shipped). BOTH required per spec → BOTH did (mirror S74/S75 pure-FE-both-app pattern — both same-SHA256 file ⟹ both rotate; frozen sibling here = ship-fail flag). Smoke 4×200: api /health/ready+/health/live + admin root + eoffice root. NO migration — prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (git diff --name-only 536dd6b~1 536dd6b -- '*Migrations*' = EMPTY; FE cannot alter schema → top did NOT advance; prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact). sys.tables=88 verified (sqlcmd COUNT excl mighist — unchanged, no schema touch). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "link file:// bấm mở Explorer / nút Copy" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88; SHA256-identical-between-2-apps is a SOURCE-claim (git), not runtime DOM-equality (file:// scheme browsers thường block từ https-origin → click có thể no-op tùy browser, đó là lý do GIỮ Copy dự phòng — không kiểm chứng được qua curl). LESSON: pure-FE-BOTH-app additive verify (6th consecutive deploy this session, both apps same component SHA256-identical) = both bundles MUST ROTATE (≠ asymmetric one-app where sibling frozen; ≠ BE-only where both frozen). Migration top + sys.tables MUST stay = prev (FE-only). No BE call-site/DTO/endpoint smoke (no API surface — render-only). TOOLING (re-confirmed S75): Bash tool EATS inline $var/$env: in powershell -Command → write PowerShell to .ps1 + run -File; certutil -hashfile … SHA256 for SHA256 (NOT findstr-pipe — quoting breaks); parse Gitea tasks via Invoke-RestMethod+native object (match head_sha -eq sha, limit=N ignored); SSH→sqlcmd base64 EncodedCommand (UTF-16LE, no $ in B64 passes bash clean), CLIXML progress-stream stdout harmless; read DB pw from prod appsettings when PROD_DB_PW empty (path C:\inetpub\solution-erp\api, key ConnectionStrings.Default). NEVER fixed code (READ-only). Tag [s76, run302, pass, fe-both-app-pe-link-hoso-file-scheme, fe-both-2files-sha256-identical-b415023b, bundle-BOTH-rotate-CcrZqfht-DniDFUB_, no-mig-top-stays-mig52, tables88-verified, deploy6of6, no-regression, test286].
  • 2026-06-16 S75 Run #301 (run_number 301, id415) sha=6df1b2d PASS ~2.5m (FE BOTH-APP additive: PE "Link hồ sơ" auto-detect render — http(s)→hyperlink bấm-mở (giữ nguyên) / đường dẫn ổ mạng O:\...→chữ+nút Copy. 2 file PeDetailTabs.tsx fe-user+fe-admin SHA256-IDENTICAL da0884a5…, NO BE/mig/index.css/Employee-page; deploy 5/5 session after #297/#298/#299/#300 all PASS): Push 2 files {fe-admin,fe-user}/src/components/pe/PeDetailTabs.tsx. .tsx → not in paths-ignore → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default. Run IN-PROGRESS first poll (running 14:31) — correctly did NOT verify-bundle-mid-flight (anti#3); pre-deploy baseline captured BEFORE poll-loop: admin PxiZQkaw (S74 #300 live) + user B36hGoKd (S74 #300 live) — both == spec baseline. Polled iter4 status=success (started ~14:31 → success 14:33:21 ≈2.5m — fastest streak, FE-only no big BE build). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 286 baseline (45D+241I; FE-only ⟹ 0 BE call-site risk) passed; conclusion empty — tasks endpoint terminal=status:success doesn't populate conclusion, trust success). ★ BUNDLE BOTH ROTATE (the change-point — FE-BOTH-app, both PeDetailTabs changed ⟹ both bundles MUST rotate; verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient — anti#3): admin ROTATE PxiZQkaw→I1fpLeYw ✓ (Link-hồ-sơ auto-detect shipped) + user ROTATE B36hGoKd→DrQYkzh0 ✓ (same component shipped). BOTH required per spec → BOTH did (mirror S74 pure-FE-both-app pattern — both same-SHA256 file ⟹ both rotate; frozen sibling here = ship-fail flag). Smoke 3×200: api /health/ready + admin root + eoffice root. NO migration — prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (git diff --name-only 6df1b2d~1 6df1b2d -- '*Migrations*' = EMPTY; FE cannot alter schema → top did NOT advance; prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact). sys.tables=88 verified (sqlcmd COUNT excl mighist — unchanged, no schema touch). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "hyperlink bấm-mở / nút Copy hiện đúng" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88; SHA256-identical-between-2-apps is a SOURCE-claim (git), not runtime DOM-equality (auto-detect http-vs-O:\ branching is component logic, not provable by curling bundle). LESSON: pure-FE-BOTH-app additive verify (5th consecutive deploy this session, both apps same component SHA256-identical) = both bundles MUST ROTATE (≠ asymmetric one-app where sibling frozen; ≠ BE-only where both frozen). Migration top + sys.tables MUST stay = prev (FE-only). No BE call-site/DTO/endpoint smoke (no API surface — render-only change). TOOLING (re-confirmed S74): Bash tool EATS inline $var/$env: in powershell -Command → write PowerShell to .ps1 + run -File; parse Gitea tasks via Invoke-RestMethod+native object (match head_sha -eq sha, limit=N ignored); SSH→sqlcmd base64 EncodedCommand (UTF-16LE, no $ in B64 passes bash clean), CLIXML progress-stream stdout harmless; read DB pw from prod appsettings when PROD_DB_PW empty (path C:\inetpub\solution-erp\api). NEVER fixed code (READ-only). Tag [s75, run301, pass, fe-both-app-pe-link-hoso-autodetect, fe-both-2files-sha256-identical, bundle-BOTH-rotate-I1fpLeYw-DrQYkzh0, no-mig-top-stays-mig52, tables88-verified, deploy5of5, fastest-2.5m, ps1-file-not-inline-dollar, no-regression, test286].
  • 2026-06-16 S74 Run #300 (id414) sha=91aaf05 PASS ~5m (FE BOTH-APP list-redesign: bảng list 3-cột → flex-row gọn hết-tràn-ngang-rail + đồng nhất cỡ chữ (header text-xl→lg, list name 13px) — 2 file EmployeesListPage.tsx fe-user+fe-admin SHA256-IDENTICAL 37d7cc6c…, NO BE/mig/index.css; deploy 4/4 session after #297/#298/#299 all PASS): Push 2 files {fe-admin,fe-user}/src/pages/hrm/EmployeesListPage.tsx. .tsx → not in paths-ignore → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default (buKL3...buKL3TGBkD0wDDbYVw65QeX9). Run IN-PROGRESS first poll (running 14:22) — correctly did NOT verify-bundle-mid-flight (anti#3); pre-deploy baseline snapshot captured BEFORE poll-loop: admin xkSz9BfE (S72 #298 live) + user BumgrwCJ (S71 #297 live, FROZEN-streak through S73 BE-only) — both == spec baseline. Polled iter5 status=success (started ~14:20 → success ~14:25 ≈5m). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 286 baseline (45D+241I; FE-only ⟹ 0 BE call-site risk) passed; conclusion empty — tasks endpoint terminal=status:success doesn't populate conclusion, trust success). ★ BUNDLE BOTH ROTATE (the change-point — FE-BOTH-app, both pages changed ⟹ both bundles MUST rotate; verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient — anti#3): admin ROTATE xkSz9BfE→PxiZQkaw ✓ (EmployeesListPage flex-row shipped) + user ROTATE BumgrwCJ→B36hGoKd ✓ (same page shipped — user un-froze from S71-S73 frozen-streak, correct now fe-user finally changed). BOTH required per spec → BOTH did. ⚠️ Distinct from S68 (cross-stack BE+FE both-rotate) — this is PURE-FE-both-app both-rotate (no BE/mig). Smoke 3×200: api /health/ready + admin root + eoffice root. NO migration — prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (commit 0 mig files; FE cannot alter schema → top did NOT advance; prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact). sys.tables=88 verified (sqlcmd COUNT excl mighist — unchanged, no schema touch). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "flex-row gọn/hết-tràn-rail/đồng nhất cỡ chữ" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88; SHA256-identical-between-2-apps is a SOURCE-claim (git), not runtime DOM-equality. **LESSON: pure-FE-BOTH-app verify (both apps same file, SHA256-identical) = both bundles MUST ROTATE (≠ asymmetric S70/S71 one-app where sibling stays frozen; ≠ S73 BE-only where both stay frozen) — a frozen sibling here would = ship-fail flag; user un-froze from multi-session frozen-streak = EXPECTED when fe-user finally changes (streak-break normal). Migration top + sys.tables MUST stay = prev (FE-only). No BE call-site/DTO/endpoint smoke (no API surface). TOOLING (re-confirmed S73): Bash tool wrapper EATS inline $var/$matches[1]→[1] in powershell -Command strings AND $env:→mangled — MUST write PowerShell to a .ps1 file + run -File (no inline $ survives bash eval); python3 broken on box. Parse Gitea tasks via Invoke-RestMethod+native object (match by head_sha -like sha*); limit=N ignored. SSH→sqlcmd: base64-encode (UTF-16LE via [Convert]::ToBase64String([Text.Encoding]::Unicode...)) the query .ps1 → ssh vps "powershell -EncodedCommand $B64" (B64 has no $ so passes bash clean); CLIXML progress-stream on stdout harmless, data lines clean. Read DB pw via ssh→appsettings.Production.json ConvertFrom-Json .ConnectionStrings.Default when PROD_DB_PW empty (path C:\inetpub\solution-erp\api). NEVER fixed code (READ-only). Tag [s74, run300, pass, fe-both-app-list-redesign-flexrow, fe-both-2files-sha256-identical, bundle-BOTH-rotate-PxiZQkaw-B36hGoKd, user-unfroze-from-s71streak, no-mig-top-stays-mig52, tables88-verified, deploy4of4, ps1-file-not-inline-dollar-eaten, no-regression, test286].
  • 2026-06-16 S73 Run #299 (id413) sha=bcd619d PASS ~3m (TESTS-ONLY BE: +3 files tests/SolutionErp.Infrastructure.Tests/Application/{DepartmentTreeTests,PeHoSoLinkTests,HrmProfilePermissionSeedTests}.cs = +23→286 total (45D+241I). NO prod code, NO FE, NO migration; deploy 3/3 session after #297/#298 both success): .cs not in paths-ignore → full pipeline RAN. Run IN-PROGRESS first poll (running 14:06) → polled iter4 status=success (~14:09 ≈3m); conclusion empty (terminal=success, trust). CI test gate = the focus: gate runs BOTH test projects BEFORE build ⟹ status=success ⟹ 23 new tests passed on CI (not just local 286). Could NOT extract exact CI count — Gitea runs/413/logs+jobs endpoints all 404 anon (logs web-UI-only without auth token; GITEA_TOKEN empty). Inferred 286 from success (gate-passes-pre-build invariant) — NOT log-confirmed numerically. ★ BUNDLE BOTH-FROZEN (BE-only ⟹ neither app may rotate), verified AFTER status=success +2nd-fetch STABLE no transient (anti#3): admin xkSz9BfE FROZEN ✓ + user BumgrwCJ FROZEN ✓ (==pre-deploy baseline both; rotate-on-BE-only = anomaly→none). Smoke 4×200: api /health/ready+/health/live+admin root+eoffice root. NO migration — prod __EFMigrationsHistory top=AddHoSoLinkToPurchaseEvaluation(Mig52)==repo HEAD GIỮ NGUYÊN ✓ (prev-2 Mig51 AddDepartmentParentId+Mig50 ReplaceBudgetModule… chain intact; commit 0 mig files). sys.tables=88 ✓ (sqlcmd excl mighist, unchanged). 0 regression, NO prod-data mutation (read curls+SELECT-only). TOOLING LESSON (supersedes S72 #298): python3/py -3 on this box = BROKEN ZKBioTime embed (AssertionError: SRE module mismatch on import re/json) — version banner works but stdlib import dies. USE PowerShell Invoke-RestMethod+native object access (NOT ConvertFrom-Json on cached temp file — temp got stale snapshot once) to parse Gitea tasks; limit=N param IGNORED (returns full 299, match by id anyway). SSH→sqlcmd: nested bash→ssh→PS ate $var (S42) → use iconv UTF-16LE|base64→powershell -EncodedCommand; CLIXML progress-stream noise on stdout is harmless, data lines clean. Read DB pw from prod appsettings.Production.json→ConnectionStrings.Default when PROD_DB_PW empty (path C:\inetpub\solution-erp\api). NEVER fixed code (READ-only). Tag [s73, run299, pass, tests-only-be, +23-test-286-CI-gate-inferred-not-logcount, bundle-both-frozen-xkSz9BfE-BumgrwCJ, no-mig-top-mig52, tables88, deploy3of3, python3-broken-use-powershell-IRM, no-regression].
  • 2026-06-16 S72 Run #298 (run_number 298, id412) sha=292d64d PASS ~4m20s (FE-Admin MIRROR "Hồ sơ Nhân sự" master-detail từ fe-user: overwrite EmployeesListPage.tsx cũ 1200→1602 dòng SHA256-IDENTICAL với fe-user + index.css +4 accent palette teal/amberx/violet/greenx + utility .icon-chip/.app-gradient-brand/.card-accent/.stat-value — FE-ADMIN-ONLY 2 files, NO BE, NO migration, NO fe-user; ASYMMETRIC NGƯỢC deploy-1 S71 #297 (đó fe-user-rotate/admin-frozen, đây admin-rotate/user-frozen)): Deploy 2/2 trong session (deploy-1 = ab4e681 #297 PASS fe-user BumgrwCJ; #297=success TRƯỚC push này nên KHÔNG bị cancel — confirmed in tasks list). Push 2 files fe-admin/src/index.css(+81) + fe-admin/src/pages/hrm/EmployeesListPage.tsx(1020+/537−, 1476-line overwrite). .tsx+.css → not in paths-ignore → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default (buKL3...buKL3TGBkD0wDDbYVw65QeX9). Run IN-PROGRESS first poll (running 13:52) — correctly did NOT verify-bundle-mid-flight (anti#3); pre-deploy baseline snapshot captured BEFORE poll: admin BDwV5d0X (S68 frozen-since, must rotate) + user BumgrwCJ (S71 deploy-1 live, must stay frozen). ⚠️ POLL-PARSER BUG: my tr ','|grep -A4 '"id":412' returned empty status ×10 iter (false negative — grep -A on tr-newlined JSON misanchors) → re-queried with python3 json.load → status=success (started 13:51:27 → updated 13:55:47 ≈4m20s). LESSON-tooling: parse Gitea tasks JSON via python3 NOT tr/grep -A (multi-field-per-object splits wrong); always cross-check with a clean parser before trusting empty-status. CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 263 baseline (45D+218I; FE-only ⟹ 0 BE call-site risk) passed; conclusion empty — trust success). ★ BUNDLE ASYMMETRIC NGƯỢC-deploy-1 (the change-point — FE-one-app verify) ALL PASS, verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient (anti#3): admin ROTATE BDwV5d0X→xkSz9BfE ✓ (EmployeesListPage mirror + accent index.css shipped — admin un-froze from S68's BDwV5d0X frozen-streak S68→S71, now rotated correctly cho fe-admin change) + user FROZEN BumgrwCJ==deploy-1-baseline ✓ (fe-user 0 files touched deploy-2 → MUST NOT rotate = correct; rotate-when-untouched = anomaly→flag). Health api live+ready 200/200 (real endpoints /health/ready+/health/live, NOT /health) + admin/eoffice root 200. NO migration — prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact; commit 0 mig files — FE cannot alter schema → top did NOT advance). sys.tables=88 verified (sqlcmd COUNT excl mighist — unchanged, no schema touch). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "mirror identical / chữ xanh đậm brand-800 / accent đẹp" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88; SHA256-identical-with-fe-user is a SOURCE-claim (git diff), not runtime-verified by me (can't compare rendered DOM cross-app). LESSON: deploy-2 of same-session asymmetric-NGƯỢC = (a) verify prior deploy (#297) NOT cancelled by this push (it was success first — confirmed in tasks); (b) status=success ⟹ deploy ran; (c) THIS-app bundle ROTATE + OTHER-app FROZEN — direction FLIPS vs deploy-1 (deploy-1 fe-user-rotate/admin-frozen → deploy-2 admin-rotate/user-frozen); admin un-froze from multi-session frozen-streak = EXPECTED when fe-admin finally changes (frozen-streak break is normal, not anomaly); (d) migration top + sys.tables MUST stay = prev (FE-only). Re-query Gitea JSON with python3 when tr/grep parse yields empty — parser bug ≠ run-stuck. No BE call-site/DTO/endpoint smoke (no API surface). SHA256-source-identical = git-level claim, runtime-mirror-equality not provable by curl. NEVER fixed code (READ-only). Tag [s72, run298, pass, fe-admin-mirror-hoso-from-feuser, fe-admin-only-2files, bundle-asymmetric-NGUOC-admin-rotate-user-frozen, admin-unfroze-from-s68-streak, no-mig-top-stays-mig52, tables88-verified, deploy2of2-prior297-not-cancelled, poll-parser-bug-use-python3, no-regression, test263].
  • 2026-06-16 S71 Run #297 (run_number 297, id411) sha=ab4e681 PASS ~4m21s (FE-User "Hồ sơ NS" cosmetic: đồng nhất font/size + chữ đen text-slate-900/800→xanh-đậm text-brand-800 cho value/tên + bỏ text-xs nhánh mono — FE-USER-ONLY 1 file, NO BE, NO migration; 4th consecutive same-page S68→S71): Push 1 file fe-user/src/pages/hrm/EmployeesListPage.tsx (className màu chữ + size, KHÔNG đụng BE/Mig/dep/fe-admin). .tsx → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default (len24). Run IN-PROGRESS first poll (running 13:43) — correctly did NOT verify-bundle-mid-flight (anti#3), pre-deploy baseline snapshot captured BEFORE poll iter0: user DbVv6rsf (S70 baseline) + admin BDwV5d0X. Polled iter5 status=success (started ~13:42 → success ~13:46:48 ≈4m21s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 263 baseline (45D+218I; FE-only ⟹ 0 BE call-site risk) passed; conclusion empty — trust success). ★ BUNDLE ASYMMETRIC (the change-point — FE-one-app verify) ALL PASS, verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient (anti#3): user ROTATE DbVv6rsf→BumgrwCJ ✓ (EmployeesListPage cosmetic shipped — re-rotated from S70's DbVv6rsf, 4th consecutive same-page FE-user deploy = new content-hash each, normal) + admin FROZEN BDwV5d0X==baseline ✓ (fe-admin 0 files touched → MUST NOT rotate = correct; rotate-when-untouched = anomaly→flag). Health api live+ready 200/200 (⚠️ /health spec'd → 404, real endpoint /health/ready+/health/live per iis-deploy-runbook skill — both 200, API healthy) + admin/eoffice root 200. NO migration — prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (commit 0 mig files — git diff --name-only | grep Migrations/ = NONE; FE-only cannot alter schema → top did NOT advance; prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact). sys.tables=88 verified (sqlcmd COUNT excl mighist — unchanged, no schema touch). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "chữ xanh đậm/đồng nhất size" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged+tables88. LESSON: pure-FE-one-app cosmetic verify (4th consecutive same-page) = (a) status=success ⟹ deploy ran; (b) target-app bundle ROTATE + sibling-app FROZEN (re-rotate of same page across sessions = NORMAL, each deploy=new hash; rotate-sibling-when-untouched = anomaly); (c) migration top + sys.tables MUST stay = prev (FE cannot alter schema — advancing = bug); (d) health 200×2 (/ready+/live, NOT /health). ⚠️ /health 404 ≠ unhealthy — SOL uses /health/ready+/health/live (skill-doc), spec literal /health is wrong-path; cross-check skill before flagging. No BE call-site/DTO/endpoint smoke needed (no API surface). Tokens empty: anon Gitea + prod-appsettings DB pw works. NEVER fixed code (READ-only). Tag [s71, run297, pass, fe-user-hoso-cosmetic-brand800, fe-only-1file, bundle-asymmetric-user-rotate-admin-frozen, no-mig-top-stays-mig52, tables88-verified, health-ready-not-health, no-regression, test263].
  • 2026-06-16 S70 Run #295 (run_number 295, id409) sha=456c7a7 PASS ~4m21s (FE-User "Hồ sơ Nhân sự" layout 2-cột: cây tổ chức + list chồng cột trái, detail cột phải + tô màu panel chi tiết — FE-USER-ONLY 1 file, NO BE, NO migration; follow-up S69 Run #294 same page): Push ec517f7..456c7a7 1 file fe-user/src/pages/hrm/EmployeesListPage.tsx (309+/262−, 571-line layout rewrite). .tsx → full pipeline RAN. Both tokens SET this run (GITEA_TOKEN+PROD_DB_PW present — unlike S65-S69 empty; DB pw still works via prod appsettings.Production.json→ConnectionStrings.Default buKL3... len24). Run IN-PROGRESS first poll (running 11:36) — correctly did NOT verify-bundle-mid-flight (anti#3), polled iter6 status=success (started ~11:36 → success 11:40:46 ≈4m21s). Pre-deploy baseline snapshot iter0: user CZfo_PFZ (S69 baseline) + admin BDwV5d0X — captured BEFORE poll, correct timing. CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 263 baseline (45D+218I; FE-only ⟹ 0 BE call-site risk) passed; conclusion empty — trust success). ★ BUNDLE ASYMMETRIC (the change-point — FE-one-app verify) ALL PASS, verified AFTER status=success +re-confirm STABLE 2nd-fetch identical no transient (anti#3): user ROTATE CZfo_PFZ→DbVv6rsf ✓ (EmployeesListPage layout-2col shipped — re-rotated from S69's CZfo_PFZ, consecutive FE-user deploys on SAME page = new content-hash each, normal) + admin FROZEN BDwV5d0X==baseline ✓ (fe-admin 0 files touched → MUST NOT rotate = correct; rotate-when-untouched = anomaly→flag). Health api live+ready 200/200 + admin/eoffice root 200. NO migration — prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation (Mig 52) == repo HEAD GIỮ NGUYÊN ✓ (commit 0 mig files — git diff --name-only | grep Migrations/ = NONE; FE-only cannot alter schema → top did NOT advance past S68's Mig 52; prev-2 = AddDepartmentParentId Mig51 + ReplaceBudgetModuleWithPeWorkItemBudgets Mig50 chain intact). sys.tables stays 88 by construction (no schema touch — skipped explicit count, FE-only). 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only). Visual "2-cột đẹp/tô-màu" NOT verified (anh xem mắt) — only ship+rotate+health+mig-unchanged. LESSON: pure-FE-one-app verify (3rd consecutive same-page FE-user S68→S69→S70) = (a) status=success ⟹ deploy ran; (b) target-app bundle ROTATE + sibling-app FROZEN (asymmetric — re-rotate of already-rotated page across sessions is NORMAL, each deploy=new hash; rotate-sibling-when-untouched = anomaly); (c) migration top MUST stay = prev (FE cannot alter schema — advancing top on FE-only commit = bug); (d) health 200×3. No BE call-site/DTO/endpoint smoke needed (no API surface). Tokens-present vs empty: both paths work — when GITEA_TOKEN set use token header, but anon also fine for public repo. NEVER fixed code (READ-only). Tag [s70, run295, pass, fe-user-hoso-2col-layout, fe-only-1file, bundle-asymmetric-user-rotate-admin-frozen, no-mig-top-stays-mig52, tables88, no-regression, test263, tokens-present].
  • 2026-06-16 S68 Run #293 (run_number 293, id407) sha=5a0aaa4 PASS ~5m00s (FE-User "Hồ sơ Nhân sự" 3-panel master-detail 5-tab rewrite + PE "Link hồ sơ" hyperlink NAS + rename "Dự trù PRO"→"Ngân sách PRO" — CROSS-STACK BE Mig 52 + FE×2; 2-commit 318860a(FE-User EmployeesListPage) + 5a0aaa4(PE cross-stack)): Push 6ce5803..5a0aaa4 15 files: BE PurchaseEvaluation.cs(+HoSoLink) + PurchaseEvaluationConfiguration.cs + PurchaseEvaluationFeatures.cs(Create/Update cmd +trailing optional HoSoLink=null) + PurchaseEvaluationDtos.cs(Detail DTO +hoSoLink) + Mig 52 20260616035929_AddHoSoLinkToPurchaseEvaluation(3-file) + FE EmployeesListPage.tsx(fe-user, 3-panel rewrite cây-tổ-chức+list+5-tab) + PE×2 (PeDetailTabs.tsx+PeWorkspaceCreateView.tsx+types/purchaseEvaluation.ts both apps, mục E hyperlink + label rename). .cs+.tsx+Mig → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW empty → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default (buKL3...). Run IN-PROGRESS first poll (running 11:15) — correctly did NOT verify-bundle-mid-flight (anti#3), polled iter5 status=success (started 11:13:57 → success 11:18:57 ≈5m). CI gate (both proj pre-deploy ⟹ status=success ⟹ test gate 263 baseline (45D+218I; BE Create/Update +trailing optional param=null ⟹ call-sites UNBROKEN) passed; conclusion empty — trust success). ★ MIGRATION APPLIED (the KEY): prod __EFMigrationsHistory top = 20260616035929_AddHoSoLinkToPurchaseEvaluation ✓ (Mig 52; ADVANCED from prev top Mig 51 AddDepartmentParentId; DbInitializer auto-migrate on startup ran — api ready=200 confirms boot OK post-mig). Mig content = AddColumn HoSoLink nvarchar(1000) maxLength=1000 NULL (AddColumn-only, NO new table → sys.tables stays 88 verified). DB-level: sys.columns WHERE object_id=OBJECT_ID('PurchaseEvaluations') AND name='HoSoLink' = 1 row ✓ (column physically present prod). ★ BUNDLE BOTH ROTATE (verified AFTER status=success +re-confirm STABLE 2nd-fetch no transient — anti#3): admin Df06fmpq→BDwV5d0X ✓ (PE files shipped — was FROZEN since S67, now correctly rotated) + user DxK3fCfh→DXkyUjtQ ✓ (EmployeesListPage + PE files shipped). Both required to rotate per spec → BOTH did. Health api live+ready 200/200 + admin/eoffice root 200. ★ NEW-FIELD live smoke (admin bearer, route /api/auth/login field accessToken len468): GET /api/purchase-evaluations/{48154149...} (phiếu thật PE/2026/A/015) = 200 ✓ JSON detail carries "hoSoLink":null (key PRESENT, null cho phiếu cũ = backward-compat ✓ — Detail DTO wired AND Create/Update signature-change did NOT break GET path) · GET /api/departments/tree = 200 ✓ (Phase B dependency from Mig 51, still live). 0 regression. NO prod-data mutation (read-only GETs + sqlcmd SELECT-only). LESSON: nullable-AddColumn cross-stack verify = (a) __EFMigrationsHistory top ADVANCED to new Mig (Mig 52) + api ready=200 (auto-migrate boot OK); (b) sys.tables=88 unchanged (AddColumn ≠ new table) + sys.columns confirms col present; (c) Detail DTO field-presence via authed GET on a REAL phiếu — "hoSoLink":null for old rows proves both DTO-wiring AND backward-compat (trailing-optional Create/Update param won't break GET — confirmed 200); (d) cross-stack BE+FE×2 = BOTH bundles rotate (admin un-froze from S67). Tokens empty: anon Gitea + prod-appsettings DB pw works (Bash tool is POSIX bash NOT PS despite shell=PowerShell env — $env:/jq/python3 unavailable; parse via tr ','|grep + read pw via ssh→appsettings). Tag [s68, run293, pass, pe-hosolink-mig52-applied, fe-user-hoso-3panel-rewrite, pe-link-hoso-hyperlink, rename-dutru-to-ngansach, bundle-both-rotate-admin-unfroze, hosolink-null-in-dto-backward-compat, no-new-table-88, test263].
  • (S67 Run #292 [dept-parentid Mig51-applied-after-#291-fail, retry-chain CS7036-fix, bundle-asymmetric, tree-endpoint-200] + S65 #289 [hrm-hoso public-readonly seed-only, asymmetric read200/write403] → FIFO-trimmed, full verbatim git d2f52ba + archive/2026-06.md)
  • Older runs (S66 #290 ← S62 #286 06-13 ← … → S29 #232) full verbatim archived → git d2f52ba + archive/2026-06.md (incl #291 06-16 FAIL forensic [lesson=gotcha #65 + #292-inline] + #383 ex-VITRILAC); pre-S38 → archive/2026-05-{runs,q2,q3,q4}.md.

🔄 Curate trigger

  • ~30KB → archive recent runs → L2 archive/<period>.md. Dup failure patterns → merge. Stale >3mo → remove.

  • Last curate: 2026-06-16 S66 em main (86.8→29.2KB — 2.9× over-cap fix): byte-exact sed move Run #286→#232 (S62→S29, incl #291 forensic [lesson=gotcha #65 + #292-inline] + #383 ex-VỊ-TRÍ-LẠC) → archive/2026-06.md; refreshed essentials Mig 50→52 + bundle-live S69→S70 #295. Kept baseline (gotcha patterns + Stage 0-5 + Stage 4.6 + 10-point + Discovery #4-8) + 6 current 06-16 runs #289-#295.
  • Prev curate: 2026-05-29 S40 em main proxy (35.3→~21KB): archived Run #359/#243/#242/#241/#240 + S35/S36 startup → q4 + git d2f52ba; refreshed stale 120→130 test + Mig 34→40. Prev: S34 q3 · S32 q2 · S22 runs.