vietreport-admin/solution-erp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b128f30a35d51e70d74e3da1f60dde050a685531
solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md
pqhuy1987 b128f30a35 wal: flush 20260714T1129
2026-07-14 11:29:32 +07:00

19 KiB
Raw Blame History

CI/CD Monitor Agent — Persistent Memory

Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 archive/ on-demand · L3 RAG search_memory just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → git d2f52ba + archive/2026-05-{runs,q2,q3,q4}.md + archive/2026-06.md + archive/2026-07.md. 🗺️ Lookup map: archive/_INDEX.md — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + 2026-0{5,6}.gist.md theo nhu cầu. See 🔄 Curate-trigger log for archive-move history.

🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.

⚠️ cicd-lane return-garble tracker (#53)

cicd-spawn đôi khi trả memory-curation slice ("Compressing S112/S111") thay verdict (×5/6 qua S114; cumulative ×18 all-agents → feedback_agent_return_garble_recover, root-cause = curate-at-return). Recover: em-main-solo curl-verify. S115 (#385)+S116 (#386)+S117 (#387) cicd-lane CLEAN-return no-garble ×3 — verdict delivered OK.

🚨 Recurring CI/CD bug patterns (catch priority)

  • #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded .gitea/workflows/deploy.yml. KHÔNG revert.
  • #40 npm cache tsc not found — build_fe_admin fail post cache: npm. DISABLED rolled back a21790d. KHÔNG re-enable.
  • #41 paths-ignore docs-only skip — code commit không trigger CI? Check git diff --name-only HEAD~1 HEAD vs paths-ignore 7-entry từ D10 S111 8aa3869 (#495): ['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']. Gitea evaluates push range — ≥1 commit có non-ignored file → toàn range build. ⚠️ AS-6 BASELINE SHIFT: từ #495, memory-push chứa memory-budget.json/agent-memory-file/run-trace HẾT trigger CI (pattern #490/#492 DEAD).
  • #25 IIS WebSocket — notification-hub/negotiate 401/404 prod. Fix: WebSocket module enable web.config site api (skill iis-deploy-runbook).
  • #48 SQLite tie-break — OrderByDescending(CreatedAt).First() pick wrong khi 2+ .Add() cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy.
  • Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).
    • 🔴 #69 (S72 #312) — FE bundle hash NON-DETERMINISTIC per rebuild. deploy.yml Remove-Item fe-*\* -Exclude web.config+Copy-Item dist\* runs UNCONDITIONALLY every run (path-filter gates whole-workflow, NOT per-step). ⟹ "BE/governance ⟹ frozen" can be FALSE; rotation possible every deploy. To detect REAL FE change, diff fe-*/src in commit/range, NOT hash-delta.
    • ⚠️ SPA-fallback 200 trap (S72): server rewrites /*→/index.html so GET /assets/index-<ANYTHING>.js returns 200 even for fake hash (control ZZdoesnotexist0.js→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html <script src>, GET that exact path + check size_download LARGE (real ~1.6MB vs fake ~919b) + Last-Modified in deploy window + 2nd-fetch byte-stable (no mid-deploy transient).
    • 🟢 #69-REFINE (S83 #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change. #69 "rotates EVERY deploy" NOT absolute: identical Vite source ⟹ byte-identical output ⟹ same content-hash. MOST RELIABLE ship-proof = Last-Modified on real bundle IN deploy-window + real-size, NOT hash-delta (hash-delta false-NEG when src unchanged, false-POS risk per #69). 0-FE-change ⟹ hash EITHER same (#333/#485-487/#495/#385-S115) OR rotated (#312/#489) — both benign; ship-proof = fresh LM-in-window + real-size + marker.
  • Migration drift prod vs repo — compare ls .../Persistence/Migrations/*.cs vs sqlcmd __EFMigrationsHistory. Fix: check Program.cs/DbInitializer app.MigrateDatabase() + app pool recycle. Mig-applied proof = __EFMigrationsHistory top==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.

📋 5-stage checklist (EVERY run)

  • Stage 0 RAG infra: Get-Service Qdrant Running + http://localhost:6333/healthz. Collection proj_solution_erp.
  • Stage 1 Push+filter: git log -1 --format='%H %s' + git log origin/main..HEAD empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
  • Stage 2 Gitea poll (max 10 iter × 60s): API .../actions/tasks?limit=5 (NOT /runs 404). Match head_sha (NOT run_number — cancelled runs shift numbering). ⚠️ updated_at stale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter.
  • Stage 3 Test gate: baseline 501 PASS (45 Domain + 456 Infra; S117 #387 316a82f — 495+6 PE approved-price-range tests [NEW PeApprovedPriceFinalizeTests + PeFinalizeProjectionTests mod]; Infra 450→456, Domain frozen 45; prior 495 [S116 #386 88368fd 486+9 submit-guard/finalize/budget], 486, 477, 458, 440, 436, 434, 421). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (tasks terminal=status:success, conclusion NOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK.
  • Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=accessToken route /api/auth/login) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns 411 IIS Length-Required pre-auth NOT 404; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.
    • Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
    • Discovery: ASP.NET 10 record enum needs numeric input unless JsonStringEnumConverter (SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth \\\\SQLEXPRESS 4-backslash. INFRASTRUCTURE seed MUST run (NOT inside if(!demoSeedDisabled)); DEMO seed gated → gotcha #51.
  • Stage 5 Report PASS/FAIL + evidence + MEMORY update.

⚠️ Anti-patterns (DO NOT)

  1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)

  • Gitea: git.baocaogiaoduc.vn/vietreport-admin/solution-erp · workflow .gitea/workflows/deploy.yml · paths-ignore 7-entry (D10 S111 #495): ['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']. Anon API works (public repo) when GITEA_TOKEN absent.
  • Prod: api/admin/eoffice .solutions.com.vn · SSH ssh vietreport-vps (Administrator, id_ed25519) · IIS phys paths: API C:\inetpub\solution-erp\api · admin \fe-admin · user \fe-user. DB .\SQLEXPRESS/SolutionErp/vrapp SQL-auth. Conn key = ConnectionStrings.Default (NOT DefaultConnection). DB pw = buKL3TGBkD0wDDbYVw65QeX9 (User Id=vrapp; ⚠️ Password field has NO vrapp/ prefix — S89 login-failed when prefix included; read verbatim from prod appsettings.Production.json ConnectionStrings.Default when $env:PROD_DB_PASSWORD empty). ⚠️ skill-doc path C:\inetpub\apps\SolutionErp\Api STALE.
    • SSH→PS quoting (S42): nested bash→ssh→powershell mangles $var/\" → use iconv UTF-16LE | base64 → powershell -EncodedCommand $B64; OR write ps1-file + powershell -File $(cygpath -w). sqlcmd over SSH direct -W -h -1 pw-inline. sys-catalog string-concat → COLLATE DATABASE_DEFAULT (collation conflict).
  • Tests baseline: 501 PASS (45 Domain + 456 Infra · 0 fail/skip; S117 #387 316a82f; prior 495/486/477/458/440/436/434/421). Phase 9 UAT skip OK. 🔑 Concrete count via web-UI log path https://git.baocaogiaoduc.vn/vietreport-admin/solution-erp/actions/runs/{run_number}/jobs/0/logs (~18KB, NO /api/v1/ prefix) → per-project Passed! - Failed: 0, Passed: N, Skipped: 0. ⚠️ S117 REFINE: /api/v1/…/actions/runs/{n}/jobs/0/logs AND /api/v1/…/tasks/{id}/logs BOTH now 404 (Gitea) — ONLY the bare web-UI runs path (no api/v1) returns log body; status=success alone still ⟹ gate-passed (tests run pre-deploy).
  • Mig latest repo: Mig 66 AddPeApprovedPriceRange (20260714035636, S117 316a82f; single AddColumn PurchaseEvaluations.ApprovedPriceMaxAmount decimal(18,2) nullable — cột giá-chốt MAX của "duyệt theo khoảng Min–Max" src ProMinMax; VERIFIED-APPLIED-PROD COL_LENGTH=9 + history-top 20260714035636_AddPeApprovedPriceRange + sys.tables 89 giữ [AddColumn no-new-table]). Prev Mig 65 BackfillPeQuoteIsSelectedAndWinnerInvariant / 64 AddSupplierPublishState / 62 ExpandSupplierFields (see docs/CLAUDE.md migration table). Path src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/. Prod check sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC. ⚠️ sys.tables(is_ms_shipped=0)=89 — AddColumn/data-only don't change count; commit touching no new table ⟹ 89 correct, don't FAIL. enum-additive = int NO-mig (history NOT-advance + empty git diff -- '*Migrations*').
  • Bundle hash live: current ship-baseline = S117 #387 (admin js cFI7ih4a/css CfunQHbd · user js MxgooVZw/css DGqqHQXS; size 1,656,696/1,563,044b; LM 04:18:21/04:19:16 GMT = 11:18/11:19+07 07-14) — JS-rotate ×2/CSS-frozen from S116 (Dd55jBpj/DVDr1UQ4). See S117 recent-run entry for markers. ⚠️ id≠run_number (head_sha-keyed). PER #69 TWO-MODE hash is SNAPSHOT-per-run NOT stable-baseline (rotate on real-src OR benign-0-src; freeze on 0-src) — ALWAYS re-snapshot LIVE post-success; ship-proof = LM-in-window + real-size (vs 919b SPA-trap) + marker-grep, NOT hash-delta. js-rotate/css-frozen = FE-logic-no-style fingerprint.
  • Bearer: admin admin@solutions.com.vn/Admin@123456 (full) · UAT nv.test@solutions.com.vn/TestUser@123456 (Drafter CCM, gotcha #44 check).

🔑 Critical config (flag commit nếu tái xuất)

Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)

🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)

Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.

📊 Run stats baseline

BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate. Recent runs all ~4m40-5m.

📅 Recent runs (compressed — verbatim → archive via _INDEX.md)

  • 🟢 S117 #387 (rn 387, id 501) 316a82f PE endsBeforeCeo-pointer-aware + duyệt-theo-khoảng-Min–Max + Mig 66 AddPeApprovedPriceRange APPLIED-PROD GO 5m36s (11:14:09→11:19:45+07 2026-07-14; 19-file: BE 7 [Domain/Config/Features/Service/IService/Dtos/Controller] + FE 6 [PeDetailTabs+PeWorkflowPanel+types ×2-app] + Mig 3 + test 2 [NEW PeApprovedPriceFinalizeTests + PeFinalizeProjectionTests mod]): status=success · gate CONCRETE web-UI-path runs/387/jobs/0/logs (NO /api/v1/): Domain 45 + Infra 456 = 501 (+6 vs S116 495; Infra 450→456, Domain frozen 45; 0-fail/0-skip) · Mig 66 APPLIED-PROD sqlcmd COL_LENGTH('PurchaseEvaluations','ApprovedPriceMaxAmount')=9 + history-top 20260714035636_AddPeApprovedPriceRange + sys.tables 89 giữ (AddColumn decimal(18,2) no-new-table) · FE JS-ROTATE ×2/CSS-frozen admin js Dd55jBpj→cFI7ih4a (css CfunQHbd frozen) · user js DVDr1UQ4→MxgooVZw (css DGqqHQXS frozen) = FE-logic-no-style fingerprint · byte-verify #77 BOTH-app NEW bundle: cả Min và Max=1 + duyệt theo khoảng=1 + ProMinMax=2 + Min–Max(en-dash)=1 PRESENT (all=0 in OLD DVDr1UQ4 captured pre-deploy — NEW-code ship proof; ⚠️ PRO Min=1 in OLD = pre-existing label ProMin:'PRO Min' = FALSE-discriminator) · size real 1,656,696/1,563,044b (vs 919b SPA-trap) · LM admin 04:18:21/user 04:19:16 GMT = 11:18/11:19+07 IN-window · smoke ALL 200 (api ready+live, admin, eoffice) · PE 401-secured: GET-list no-token+bogus→401, POST /transitions bogus+no-token→401, /select-winner bogus→401 (no 500) · 🔑 pre-deploy bundle-snapshot BEFORE poll-success (captured OLD-hash+OLD-feature=0) → post-success re-snapshot = clean rotation proof. cicd-lane CLEAN-return.
  • (S116 #386 88368fd PE-nhập-giá-ÂM ô-báo-giá + submit-guard all-zero [gate 495=45+450, js-rotate×2 D51OYyGV→Dd55jBpj/BVdssm5S→DVDr1UQ4 css-frozen, byte-verify 4-str PRESENT, 🔑 poll-til-success TRƯỚC re-snapshot tránh frozen-trap] + S115 #385 81b59f0 GOVERNANCE hmw.js+.ps1 non-ignored [gate 486 frozen, bundle FROZEN×4 fresh-LM+identical-size = #69-REFINE 0-src proof, #41 non-.md-in-.claude STILL triggers] + S114 #384 d436af2 PE multi-NCC per-hạng-mục + Mig 65 backfill-invariant APPLIED-PROD [gate 486, js-rotate×2 acprMqfB→D51OYyGV/Ck3BTDdA→BVdssm5S, byte trúng thầu theo từng hạng mục, select-winner 401] — all GO ~5m, FE real-ship, smoke ALL 200 · full verbatim → archive/2026-07.md via _INDEX)
  • (S111 #495 8aa3869 H22-WAL-adopt [deploy.yml SELF-change +2 paths-ignore, run parsed+ran-full = workflow-self-change PASS; bundle FROZEN ×6-streak; 🔑 AS-6 baseline-shift: post-#495 memory-push (budget.json/agent-memory/run-trace) HẾT trigger CI, phân-loại giờ soi file ngoài 7-glob] + S112 #496 e100ef0 PE-workflow+Supplier-import-Phase-B+Mig63 AddSupplierImportSourceFields [gate 458, 🔑 411-refine: bodyless-POST→411 ANY-route → re-probe WITH-body → 401/415-matched vs 404-missing] + S113 #383 5fa11b5 Supplier-import-v2+Mig64 AddSupplierPublishState-APPLIED-PROD [IsPublic 37/37, gate 477] — all GO ~5m, FE real-ship, Mig-applied-prod, smoke ALL 200 · full verbatim → archive/2026-07.md via _INDEX)
  • (S107-closeout→S110-spot #490-#494 GOVERNANCE/MEMORY-push GO ×5 [d74f67f·69957fd·f5a5c3e·6ee60cd·ac3f122 — trigger #41 non-.md-in-push: budget.json ×2 + hmw.js ×3, pattern DEAD từ D10 #495] — bundle FROZEN-streak ×5 sau ROTATE #489 [#69 TWO-MODE; LM-in-window + real-size 1,644,931/1,551,280b = signal duy nhất] · Mig 62 + sys.tables 89 frozen · gate 440 · smoke 3×200 ALL · 🔑 #494 spot-detection: LM-prod-newer-than-last-verified-window = missed-run detector · full verbatim → archive/2026-07.md via _INDEX)
  • (S107 #489 21387cd GOVERNANCE-ONLY (+office-document 12th sub) GO 5m33s — JS ROTATED-benign ChSNAJBK→BVyH-FY7 / Bn2w1znH→EDsZebvn CSS-frozen, 🔑 #69 TWO-MODE CONFIRMED (0-fe-src ROTATE #312/#489 hoặc FROZEN #333/#485-487 — ship-proof = LM-in-window + real-size, hash-delta phụ) · S104 #488 778fc98 FULL-STACK Supplier 9→27 + Mig 62 ExpandSupplierFields APPLIED-PROD — AddColumn×18-exact + sys.tables-89-invariant + un-cram 4-NCC Note=NULL/Pkg/Bank/LegalRep-populated-EXACT + DTO-18-new-keys-live + JS-rotate-real-src + 🔑 FULL-STACK-SCHEMA-ADD+DATA-UN-CRAM PASS-pattern — full verbatim → archive/2026-07.md via _INDEX)
  • (S103 #487 + S100 #486 + S98 #485 GOVERNANCE/SCRIPTS-ONLY GO ×3 [.ps1/.json/.md-mix non-ignored → #41] — bundle FROZEN #69-REFINE ×4, GOVERNANCE-PASS stable [#476/#485-487] · 🔧 python time.sleep foreground-OK (chỉ bash-sleep-block) · full verbatim → archive/2026-07.md (+#485 git c112be9) via _INDEX)
  • (S96+S97 FINAL-REVIEW #478-#484 11-commit ALL-HEALTHY whole-session-GO [backfill A/005=1/A/008=1/A/010=0-exact + 88-excl/89-incl reconcile; #69 rotation≠ship-signal] · S97 #484 fb2dfa9 DATA-FIX Mig61 🔑 DATA-FIX-BACKFILL PASS = CI + Mig-top==HEAD + target-rows-flip-EXACT + control-untouched + derived-API-field-FLIP-live — verbatim → archive/2026-07.md via _INDEX)
  • (S96-S97 endsBeforeCeo-saga + PE-fixes #479-#484 all PASS 440 — key: #482 Mig60 CROSS-STACK [🔴 EF-TERNARY-SQL-translate-LIVE list-200-NOT-500 · #77 field-PRESENT] · #481/#483 FE-only [#77 new-str-PRESENT/old-ABSENT] · #480 baseline-440 +PeSuggestedPriceSetterAuthzTests · JS-rotate/CSS-frozen · verbatim → git + 2026-06.md via _INDEX)
  • (S90→S96 #465·#470·#472·#473·#476·#477 all PASS — key: #477 🔑 DATA-SEED-PASS = pre/post-DB-row-EXACT-delta + DLL-mtime + w3wp-recycle + Mig-frozen + API-payload-contains-row [⚠️ sqlcmd -W diacritic-LIKE false-NEG → _-wildcard; base64-EncodedCmd-FAIL → .ps1-File] · #472 ⚠️ PROD Identity tables NON-prefixed Roles NOT AspNetRoles · #470 GO-LIVE 🔑 no-resurrect = pre+post-DB-count-diff + dll-in-window + w3wp≥run-window [integrated-auth -E] · #476 GOVERNANCE-PASS baseline · verbatim → git + 2026-06.md via _INDEX)
  • (S69b→S89 runs #307·#313·#315·#318·#330·#333·#342-#348 all PASS — key: Mig54-59 applied-prod · #345 PATCH-401+7-authz · #344 backfill-winners=13 · #343 BE-only ship-proof=Infra.dll-mtime · #342 INVERTED-PASS-FROZEN [#73 TZ] · #346 FE-only js+css-BOTH-rotate · #348 411→401 endpoint-wired · full verbatim → git + 2026-06.md via _INDEX)
  • Older runs (S83 #325 FE-budget-indent ← S77 #329 banner ← S75 #301 ← S62 #286 ← … → S29 #232) full verbatim → git d2f52ba + archive/2026-06.md (incl #291 06-16 FAIL forensic [gotcha #65]); pre-S38 → archive/2026-05-{runs,q2,q3,q4}.md.

🔄 Curate trigger

  • ~30KB → archive recent runs → L2 archive/<period>.md (byte-exact append) + _INDEX.md substring pointer. Dup failure patterns → merge. Stale >3mo → remove.

  • Last curate: 2026-07-13 S115 (self, PostToolUse 21.6→<17KB): #384/#383/#496/#495 (S114/S113/S112/S111) verbatim → archive/2026-07.md + _INDEX +4 pointers → S114 compact-stub + S111-113 merged-stub; top S114 garble-block → 1-line tracker.
  • Prev: S111 07-12 self 20.5→17 (#490-#494→archive) · S107 07-10 self 20.4→16.9 (#489/#488→archive) · S104 07-06 self 20.5→17.9 · S98 07-02 em-main 19.9→17.06 (2026-07.md NEW) · S80 06-20 em-main 38.8→16 · S70 06-17 Harness-9 65.2→23.2 · earlier S66 · S40.