20 KiB
CI/CD Monitor Agent — Persistent Memory
-
07-28 verify
a52a91e(PE: cảnh báo amber ô Địa điểm khi dự án thiếu Location) [S157, FE-ONLY 2-app, CLEAN-return]: VERDICT PASS 9/9 — Run #419 (id=532) success 5m40s (09:40:18→09:45:58+07); gate 562 EXACT từ log (45 Domain + 517 Infra, 0 fail/skip) — khớp canonicaldocs/STATUS.md:452cả TỔNG lẫn TÁCH 45/517 (khớp thành-phần > khớp tổng). NO-Mig đúng kỳ vọng:git diff f4494cf..a52a91e -- 'src/Backend/*' '*Migrations*' | wc -l= 0 ⇒ Mig vẫn 68, 89-table, KHÔNG query DB (không có gì để đối chiếu). 🔑 Phép quyết định = byte-marker #77, KHÔNG tin status CI: tải thẳng JS prod,bytes.count(marker.encode())4 chuỗi VN độc lập → 2/2 app, count=2 mỗi app = KHỚP SỐ FILE/app trong diff (PeHeaderForm+PeWorkspaceCreateView) — "đúng-BỘI-SỐ" mạnh hơn "tìm thấy ≥1", bắt được cả ca ship nửa-vời. Bundle js rotate CẢ 2 (adminBK8w3QxU→8-QYcMOg· eofficeDYnxSI0T→DptYR4wL; cssDiNJjaoo/BRCvEHvCFROZEN ×2 = #69-REFINE lành tính — utilitytext-amber-700/text-[11px]đã có sẵn) real-size 1.675.282/1.575.513B, LM 09:44:33/09:45:30+07 in-window. 🆕 Control âm 2 TẦNG (tái dụng): (i) fake-hash→200 nhưngct=text/html900/876B marker=0; (ii) hash CŨ cũng rơi về text/html ⇒ bundle cũ BIẾN MẤT khỏi đĩa = chứng GHI-ĐÈ thật (mạnh hơn hash-delta đơn thuần). Smoke: health live+ready 200 · login 200 tok-468 ·/api/projects200 (đúng nguồnlocationfeature đọc) · PE/menus/suppliers 200. ⚠️ Self-correct on-disk: viết "range=6 file" SAI (đó là số của 2 commit) — range thật 13 commit/26 file (11wal:xen giữa); đính chính ngay, BE=0 không đổi. Tag[run-419, pass-9-9, fe-only, marker-2of2-count-exact, old-hash-gone-control, js-rotate-css-frozen, gate-562] -
(S155 #416
b1bae77Workflow duyệt-TẠI-CHỖ + cờ F6AllowApproverDeleteGO 6/6 — gate 551 [45+506, khớp local 2-nguồn]; Mig 68AddPeAllowApproverDeleteAPPLIED-PROD [history-top + INFORMATION_SCHEMA cùng có, 89-table]; 🔑 reviewer-FLAG F8 "mig rớt khỏi commit" loại-trừ 3 lớp — mạnh nhất =GET /approval-workflows-v2trảallowApproverDelete×54 materialize qua EF từ DB thật (thiếu cột thì query đã ném) · verbatim →archive/2026-07.md@S157-curate) -
(S135 #407
4510ceamã-dự-án normalize + PE attach-gate GO 5/5 — gate 532 [45+487, +4 delta-proof]; NO-Miggit diff -- *Migrations*EMPTY + history-top unchanged; js-rotate/css-frozen ×2 · verbatim →archive/2026-07.md@S155-curate) -
(S134 #399
6664a61PE lũy-kế tạm-tính (0-mig) GO 5/5 — gate 528 [45+483, +8 PePendingAccumulation]; js+css rotate ×2 adminBVCukFP5/CBzUlgOAeofficeDoxyEOxh/BHsBUA8e; byte-verify #77 urllib-BYTES; 🔑 ct=text/html discriminator SẠCH HƠN size-919B [shell size drift per-build];body.count(marker.encode())bulletproof cp1252 · verbatim →archive/2026-07.mdvia_INDEX) -
(S133 #398
d4fbcedPE budget-freeze + Mig 67AddPeApprovedBudgetSnapshotAPPLIED-PROD GO 5/5 — gate 520 [45+475]; 11-col ApprovedBudget* + backfill 18/18 DaDuyet + 89-table; js+CSS rotate ×2 [badge utility-class → CSS content-hash đổi, benign #69]; python text=True crash cp1252 → text=False bytes+json.loads · verbatim →archive/2026-07.mdvia_INDEX)
Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2
archive/on-demand · L3 RAGsearch_memoryjust-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → gitd2f52ba+archive/2026-05-{runs,q2,q3,q4}.md+archive/2026-06.md+archive/2026-07.md. 🗺️ Lookup map:archive/_INDEX.md— 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim +2026-0{5,6}.gist.mdtheo nhu cầu. See 🔄 Curate-trigger log for archive-move history.
🎯 Role baseline
Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.
⚠️ cicd-lane return-garble tracker (#53)
cicd-spawn đôi khi trả memory-curation slice ("Compressing S112/S111") thay verdict (×5/6 qua S114; cumulative ×18 all-agents → feedback_agent_return_garble_recover, root-cause = curate-at-return). Recover: em-main-solo curl-verify. S115 (#385)+S116 (#386) cicd-lane CLEAN-return no-garble ×2; S117 (#387) garble→SendMessage-resume (em-main curl cross-verify khớp, verdict disk-true) — return-mechanics garbled NHƯNG verdict-substance OK. S118 (#388+#389) cicd-lane garble ×2 → em-main-solo curl+sqlcmd recover (#388 durability tra.bui=20 seed-race "9 upgraded" · #389 Reports.Read→Admin-only+401), both verdict-substance disk-verified OK; curate-at-return root-cause persists (harvest-curator S118-end bắt nhãn "CLEAN-return" tự-dán sai → habitual pattern).
🚨 Recurring CI/CD bug patterns (catch priority)
- #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log
dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded.gitea/workflows/deploy.yml. KHÔNG revert. - #40 npm cache
tsc not found—build_fe_adminfail postcache: npm. DISABLED rolled backa21790d. KHÔNG re-enable. - #41 paths-ignore docs-only skip — code commit không trigger CI? Check
git diff --name-only HEAD~1 HEADvspaths-ignore7-entry từ D10 S1118aa3869(#495):['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']. Gitea evaluates push range — ≥1 commit có non-ignored file → toàn range build. ⚠️ AS-6 BASELINE SHIFT: từ #495, memory-push chứamemory-budget.json/agent-memory-file/run-trace HẾT trigger CI (pattern #490/#492 DEAD). - #25 IIS WebSocket —
notification-hub/negotiate401/404 prod. Fix: WebSocket module enableweb.configsite api (skilliis-deploy-runbook). - #48 SQLite tie-break —
OrderByDescending(CreatedAt).First()pick wrong khi 2+.Add()cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy. - Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (
curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).- 🔴 #69 (S72 #312) — FE bundle hash NON-DETERMINISTIC per rebuild.
deploy.ymlRemove-Item fe-*\* -Exclude web.config+Copy-Item dist\*runs UNCONDITIONALLY every run (path-filter gates whole-workflow, NOT per-step). ⟹ "BE/governance ⟹ frozen" can be FALSE; rotation possible every deploy. To detect REAL FE change, difffe-*/srcin commit/range, NOT hash-delta. - ⚠️ SPA-fallback 200 trap (S72): server rewrites
/*→/index.htmlso GET/assets/index-<ANYTHING>.jsreturns 200 even for fake hash (controlZZdoesnotexist0.js→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html<script src>, GET that exact path + checksize_downloadLARGE (real ~1.6MB vs fake ~919b) +Last-Modifiedin deploy window + 2nd-fetch byte-stable (no mid-deploy transient). - 🟢 #69-REFINE (S83 #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change. #69 "rotates EVERY deploy" NOT absolute: identical Vite source ⟹ byte-identical output ⟹ same content-hash. MOST RELIABLE ship-proof =
Last-Modifiedon real bundle IN deploy-window + real-size, NOT hash-delta (hash-delta false-NEG when src unchanged, false-POS risk per #69). 0-FE-change ⟹ hash EITHER same (#333/#485-487/#495/#385-S115) OR rotated (#312/#489) — both benign; ship-proof = fresh LM-in-window + real-size + marker.
- 🔴 #69 (S72 #312) — FE bundle hash NON-DETERMINISTIC per rebuild.
- Migration drift prod vs repo — compare
ls .../Persistence/Migrations/*.csvssqlcmd __EFMigrationsHistory. Fix: checkProgram.cs/DbInitializerapp.MigrateDatabase()+ app pool recycle. Mig-applied proof =__EFMigrationsHistorytop==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.
📋 5-stage checklist (EVERY run)
- Stage 0 RAG infra:
Get-Service QdrantRunning +http://localhost:6333/healthz. Collectionproj_solution_erp. - Stage 1 Push+filter:
git log -1 --format='%H %s'+git log origin/main..HEADempty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return). - Stage 2 Gitea poll (max 10 iter × 60s): API
.../actions/tasks?limit=5(NOT/runs404). Matchhead_sha(NOT run_number — cancelled runs shift numbering). ⚠️updated_atstale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter. - Stage 3 Test gate: baseline 509 PASS (45 Domain + 464 Infra; S118 #389
c808b9a— 505+4AuthorizePolicyRegressionTestsReportsController Reports.Read gate; Infra 460→464, Domain frozen 45; prior 505 [#388c2135baProcurementMasterAccessSeed], 501, 495, 486, 458, 440, 421). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (tasksterminal=status:success,conclusionNOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK. - Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=
accessTokenroute/api/auth/login) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns 411 IIS Length-Required pre-auth NOT 404; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.- Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
- Discovery: ASP.NET 10 record enum needs numeric input unless
JsonStringEnumConverter(SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth\\\\SQLEXPRESS4-backslash. INFRASTRUCTURE seed MUST run (NOT insideif(!demoSeedDisabled)); DEMO seed gated → gotcha #51.
- Stage 5 Report PASS/FAIL + evidence + MEMORY update.
⚠️ Anti-patterns (DO NOT)
- ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)
🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)
- Gitea:
git.baocaogiaoduc.vn/vietreport-admin/solution-erp· workflow.gitea/workflows/deploy.yml· paths-ignore 7-entry (D10 S111 #495):['docs/**','**/*.md','.claude/skills/**','.claude/agent-memory/**','.claude/workflows/runs/**','.gitignore','scripts/**.md']. Anon API works (public repo) when GITEA_TOKEN absent. - Prod: api/admin/eoffice
.solutions.com.vn· SSHssh vietreport-vps(Administrator, id_ed25519) · IIS phys paths: APIC:\inetpub\solution-erp\api· admin\fe-admin· user\fe-user. DB.\SQLEXPRESS/SolutionErp/vrappSQL-auth. Conn key =ConnectionStrings.Default(NOTDefaultConnection). DB pw =buKL3TGBkD0wDDbYVw65QeX9(User Id=vrapp; ⚠️ Password field has NOvrapp/prefix — S89 login-failed when prefix included; read verbatim from prodappsettings.Production.jsonConnectionStrings.Default when$env:PROD_DB_PASSWORDempty). ⚠️ skill-doc pathC:\inetpub\apps\SolutionErp\ApiSTALE.- SSH→PS quoting (S42): nested bash→ssh→powershell mangles
$var/\"→ useiconv UTF-16LE | base64→powershell -EncodedCommand $B64; OR write ps1-file +powershell -File $(cygpath -w). sqlcmd over SSH direct-W -h -1pw-inline. sys-catalog string-concat →COLLATE DATABASE_DEFAULT(collation conflict).
- SSH→PS quoting (S42): nested bash→ssh→powershell mangles
- Tests baseline: 562 PASS (45 Domain + 517 Infra · 0 fail/skip; run #419
a52a91e07-28, khớp canonicaldocs/STATUS.md:452cả tách-phần; prior climb 551/532/528/520/509/505/501/495/486/458/440/421). Phase 9 UAT skip OK. 🔑 Concrete count via web-UI loghttps://git.baocaogiaoduc.vn/vietreport-admin/solution-erp/actions/runs/{run_number}/jobs/0/logs(~18KB, NO/api/v1/prefix — api/v1 runs+tasks logs BOTH 404) → per-projectPassed! - Failed: 0, Passed: N, Skipped: 0. Delta-proof new-tests = compare prev-run count (dotnet summary hides names). status=success alone ⟹ gate-passed (tests pre-deploy). - Mig latest repo: Mig 68
AddPeAllowApproverDelete(20260727033522, S155b1bae77— AddColumnAllowApproverDeletebit NOT NULL def-0 trênApprovalWorkflowLevels, no-new-table; VERIFIED-APPLIED-PROD history-top + col-present + sys.tables 89). Prev Mig 67AddPeApprovedBudgetSnapshot/ 66AddPeApprovedPriceRange/ 65 backfill-invariant / 62ExpandSupplierFields(full table → docs/CLAUDE.md). Pathsrc/Backend/SolutionErp.Infrastructure/Persistence/Migrations/(68.csexcl Designer/Snapshot). Prod checksqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC. ⚠️sys.tables(is_ms_shipped=0)=89 — AddColumn/data-only/enum-additive don't change count; no-new-table commit ⟹ 89 correct, don't FAIL. NO-Mig proof = history NOT-advance + emptygit diff <base>..<head> -- '*Migrations*'. - Bundle hash live: ship-baseline = run #416 (07-27) admin js
CiUBEEJr/cssDX5ew0wg· eoffice jsDFh7GK0t/cssBHsBUA8e(real-size 1,670,035/1,570,257b; LM 12:41:28/12:42:22+07). PER #69 TWO-MODE = SNAPSHOT-per-run NOT stable-baseline (rotate on real-src, freeze byte-id on 0-src) — ALWAYS re-snapshot LIVE post-success; ship-proof = LM-in-window + real-size (vs 900bct=text/htmlSPA-trap) + marker-grep, NOT hash-delta. js-rotate/css-frozen = FE-logic-no-style fingerprint (S134-refine:ct=text/htmlcleaner discriminator than 919b, shell size drifts per-build). ⚠️ id≠run_number (head_sha-keyed). - Bearer: admin
admin@solutions.com.vn/Admin@123456(full) · UATnv.test@solutions.com.vn/TestUser@123456(Drafter CCM, gotcha #44 check). - API Serilog log (S118 correction): real sink =
C:\inetpub\solution-erp\api\logs\solution-erp-YYYYMMDD.log(lowercaselogs, rolling daily). ⚠️ skill-runbook path…\Logs\log-YYYYMMDD.txtSTALE → empty grep. MinimumLevel Default=Information (DbInitializerLogInformationseed/revoke lines DO persist) — Microsoft/System overridden Warning (no "Now listening" INFO). Startup seed-race audit pattern (durability deploys): grep this log for revoke-line THEN seeder-line same-second → "N added + M upgraded" proves seeder actively flipped rows revoke knocked down (0-upgraded = seeder no-op = FAIL flag).
🔑 Critical config (flag commit nếu tái xuất)
Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)
🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)
Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.
📊 Run stats baseline
BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate. Recent runs all ~4m40-5m.
📅 Recent runs (compressed — verbatim → archive via _INDEX.md)
- (S128 #395
289ba96+ S129 #396295c70c+ S132 #39772f68d1GOVERNANCE/SCRIPTS/TOOLING GO — gate 509 [45+464, 0 fail/skip]; smoke 4×200; bundle benign #69 [S128 FULL-ROTATE from S117 →yCkX-2Bp/DKSD4p9k; S129+S132 FROZEN byte-id 0-fe-src] ship-proof = LM-in-window + real-size ≠919b NOT hash-delta; NO-BE/NO-Mig [Mig 66/89-table]; 🔑 AS-6 #41: soi TỪNG file vs 7-glob —.ps1thoátscripts/**.mdglob +.claude/rag.json/.session-counter.jsonthoát ⟹ non-.md governance VẪN trigger CI;conclusion:Nonedù success → truststatus[tái-xác-nhận S126/S128/S129/S132] · verbatim →archive/2026-07.md) - (S118 #389
c808b9aSECURITY-AUTHZ ReportsController→Reports.Read GO 5m34s [gate 509=45+464 (+4 authz-reflection vs 505); DBReports.Read→Admin-ONLY sqlcmd (Procurement CanRead=0 → non-admin 403 by-construction); no-token 401 class-[Authorize]intact; NO-FE/NO-Mig; cicd-lane GARBLED #53 → em-main curl+sqlcmd recovered, verdict disk-TRUE] · +S117 #387316a82fPE duyệt-khoảng Min–Max + Mig 66AddPeApprovedPriceRangeAPPLIED-PROD [gate 501; js-rotate×2 css-frozen; byte-verify #77 4-str PRESENT; Mig66COL_LENGTH=9+sys.tables 89; 🔑 pre-deploy snapshot BEFORE poll-success] · verbatim →archive/2026-07.mdvia_INDEX) - (S111-S116 #383-#386/#495/#496 PE multi-NCC per-hạng-mục (Mig 65 backfill) + nhập-giá-ÂM + submit-guard + Supplier-import Phase-B/v2 (Mig63
AddSupplierImportSourceFields/Mig64AddSupplierPublishStateAPPLIED-PROD, IsPublic 37/37) + H22-WAL-adopt (deploy.yml self-change +2 paths-ignore) + GOVERNANCE GO [gate 458→495; js-rotate/css-frozen; 🔑 AS-6 baseline-shift memory-push HẾT trigger; 411-refine bodyless-POST→411 → re-probe-with-body 401/415≠404; #69-REFINE 0-src FROZEN-proof; byte-verify #77 str-PRESENT] · verbatim →archive/2026-07.mdvia_INDEX) - (S98-S110 + S126 #485-#494/#394 GOVERNANCE/MEMORY/TOOLING-push FROZEN-streak sau ROTATE #489 (#69 TWO-MODE; ship-proof = LM-in-window + real-size, hash-delta phụ); gate 440→509; Mig 62 + 89-table frozen · 🔑 #494 LM-newer-than-verified-window = missed-run detector; bare
/health=404 UNMAPPED (đừng FAIL — thật/health/live+/ready) · verbatim →archive/2026-07.mdvia_INDEX) - (PASS-pattern library S69b→S104 #307-#494 all GO — 🔑 reusable verify-recipes: SCHEMA-ADD+DATA-UN-CRAM [#488 Supplier 9→27 Mig62: AddColumn×18-exact + 89-invariant + un-cram 4-NCC + DTO-live] · DATA-FIX-BACKFILL [#484 Mig61: Mig-top==HEAD + target-rows-flip-EXACT + control-untouched + derived-API-FLIP-live] · DATA-SEED-PASS [#477: pre/post-DB-row-delta + DLL-mtime + w3wp-recycle]; #482 Mig60 EF-ternary-SQL 200≠500; #472 PROD Identity NON-prefix
Roles; #470 no-resurrect DB-count-diff; Mig54-62 applied-prod; #348 411→401 · verbatim → git +2026-06/07.md) - Older (S83 #325 ← S29 #232) → git
d2f52ba+archive/2026-06.md(incl #291 06-16 FAIL [gotcha #65]); pre-S38 →archive/2026-05-*.
🔄 Curate trigger
-
~30KB → archive recent runs → L2
archive/<period>.md(byte-exact append) +_INDEX.mdsubstring pointer. Dup failure patterns → merge. Stale >3mo → remove. - Last curate: 2026-07-14 S118 (self #389-run: S117 #387 verbatim→
archive/2026-07.mdmoved-not-cut byte-id, baselines 505→509; prev S115 self 21.6→<17KB): #384/#383/#496/#495 (S114/S113/S112/S111) verbatim →archive/2026-07.md+_INDEX+4 pointers → S114 compact-stub + S111-113 merged-stub; top S114 garble-block → 1-line tracker. - Prev: S111 07-12 self 20.5→17 (#490-#494→archive) · S107 07-10 self 20.4→16.9 (#489/#488→archive) · S104 07-06 self 20.5→17.9 · S98 07-02 em-main 19.9→17.06 (
2026-07.mdNEW) · S80 06-20 em-main 38.8→16 · S70 06-17 Harness-9 65.2→23.2 · earlier S66 · S40.