vietreport-admin/solution-erp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
65b434a3702c5490ead6daf2d11d856bfbac6cec
solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md
pqhuy1987 65b434a370 [CLAUDE] Docs: Harness-17 deep-reverify (WF3) — fault-injection teeth-proof + cross-doc + AI_INFRA update 
- WF3 wf_6f44eddf-680 (anh-directed 3rd pass): Lane A fault-injection PASS (13 inject prove teeth: A1/A2/C-spectrum/B1/B2/A3 all catch faults, no-false-flag, off-by-one correct, 8/8 md5 match restore) + Lane C integration PASS (0 application-defect, anti-Goodhart flip proofs, numbering intact, mfe-eval A2 not broken)
- Lane B empty-return (#53) → em-main self-gate recover (N=3/HCV/run-id/hash consistent, detector 27)
- engine §I banner: inline WF2 wf_8e73c1d6-ee6 + WF3 wf_6f44eddf-680 (was WF2-pointer-only)
- adap-report: +Deep-reverify(WF3) section + 2 brittleness honest-noted (fail-safe over-flag, not fixed per Lane A) + dogfood-C3-round2
- email AI_INFRA update (hash 8c94fa348514): deep-reverify notify
- dogfood: WF3 reviewer self-write reverted (D9/AS-10 round-2); cicd-monitor verdict kept (legit normal-spawn) — Run #476 success, prod 200x3, Mig 59
- 0 .ps1 changed → CI-skip (docs-only); 0 production code

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 09:31:55 +07:00

17 KiB
Raw Blame History

CI/CD Monitor Agent — Persistent Memory

Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 archive/ on-demand · L3 RAG search_memory just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → git d2f52ba + archive/2026-05-{runs,q2,q3,q4}.md + archive/2026-06.md. 🗺️ Lookup map: archive/_INDEX.md — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + 2026-0{5,6}.gist.md theo nhu cầu. S80 curate: moved 3 huge run-entries (#330/#318/#325) → archive.

🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.

🚨 Recurring CI/CD bug patterns (catch priority)

  • #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded .gitea/workflows/deploy.yml. KHÔNG revert.
  • #40 npm cache tsc not found — build_fe_admin fail post cache: npm. DISABLED rolled back a21790d. KHÔNG re-enable.
  • #41 paths-ignore docs-only skip — code commit không trigger CI? Check git diff --name-only HEAD~1 HEAD vs paths-ignore: ['docs/**','**/*.md','.claude/skills/**']. Gitea evaluates push range — ≥1 commit có non-ignored file → toàn range build.
  • #25 IIS WebSocket — notification-hub/negotiate 401/404 prod. Fix: WebSocket module enable web.config site api (skill iis-deploy-runbook).
  • #48 SQLite tie-break — OrderByDescending(CreatedAt).First() pick wrong khi 2+ .Add() cùng frozen-clock. Fix: discriminator filter BEFORE OrderBy.
  • Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'), NOT by GETting a hash-named asset directly. ⚠️ verify MUST sau status=success (Run #242 false-positive: check khi "running" → stale hash).
    • 🔴 #69 (S72 Run #312) — FE bundle hash NON-DETERMINISTIC per rebuild. deploy.yml Remove-Item fe-*\* -Exclude web.config + Copy-Item dist\* runs UNCONDITIONALLY every run (path-filter gates whole-workflow trigger, NOT per-step). Identical FE source ⟹ DIFFERENT hash each deploy (Vite/rolldown non-reproducible chunk-id). PROVEN: governance-only 18fced6 (0 files in fe-*/src) rotated BOTH bundles. ⟹ "BE-only/governance ⟹ bundle frozen" is FALSE; rotation EXPECTED every deploy. To detect REAL FE change, diff fe-*/src in commit/range, NOT hash delta.
    • ⚠️ SPA-fallback 200 trap (S72): server rewrites /*→/index.html so GET /assets/index-<ANYTHING>.js returns 200 even for fake hash (control ZZdoesnotexist0.js→200, size~919b text/html). Old-hash-still-200 MEANINGLESS. RELIABLE signal = parse index.html <script src>, GET that exact path + check size_download LARGE (real ~1.6MB vs fake ~919b) + Last-Modified in deploy window + 2nd-fetch byte-stable (no mid-deploy transient).
    • 🟢 #69-REFINE (S83 Run #333) — hash can stay IDENTICAL across deploy when fe-*/src 0-change. #69 said "rotates EVERY deploy" — NOT absolute. Run #333 (BE+tests+governance, ZERO fe-*/src diff) shipped FE (Last-Modified 17:10:35/17:11:27 BOTH in run-window 17:06→17:11) but hash UNCHANGED (admin CsJetgZH/user BVS0ApIm = #330 baseline). ⟹ identical Vite source ⟹ byte-identical output ⟹ same content-hash. Rolldown chunk-id apparently MORE deterministic than #69 feared (or only rotates on real src delta). MOST RELIABLE ship-proof = Last-Modified header on the real bundle file IN deploy-window, NOT hash-delta (hash-delta has false-NEGATIVE when src unchanged + false-POSITIVE risk noted in #69). Always cross-ref git diff fe-*/src scope: 0-FE-change ⟹ expect SAME hash + fresh Last-Modified.
  • Migration drift prod vs repo — compare ls .../Persistence/Migrations/*.cs vs sqlcmd __EFMigrationsHistory. Fix: check Program.cs/DbInitializer app.MigrateDatabase() + app pool recycle. Mig-applied proof = __EFMigrationsHistory top==repo-HEAD; stuck-old ⟹ pool didn't recycle ⟹ FAIL even if status=success+bundle-rotated.

📋 5-stage checklist (EVERY run)

  • Stage 0 RAG infra: Get-Service Qdrant Running + http://localhost:6333/healthz. Collection proj_solution_erp.
  • Stage 1 Push+filter: git log -1 --format='%H %s' + git log origin/main..HEAD empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
  • Stage 2 Gitea poll (max 10 iter × 60s): API .../actions/tasks?limit=5 (NOT /runs 404). Match head_sha (NOT run_number — cancelled runs shift numbering). ⚠️ updated_at stale ~2min (gotcha #46) → cross-check VPS mtime. Foreground-sleep BLOCKED Windows → Monitor-style busy-wait-curl-spin ~30s/iter.
  • Stage 3 Test gate: baseline 421 PASS (45 Domain + 376 Infra; S90 #465 — 419 + 2 new PeDraftVisibilityTests [phiếu nháp riêng tư người tạo, chặn leak role + V2-approver]; prior S89 #348 419, S88-quad #347 413). CI runs both proj BEFORE build/deploy → status=success ⟹ test gate passed (tasks terminal=status:success, conclusion NOT populated — trust success NOT log-numeric). Local grep undercounts (Theory/InlineData). Phase 9 UAT skip OK.
  • Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=accessToken route /api/auth/login) → 3-5 endpoint smoke 2XX (incl new — NEW-endpoint bodyless-POST returns 411 IIS Length-Required pre-auth NOT 404; re-probe WITH body → 401 confirms wired) → FE bundle hash 2 app (per #69 = real-size-vs-fake not hash-delta) → SignalR negotiate (gotcha #25) → EF mig prod==repo.
    • Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). Seed=runtime-row-insert (no history/tables advance) → verify via DB-query (e.g. Permissions count), NOT bundle-frozen-alone.
    • Discovery: ASP.NET 10 record enum needs numeric input unless JsonStringEnumConverter (SOL has NO converter → FE sends numeric). sqlcmd ssh Windows-auth \\\\SQLEXPRESS 4-backslash. INFRASTRUCTURE seed MUST run (NOT inside if(!demoSeedDisabled)); DEMO seed gated → gotcha #51.
  • Stage 5 Report PASS/FAIL + evidence + MEMORY update.

⚠️ Anti-patterns (DO NOT)

  1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

🧠 SOLUTION_ERP CI/CD essentials (S80 verified — re-ground từ docs/STATUS.md canonical)

  • Gitea: git.baocaogiaoduc.vn/vietreport-admin/solution-erp · workflow .gitea/workflows/deploy.yml · paths-ignore ['docs/**','**/*.md','.claude/skills/**']. Anon API works (public repo) when GITEA_TOKEN absent.
  • Prod: api/admin/eoffice .solutions.com.vn · SSH ssh vietreport-vps (Administrator, id_ed25519) · IIS phys paths: API C:\inetpub\solution-erp\api · admin \fe-admin · user \fe-user. DB .\SQLEXPRESS/SolutionErp/vrapp SQL-auth. Conn key = ConnectionStrings.Default (NOT DefaultConnection). DB pw = buKL3TGBkD0wDDbYVw65QeX9 (User Id=vrapp; ⚠️ Password field has NO vrapp/ prefix — S89 login-failed when prefix included; read verbatim from prod appsettings.Production.json ConnectionStrings.Default when $env:PROD_DB_PASSWORD empty). ⚠️ skill-doc path C:\inetpub\apps\SolutionErp\Api STALE.
    • SSH→PS quoting (S42): nested bash→ssh→powershell mangles $var/\" → use iconv UTF-16LE | base64 → powershell -EncodedCommand $B64; OR write ps1-file + powershell -File $(cygpath -w). sqlcmd over SSH direct -W -h -1 pw-inline. sys-catalog string-concat → COLLATE DATABASE_DEFAULT (collation conflict).
  • Tests baseline: 421 PASS (45 Domain + 376 Infra · 0 fail/skip; S90 Run #465 sha a71753b — 419 + 2 PeDraftVisibilityTests [draft-private-to-creator, chặn leak role + V2-approver]; prior S89 #348 419, S88-quad #347 413). Phase 9 UAT skip per chunk OK.
  • Mig latest repo: Mig 59 20260624084608_AddCcmBudgetPeriodToPurchaseEvaluation (S88 #345; PE +1 AddColumn CcmBudgetPeriodAmount decimal(18,2) NULL no-backfill — VERIFIED-APPLIED-PROD history-HEAD + col-exists 0-nonnull). Prev Mig 58 AddPeMultiWinnerAndLevelFinalize (S87 #344). Path src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/. Prod check sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5. ⚠️ sys.tables (is_ms_shipped=0) = 88 (S61 Budget-replace DROPPED 93→88 — narrative-93 STALE; when commit touches no schema, 88 correct, don't FAIL on 88. Always cross-ref COMMIT scope vs ambient count). enum-additive (e.g. ApprovalAttachment=5) = int-stored NO-mig → expect history NOT-advance + empty git diff -- '*Migrations*'.
  • Bundle hash live (S90 Run #465/id=465): admin js BXixQDuS/css DWQ-DgfZ · user js BMgckUPd/css Dr0-ikKX (JS BOTH ROTATED from pre-deploy live BeyLzf5i/CPe970sY by real fe-src PeDetailTabs ×2; css FROZEN DWQ-DgfZ/Dr0-ikKX — correct, FE logic-only [gỡ !readOnly gate] no new utility class). ⚠️ #465 ≠ run_number; head_sha-keyed. ⚠️ pre-deploy baseline was BeyLzf5i/CPe970sY (Run #350 web.config rotated since #348 BQq2eTP6/dHa64I59 — always re-snapshot LIVE before deploy, brief's stale prev-hash misleads). ⚠️ PER #69: hash MAY rotate per deploy OR stay frozen if 0 fe-src diff (byte-identical) — SNAPSHOT for THIS run, NOT a frozen baseline. Real FE ship = diff fe-*/src in commit/range + real-size-vs-fake (1.6/1.5MB vs ~919b SPA-trap), NOT hash delta. ASYMMETRIC-deploy framing (S66) RETIRED by #69.
  • Bearer: admin admin@solutions.com.vn/Admin@123456 (full) · UAT nv.test@solutions.com.vn/TestUser@123456 (Drafter CCM, gotcha #44 check).

🔑 Critical config (flag commit nếu tái xuất)

Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)

🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)

Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.

📊 Run stats baseline

BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate. Recent runs all ~4m40-5m.

📅 Recent runs (compressed — full verbatim → archive/2026-06.md via archive/_INDEX.md)

  • 🟢 S95 #476 (id=476, run_number 362) 18a17f5 GO ~5m08s (adopt Harness-17 memory-self-improve loop — GOVERNANCE-ONLY 0 prod code, CI-triggered-by-.ps1) — 10-file: 8 docs/md/broadcasts (paths-ignored) + 2 .ps1 scripts/governance-detectors.ps1+scripts/memory-selfimprove-audit.ps1 NOT in paths-ignore (scripts/**.md ignores MD-under-scripts ONLY) ⟹ CI runs despite 0 .NET/FE impact (✅ path-filter as-designed, NOT docs-skip). git diff --name-only 707b632 18a17f5 empty for .cs/.csproj/.tsx/*Migrations*/fe-src. status=success ⟹ test gate passed (baseline 434=45 Dom+389 Infra; tasks-API conclusion:None even-on-success — trust terminal-success NOT numeric). run 09:15:17→09:20:25+07. Smoke 3×200 (api /health/ready, admin+eoffice root). FE ship-proof (0 fe-src ⟹ #69 cosmetic-rotate): admin JS CkZ1MjC1→0T6vVJST + user _-gpX7_Y→BGXEVZO6 rotated; CSS FROZEN admin Bbtesa6B/user BmuPz_Dy (0 new Tailwind, mirrors S90/S93). Last-Modified admin 02:19:03Z/user 02:19:57Z in-window (02:15→02:20 UTC) + size 1,636,195b/1,542,518b real · SPA-trap fake-hash=200-but-919b-html (large=genuine). Mig prod=59 UNCHANGED (0 mig-file ⟹ NO sqlcmd needed, repo-scope sufficient: nothing CAN drift w/ 0-schema + repo-HEAD known-applied). ⚠️ brief prev-hash S94 CDn40vpV/CoPmPBg7 STALE — LIVE pre-deploy already CkZ1MjC1/_-gpX7_Y (#69 intermediate rotate); always re-snapshot LIVE, brief prev-hash=hint NOT truth. 🔑 GOVERNANCE-ONLY PASS = CI-green + terminal-success + JS-rotate/CSS-frozen-benign(#69) + 3×200 + Mig-59-frozen — 0 anomaly.
  • (S93 #473 8ee8f3e FULL-STACK PE tree-node Dự-án show-CODE-not-name DTO+3-projection NO-MIG baseline-434 · admin JS xz8FdmLN→CDn40vpV+user CA8Rla2w→CoPmPBg7 rotated, css FROZEN DWQ-DgfZ/Dr0-ikKX logic-only ✅ · Mig59 frozen · pre-deploy-live==brief-prev-hash no-drift; 🔑 css-frozen-on-logic-FE=NOT-fail — full verbatim → git 8ee8f3e + archive 2026-06.md via _INDEX)
  • (S92 #472 1408d4e BE-ONLY DbInitializer permission-revoke +5-menu-groups non-Admin VERIFY-PROD-EFFECT baseline-434 · 3-evidence ship-proof Infrastructure.dll+Api.dll in-window + w3wp PID10060 recycle-AT-END-window · PROD-EFFECT PROVEN: hidden-key non-Admin CanRead=0, Admin-retained=5, Pe* untouched=120, leak=0 · ⚠️ PROD Identity tables NON-prefixed Roles NOT AspNetRoles (Permissions cols RoleId/MenuKey/CanR/C/U/D) · JS-rotate-css-FROZEN #69 harmless — full verbatim → git 1408d4e + archive _INDEX)_
  • (S91 #470 55494ad GO-LIVE no-resurrect demo-NCC seed-gate DemoSeed:Disabled baseline-422 — pre+post-restart demo-code=0 + 3-evidence dll-mtime+w3wp-recycle; NEW PATTERN go-live no-resurrect = (a) pre+post DB-count diff + (b) dll in-window + (c) w3wp StartTime ≥ run-window; integrated-auth -E; full verbatim → archive 2026-06.md via _INDEX)
  • (S90 #465 a71753b FULL-STACK PE draft-private+un-gate-suggested-price js-rotate-css-FROZEN · S89 #348 1aa3bdb PE create-contract-1→N multi-winner endpoint-411→401 · S88-quad #347 73cce1f PE D1 Block-B re-key+CEO-notify · S88-bis #346 0d3c0cf FE-only PE-layout js+css-BOTH-rotate · S88 #345 9b1ee0c = Mig 59 AddCcmBudgetPeriodToPurchaseEvaluation AddColumn-no-backfill VERIFIED-PROD setter PATCH-401 + 7 authz-test → all PASS, full verbatim → git + archive 2026-06.md via _INDEX)
  • (S87 #344 f1b10e9 FULL-STACK PE CCM 3-fix + Mig 58 AddPeMultiWinnerAndLevelFinalize 2-AddColumn-bit+backfill winners=13 VERIFIED-PROD, select-winner endpoint-411→401-wired, baseline-395 · S86 #343 ae4f106 BE-only notify-exclude-drafter baseline-377 ship-proof=Infrastructure.dll-Last-Modified · S85 #342 d4f6021 DOCS+SQL-no-op INVERTED-PASS-when-FROZEN baseline-374 (gotcha #73 TZ + #69-REFINE) · S83 #333 fc1f19d BE-only PE-guard-V2-workflow-required hash-FROZEN-but-shipped baseline-356 · S78 #330 7886fd0 cross-stack enum-attach-on-approve · S76 #318 e33481e Mig56-ProInitial/Adjust backfill-4rows → all PASS, full verbatim → archive 2026-06.md via _INDEX)
  • (S83 #325 e29391e FE-only budget-subitem indent js-rotate-css-FROZEN test351 · S77 #329 e823694 FE-only banner Trả-lại · #328 424131d BE-only notify-block test-gate-KEY · #327 fa6654b FE-list-restructure · #326 b5aa72d BE-authz asymmetric · #323 #322 #321 #320 PE UX batch → all PASS, FIFO-trimmed, full verbatim git + archive _INDEX)
  • (S74-bis #315 8655ebf Mig55 CcmNote · S73 #313 1d86abc Mig54 5-cols + endpoint-401-not-404-probe · S69b #307 1f8947e Office-golive seed-16/16-DB-query-proof → archive _INDEX + git)
  • Older runs (S75 #301 ← S62 #286 ← … → S29 #232) full verbatim → git d2f52ba + archive/2026-06.md (incl #291 06-16 FAIL forensic [gotcha #65]); pre-S38 → archive/2026-05-{runs,q2,q3,q4}.md.

🔄 Curate trigger

  • ~30KB → archive recent runs → L2 archive/<period>.md (byte-exact append) + _INDEX.md substring pointer. Dup failure patterns → merge. Stale >3mo → remove.

  • Last curate: 2026-06-20 S80 (em-main, archive-gate keep-floor-hit → manual) (38.8→~16KB): moved 3 huge run-entries #330/#318/#325 (lines 73/90/96 byte-exact via sed) → archive/2026-06.md (100.6→116.7KB) + _INDEX.md +3 substring pointers (verified count=1). KEPT foundation (bug-patterns/5-stage/10-point gold) + compressed essentials + run-stubs; UPDATED stale (130→354 test Stage-3 + 263→354 essentials, Mig 55→57, bundle→#330) + dropped verbose per-run bundle-history (S86/S85/S84 → archived run-entries).
  • Prev curate: 2026-06-17 S70 Harness-9 (65.2→23.2KB dark-matter recovery, built _INDEX/gist gen:1). Earlier: S66 · S40.