diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index e548ca1..82591ec 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -1,6 +1,14 @@ # HANDOFF — Brief 5 phút cho session tiếp theo -**Last updated:** 2026-05-22 (Session 29 **FINAL wrap** — **🎯 2 big plans END-TO-END deployed prod: Plan CA + Plan B Contract V2**). 20 commits cumulative S29 push 4 CI Runs PASS (#229+#230 Plan CA + #231+#232 Plan B). **Plan CA Move "Cấu hình danh mục dùng chung" admin → eoffice**: 9 menu (Master + 4 leaf + Catalogs + 4 sub-catalogs) move + role mới `CatalogManager` + demo user `catalog.manager@solutions.com.vn / CatalogMgr@2026`. 5 chunks A+B+C+D+D2 + Hotfix 1 (resolvePath staticMap silent sidebar drop — anh UAT screenshot catch). **Plan B Contract V2 wire mirror PE Mig 22-26**: COEXIST V1+V2 (7 V1 contract giữ behavior). 9 chunks A1+A2+B+C+B2+E1+D+E2+E3 + Hotfix Reviewer (ApplicableType=Contract guard MAJOR security catch) + Hotfix CICD (SeedSampleContractWorkflowV2 OUT of DemoSeed gate — V2 path BLOCKED prod nếu gated — **gotcha #51 NEW INFRASTRUCTURE vs DEMO seed**). Mig 32+33 prod applied. Sample `QT-HD-V2-001` seeded post Hotfix CICD. fe-admin/fe-user bundle 4× rotate cumulative S29. **Multi-agent ROI S29 ~565K total**: Investigator 2 spawn ~40K + Implementer Case 2 5 spawn ~82K (1 stopped mid-task E3) + Reviewer 4 spawn ~355K (Smart Friend MAJOR catch 2 lần) + CICD Monitor 3 spawn ~90K (2 fail 529 transient em main fallback manual + 1 CRITICAL catch). Em main solo ~150K coordinate + Chunk B/B2 cross-stack + E3 finish + 2 hotfix re-edit. **Smart Friend pattern proven 4× cumulative S22 #44 + S25 #48 + S29 Reviewer #ApplicableType + S29 CICD #DemoSeed**. **MCP RAG tools added 4 agent definitions** (commit `b51fc94`) — search_memory + cross_project_search — anh restart CLI hot-reload S30+. **Test gate:** 111/111 PASS preserved (UAT mode skip per chunk, V2 ApproveV2Async 150 LOC NO test cover — defer test bundle Plan B Wrap session). **8 patterns NEW saved S29** (xem STATUS.md). **NEW capability prod end-to-end:** (1) Anh login Drafter fe-user → `/contracts/new?type=1` → Workspace dropdown pick `QT-HD-V2-001` → submit ChoDuyet → CCM `binh.le` approve → Service ApproveV2Async UPSERT opinion + advance → terminal gen mã HĐ + Phase=DaPhatHanh (mirror PE V2 flow). (2) Anh login `catalog.manager@solutions.com.vn / CatalogMgr@2026` → fe-user sidebar Master + 4 leaf + Catalogs + 4 sub-catalogs CRUD full. **Pending S30+**: anh UAT verify end-to-end V2 contract flow + restart CLI MCP RAG load + curate 4 MEMORY (Investigator 25.2KB + Implementer 35.4KB + Reviewer ~22KB + CICD 24.9KB — Implementer over hard threshold, dedicate curate session) + test bundle Plan B (regression test ApproveV2Async + ApplicableType validation). **Stats final S29:** 33 mig (+2) · 60 tables (+1) · ~148 endpoints (+1) · 38 FE pages · 111 test · 51 gotcha (+1 #51) · 25 memory · 6 skills · 4 sub-agents · 14 AppRoles (+1) · 34 active prod users (+1) · 7 V1 contracts + V2 ready · 4× bundle rotate.) +**Last updated:** 2026-05-26 (Session 31 — **🎯 RAG v1.3 Baseline PASS + retrieval.py fix**) + +**S30 (2026-05-26):** Setup RAG Framework v1.3 governance + eval framework. Bootstrap 2949 chunks (164 files). Golden set v1.1 14 queries. Baseline v1.0=0.455 FAIL. v1.1 tentative=0.364 (MCP stale — vector broken). Commit b223466 (docs-only, CI skip). + +**S31 (2026-05-26):** Diagnose vector search broken: qdrant-client 1.18 removed search() → AttributeError silently swallowed. Fix retrieval.py: search()→query_points().points. Re-run eval: **recall@5=1.000 (11/11) PASS** avg_rerank=0.847. Quality gate PASS. Gotcha #52 added. STATUS/HANDOFF updated. + +**⚠️ CRITICAL next session:** CLI restart required — retrieval.py fix in AI_INFRA source but MCP server still loaded old code. After CLI restart: live MCP search_memory will use fixed vector pipeline. + +**Last updated S29 (prev):** 2026-05-22 (Session 29 **FINAL wrap** — **🎯 2 big plans END-TO-END deployed prod: Plan CA + Plan B Contract V2**). 20 commits cumulative S29 push 4 CI Runs PASS (#229+#230 Plan CA + #231+#232 Plan B). **Plan CA Move "Cấu hình danh mục dùng chung" admin → eoffice**: 9 menu (Master + 4 leaf + Catalogs + 4 sub-catalogs) move + role mới `CatalogManager` + demo user `catalog.manager@solutions.com.vn / CatalogMgr@2026`. 5 chunks A+B+C+D+D2 + Hotfix 1 (resolvePath staticMap silent sidebar drop — anh UAT screenshot catch). **Plan B Contract V2 wire mirror PE Mig 22-26**: COEXIST V1+V2 (7 V1 contract giữ behavior). 9 chunks A1+A2+B+C+B2+E1+D+E2+E3 + Hotfix Reviewer (ApplicableType=Contract guard MAJOR security catch) + Hotfix CICD (SeedSampleContractWorkflowV2 OUT of DemoSeed gate — V2 path BLOCKED prod nếu gated — **gotcha #51 NEW INFRASTRUCTURE vs DEMO seed**). Mig 32+33 prod applied. Sample `QT-HD-V2-001` seeded post Hotfix CICD. fe-admin/fe-user bundle 4× rotate cumulative S29. **Multi-agent ROI S29 ~565K total**: Investigator 2 spawn ~40K + Implementer Case 2 5 spawn ~82K (1 stopped mid-task E3) + Reviewer 4 spawn ~355K (Smart Friend MAJOR catch 2 lần) + CICD Monitor 3 spawn ~90K (2 fail 529 transient em main fallback manual + 1 CRITICAL catch). Em main solo ~150K coordinate + Chunk B/B2 cross-stack + E3 finish + 2 hotfix re-edit. **Smart Friend pattern proven 4× cumulative S22 #44 + S25 #48 + S29 Reviewer #ApplicableType + S29 CICD #DemoSeed**. **MCP RAG tools added 4 agent definitions** (commit `b51fc94`) — search_memory + cross_project_search — anh restart CLI hot-reload S30+. **Test gate:** 111/111 PASS preserved (UAT mode skip per chunk, V2 ApproveV2Async 150 LOC NO test cover — defer test bundle Plan B Wrap session). **8 patterns NEW saved S29** (xem STATUS.md). **NEW capability prod end-to-end:** (1) Anh login Drafter fe-user → `/contracts/new?type=1` → Workspace dropdown pick `QT-HD-V2-001` → submit ChoDuyet → CCM `binh.le` approve → Service ApproveV2Async UPSERT opinion + advance → terminal gen mã HĐ + Phase=DaPhatHanh (mirror PE V2 flow). (2) Anh login `catalog.manager@solutions.com.vn / CatalogMgr@2026` → fe-user sidebar Master + 4 leaf + Catalogs + 4 sub-catalogs CRUD full. **Pending S30+**: anh UAT verify end-to-end V2 contract flow + restart CLI MCP RAG load + curate 4 MEMORY (Investigator 25.2KB + Implementer 35.4KB + Reviewer ~22KB + CICD 24.9KB — Implementer over hard threshold, dedicate curate session) + test bundle Plan B (regression test ApproveV2Async + ApplicableType validation). **Stats final S29:** 33 mig (+2) · 60 tables (+1) · ~148 endpoints (+1) · 38 FE pages · 111 test · 51 gotcha (+1 #51) · 25 memory · 6 skills · 4 sub-agents · 14 AppRoles (+1) · 34 active prod users (+1) · 7 V1 contracts + V2 ready · 4× bundle rotate.) **Last updated S28 prev:** 2026-05-22 (Session 28 **FINAL wrap** — **🎯 S28 cumulative t1→t6 + Layer A governance apply + 4 sub-agent flush MEMORY DONE**. Bro broadcast Layer A governance active 2026-05-22 (3-Layer distributed: A project-local active / B shared_global defer / C infrastructure anh pqhuy). **Quên rule cũ "mọi tương tác PHẢI store_memory" — ABANDONED** (em main S28 t2 over-reach self-authorize cross-project rule → caught S28 t4 → scope-down về SOLUTION_ERP self-discipline). Em main apply Layer A policy local: **4-category default** (decision/lesson/pattern-proven-≥2×/session-wrap) + skip list nguyên + 5 SOLUTION_ERP-specific adjustment + tag schema mandatory `[type, phase, bc-or-module]` + phase enum 11 (current phase-9 UAT) + BC/module 8 domain + 5 cross-cutting + optional prefix 8 + source_path convention `solution_erp//-` + eval ritual weekly Friday 5 metric từ 2026-05-29 → end-trial 2026-06-05 + 10 golden query draft. **4 sub-agent flush MEMORY DONE post-S27 model:inherit fix** — Investigator (aeda4 alive) + Implementer (a2a2f spawn) + Reviewer (a9e94 spawn) + CICD Monitor (a8098 spawn) — registry hot-reload verified, Recent activity entry S28 wrap góc nhìn từng con preserved cumulative S1-S27 narrative untouched. **6 chunk stored RAG S28 t2-t6** with tags schema mandatory format. **+1 memory user-level `feedback_interaction_logging_rag.md` SCOPE-DOWN S28 t4** (lesson authority boundary + implicit consent risk). **Hit rate verify 3 query**: Plan B Contract V2 rerank 0.84 ✓ + gotcha #48 rerank 0.84/0.82 ✓ + per-NV 10-surface checklist ✓ — cumulative > 0.7 threshold pass. **State chốt S28:** 31 mig · 59 tables · ~146 endpoints · 35 FE pages · **111 test pass** · 49 gotcha · **25 memory user-level (+1)** · 6 skills · 4 sub-agents verified · **3,462 RAG chunks SOLUTION_ERP** (+6 chunk S28). **0 commit remote S28** (all local docs + memory + RAG store — anh chốt push thủ công). Plan B Contract V2 wire kick-off PRIORITY HIGH S29+ — 8-chunk pre-allocated 🟦 Investigator pre-flight audit + 🟨 Implementer Case 2 mirror PE V2 (4 chunk) + 👤 Chủ trì Solo Service ApproveV2Async + 🟥 Reviewer pre-commit Smart Friend + 🟩 CICD Monitor post-deploy Run verify. Plan AI Phase 5 distributed bootstrap 4 project khác (NamGroup/DH/Ashico/Vipix) đã DONE — total 18,532 chunks indexed cross 5 project. VIPIX hosting governance docs `D:\Dropbox\CONG_VIEC\VIPIX_MULTISITE_PROJECT\docs\governance\RAG-LAYER-A-PROJECT-LOCAL.md` v1.1 (em đọc + apply scope SOLUTION_ERP). 2-week trial monitoring 2026-05-22 → 2026-06-05 weekly Friday eval — Friday 2026-05-29 first checkpoint.) diff --git a/docs/STATUS.md b/docs/STATUS.md index 3695528..a976f49 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -2,7 +2,9 @@ > **Update rule:** trước khi bắt đầu 1 task → ghi row vào `🔥 In Progress`. Xong → chuyển sang `✅ Recently Done`. -**Last updated:** 2026-05-22 (Session 29 **FINAL wrap** — **🎯 Plan CA + Plan B Contract V2 wire ALL DEPLOYED PROD**: cumulative **20 commits push 4 CI Runs PASS** (#229 + #230 + #231 + #232). **2 big plans done end-to-end:** Plan CA Move "Cấu hình danh mục dùng chung" admin → eoffice (7 commits + Hotfix 1) + Plan B Contract V2 wire mirror PE Mig 22-26 (9 chunks + Hotfix Reviewer + Hotfix CICD). **Multi-agent ROI explosive S29:** 🟦 Investigator 2 spawn ~40K (Plan CA terrain map + Plan B V1 audit) + 🟨 Implementer Case 2 5 spawn ~82K (Plan CA Chunk B + Plan B A2/C/D + E3 stopped mid-task) + 🟥 **Reviewer 4 spawn ~355K** (Plan CA Hotfix D2 password ≥12 MAJOR catch + Plan B Hotfix ApplicableType=Contract MAJOR catch — Smart Friend pattern 4× cumulative) + 🟩 **CICD Monitor 3 spawn ~90K** (2 fail 529 transient em main fallback manual + Plan B SeedSampleContractWorkflowV2 CRITICAL DemoSeed gate catch — gotcha #51 NEW). **Patterns NEW saved S29:** (1) 9-menu permission terrain map (Investigator) (2) V1+V2 coexist boundary (Investigator) (3) reference template paths cross-module mirror PE→Contract (Investigator) (4) Pattern 12-bis cross-module entity cookie-cutter mirror (Implementer) (5) Pattern 16-bis 4-place mirror cross-app reinforced 2× (Implementer) (6) Cross-module security validation mirror Cat 3 (Reviewer) (7) INFRASTRUCTURE vs DEMO seed classification gotcha #51 (CICD) (8) Race condition em main + Implementer parallel BE → stash workaround OR sequential preferred (Implementer). **3 hotfix cumulative:** Plan CA HF1 resolvePath staticMap + Plan CA D2 password ≥12 chars + Plan B Reviewer ApplicableType=Contract + Plan B CICD SeedSampleContractWorkflowV2 ungate. **NEW capability prod:** demo user `catalog.manager@solutions.com.vn / CatalogMgr@2026` (role CatalogManager — full CRUD 9 menu Master+Catalogs) + Drafter có thể create V2 contract qua Workspace dropdown `QT-HD-V2-001` sample workflow. **MCP RAG tools added cho 4 agent definitions** (commit `b51fc94`) — anh restart CLI để hot-reload. State chốt: **33 mig (+2 Mig 32+33)** · **60 tables (+1 ContractLevelOpinions)** · **~148 endpoints (+1 V2 contract create)** · **38 FE pages (+3: 4 master moved + WorkflowMatrixView counted unchanged S24)** · **111 test PASS unchanged** (UAT mode skip per chunk + cookie-cutter mirror) · **51 gotcha (+1 #51 INFRASTRUCTURE vs DEMO seed)** · **25 memory user-level unchanged** · 6 skills · 4 sub-agents (MCP RAG added pending CLI restart) · **14 AppRoles (+1 CatalogManager)** · **34 active prod users (+1 catalog.manager)** · **7 V1 contracts pinned coexist V2** · Bundle hash 4× rotate cumulative S29 (fe-admin: `BvcWrq2z→leEMWFLU→BBADl46y` + fe-user: `Bg2FNeIz→DVBLmZlt→Dgn1iU9E→DA_VI3zO`).) +**Last updated:** 2026-05-26 (Session 31 — **🎯 RAG v1.3 Baseline PASS (11/11 recall@5=1.000) + retrieval.py critical fix**. **S30 RAG Setup:** Setup RAG Framework v1.3 governance () + evaluator spec + golden set v1.1 (14 queries) + baseline runs v1.0+v1.1 + . Bootstrap 2949 chunks (164 files — fixed Anti #24 projects.json root_path + added corpus). Commit docs-only CI skip. **S31 Eval Fix:** Discover removed method → silently returns (gotcha #52 NEW). Fix: . After fix: **recall@5 = 1.000 (11/11) PASS** (prev tentative 0.364 FAIL). Quality gate updated: baseline=1.000, avg_rerank=0.847. **Sub-agents S30/S31:** 0 spawns (docs/infra only). **Commits pushed:** (docs-only S30) — S31 pending. State: **33 mig** · 60 tables · ~148 endpoints · 38 FE pages · **111 test PASS** · **52 gotcha (+1 #52 qdrant-client search() removed)** · **25 memory user-level** · 6 skills · 4 sub-agents · **RAG v1.3 ACTIVE baseline PASS** 2949 chunks · CLI restart required for live MCP vector fix.) + +**Last updated S29 FINAL (prev):** 2026-05-22 (Session 29 — **🎯 Plan CA + Plan B Contract V2 wire ALL DEPLOYED PROD**: cumulative **20 commits push 4 CI Runs PASS** (#229 + #230 + #231 + #232). **2 big plans done end-to-end:** Plan CA Move "Cấu hình danh mục dùng chung" admin → eoffice (7 commits + Hotfix 1) + Plan B Contract V2 wire mirror PE Mig 22-26 (9 chunks + Hotfix Reviewer + Hotfix CICD). **Multi-agent ROI explosive S29:** 🟦 Investigator 2 spawn ~40K (Plan CA terrain map + Plan B V1 audit) + 🟨 Implementer Case 2 5 spawn ~82K (Plan CA Chunk B + Plan B A2/C/D + E3 stopped mid-task) + 🟥 **Reviewer 4 spawn ~355K** (Plan CA Hotfix D2 password ≥12 MAJOR catch + Plan B Hotfix ApplicableType=Contract MAJOR catch — Smart Friend pattern 4× cumulative) + 🟩 **CICD Monitor 3 spawn ~90K** (2 fail 529 transient em main fallback manual + Plan B SeedSampleContractWorkflowV2 CRITICAL DemoSeed gate catch — gotcha #51 NEW). **Patterns NEW saved S29:** (1) 9-menu permission terrain map (Investigator) (2) V1+V2 coexist boundary (Investigator) (3) reference template paths cross-module mirror PE→Contract (Investigator) (4) Pattern 12-bis cross-module entity cookie-cutter mirror (Implementer) (5) Pattern 16-bis 4-place mirror cross-app reinforced 2× (Implementer) (6) Cross-module security validation mirror Cat 3 (Reviewer) (7) INFRASTRUCTURE vs DEMO seed classification gotcha #51 (CICD) (8) Race condition em main + Implementer parallel BE → stash workaround OR sequential preferred (Implementer). **3 hotfix cumulative:** Plan CA HF1 resolvePath staticMap + Plan CA D2 password ≥12 chars + Plan B Reviewer ApplicableType=Contract + Plan B CICD SeedSampleContractWorkflowV2 ungate. **NEW capability prod:** demo user `catalog.manager@solutions.com.vn / CatalogMgr@2026` (role CatalogManager — full CRUD 9 menu Master+Catalogs) + Drafter có thể create V2 contract qua Workspace dropdown `QT-HD-V2-001` sample workflow. **MCP RAG tools added cho 4 agent definitions** (commit `b51fc94`) — anh restart CLI để hot-reload. State chốt: **33 mig (+2 Mig 32+33)** · **60 tables (+1 ContractLevelOpinions)** · **~148 endpoints (+1 V2 contract create)** · **38 FE pages (+3: 4 master moved + WorkflowMatrixView counted unchanged S24)** · **111 test PASS unchanged** (UAT mode skip per chunk + cookie-cutter mirror) · **51 gotcha (+1 #51 INFRASTRUCTURE vs DEMO seed)** · **25 memory user-level unchanged** · 6 skills · 4 sub-agents (MCP RAG added pending CLI restart) · **14 AppRoles (+1 CatalogManager)** · **34 active prod users (+1 catalog.manager)** · **7 V1 contracts pinned coexist V2** · Bundle hash 4× rotate cumulative S29 (fe-admin: `BvcWrq2z→leEMWFLU→BBADl46y` + fe-user: `Bg2FNeIz→DVBLmZlt→Dgn1iU9E→DA_VI3zO`).) **Last updated S29 prev wrap1:** 2026-05-22 (Session 29 — **🎯 Plan CA wire DONE + Hotfix 1 deploy**: 6 commits + 1 hotfix push `3cb54e0..e55d96b` cumulative. Move "Cấu hình danh mục dùng chung" (9 menu: Master + Suppliers + Projects + Departments + Catalogs + 4 sub-catalogs) từ fe-admin → fe-user/eoffice + add role mới `CatalogManager` + demo user `catalog.manager@solutions.com.vn`. **Chunks:** A 80d39a0 BE Role + Seed (em main, 64 LOC) · B 06a441c FE move 4 master pages 948 LOC (Implementer Case 2 cookie-cutter mirror byte-identical SHA256 4 file) · C c995f42 Sidebar filter 2 app (em main, 14 LOC) · D 4a592cf Seed demo user (em main, 7 LOC) · D2 68bcedd Hotfix password ≥12 chars S22+2 policy (em main, Reviewer pre-push CATCH critical) · HF1 e55d96b Add 7 entries `resolvePath` staticMap fe-user (em main, anh UAT screenshot CATCH silent sidebar drop). **2 CI runs PASS:** Run #229 sha=68bcedd 3m32s + Run #230 sha=e55d96b 3m24s. Bundle hash 2× rotate: fe-admin `BvcWrq2z→leEMWFLU`, fe-user `Bg2FNeIz→DVBLmZlt→Dgn1iU9E`. **Multi-agent ROI Plan CA:** 🟦 Investigator 1 spawn ~15K (terrain map 9 menu + GOTCHA tree-inherit) + 🟨 Implementer Case 2 1 spawn ~10K (byte-identical mirror 4 page) + 🟥 Reviewer 2 spawn ~165K (CRITICAL catch password policy) + 🟩 CICD Monitor 2 spawn fail 529 transient (em main fallback manual smoke) + 👤 Chủ trì Solo Chunk A/C/D/D2/HF1 ~100K. **NEW gotcha #50** "Page move cross-app — `Layout.tsx` `resolvePath` staticMap missed mirror → silent sidebar drop" + Pattern reusable cross-project: **4-place mirror checklist** khi Implementer Case 2 cookie-cutter copy page (page + Routes + menuKeys.ts + Layout.tsx staticMap — Implementer Chunk B missed point 4). Implementer MEMORY Pattern 16-bis NEW. **State chốt S29 Plan CA:** 31 mig (no schema change) · 59 tables · ~146 endpoints · **36 FE pages (+4: fe-user master/{Suppliers,Projects,Departments,Catalogs}Page)** · **111 test pass unchanged** (UAT skip per chunk + cookie-cutter mirror) · **50 gotcha (+1 #50)** · 25 memory user-level · 6 skills · 4 sub-agents · **13 AppRoles (+1 CatalogManager)** · **34 active prod users (+1 catalog.manager@solutions.com.vn)**. Catalog manager login `CatalogMgr@2026` (15 chars policy compliant). 7 commits remote `3cb54e0..e55d96b`.) diff --git a/docs/changelog/sessions/2026-05-26-1630-s31-rag-baseline-pass.md b/docs/changelog/sessions/2026-05-26-1630-s31-rag-baseline-pass.md new file mode 100644 index 0000000..bc344bb --- /dev/null +++ b/docs/changelog/sessions/2026-05-26-1630-s31-rag-baseline-pass.md @@ -0,0 +1,95 @@ +# Session 31 — RAG v1.3 Baseline PASS + retrieval.py critical fix + +**Date:** 2026-05-26 +**Duration:** ~1.5h +**Commits:** 1 pending (S31 docs/infra) +**Sub-agents:** 0 + +## Summary + +Session 31 followed directly from S30 (RAG v1.3 setup). S30 had bootstrapped 2949 chunks but the v1.1 baseline was tentative (PENDING_RELOAD). S31 re-ran the eval after CLI restart and discovered the real root cause. + +## Root Cause Discovered: qdrant-client 1.18 removed search() + +**Symptom:** Most golden set queries returned 0 results even with 2949 Qdrant points green. + +**Root cause:** `qdrant-client 1.18.0` removed `QdrantClient.search()` method entirely. `retrieval.py:vector_search()` called `_qdrant.search(...)` which raised `AttributeError`. This exception was silently swallowed by `except Exception: continue` → `vec_results = []` always → pipeline fell back to BM25-only. + +**Diagnosis chain:** +1. q01/q02/q03/q08 worked (BM25 exact match found all tokens in same chunk) +2. q04/q05/q06/q07/q09/q10/q11 returned 0 (BM25 strict AND failed, vector broken) +3. Qdrant HTTP `/collections/proj_solution_erp` → `status=green`, `points_count=2949` ✓ +4. BM25 SQLite FTS: `MediatR` → 3 hits, `fallback` → 173 hits, `ApprovalWorkflow` → 40 hits (data IS there) +5. `python -c "from qdrant_client import QdrantClient; c=QdrantClient(...); c.search"` → AttributeError confirmed +6. Tested `query_points(query=...).points` → worked, 5 results returned + +## Fix Applied + +**File:** `D:\Dropbox\CONG_VIEC\AI_INFRA\claude-rag\lib\retrieval.py` — `vector_search()` function + +```python +# Before (broken in qdrant-client 1.18): +hits = _qdrant.search( + collection_name=c, + query_vector=query_vector, + limit=top_k, + with_payload=True, +) +for h in hits: + results.append({"chunk_id": h.id, ...}) + +# After (qdrant-client 1.12+ query_points API): +resp = _qdrant.query_points( + collection_name=c, + query=query_vector, # param renamed + limit=top_k, + with_payload=True, +) +for h in resp.points: # access .points on response + results.append({"chunk_id": h.id, ...}) +``` + +## Eval Results + +| Version | recall@5 | hits/11 | Pass | +|---|---|---|---| +| v1.0 (S30) | 0.455 | 5/11 | ❌ FAIL | +| v1.1 tentative (S30 stale MCP) | 0.364 | 4/11 | ❌ FAIL | +| **v1.1 FINAL (S31 after fix)** | **1.000** | **11/11** | **✅ PASS** | + +**avg_top1_rerank:** 0.847 (gate threshold 0.65) ✅ + +**Negative query behavior (q12/q13/q14):** +- q12 GraphQL: rerank 0.43 (< 0.7 threshold — not a false positive) +- q13 Redis: rerank 0.38 — correct exclusion +- q14 Kubernetes: rerank 0.43 — correct exclusion + +## Files Updated + +**AI_INFRA:** +- `claude-rag/lib/retrieval.py` — vector_search() fix +- `claude-rag/eval_v11.py` — temp eval script (not in SOLUTION_ERP) + +**SOLUTION_ERP:** +- `eval/runs/2026-05-26-baseline-v1.1-final.json` — official baseline record +- `eval/trial-state-lock.json` — quality_gate.pass=true, baseline_recall=1.000 +- `docs/gotchas.md` — gotcha #52 added +- `docs/STATUS.md` — S31 entry added +- `docs/HANDOFF.md` — S31 entry added + +## New Gotcha #52 + +`qdrant-client 1.18` removed `search()` API. When library upgrades break internal API, silent `except Exception: continue` in retrieval pipeline produces no error but returns empty results. Fix: use `query_points().points`. Lesson: pin qdrant-client version OR add health check `hasattr(_qdrant, 'query_points')`. + +## Next Session + +**CRITICAL:** CLI restart to pick up retrieval.py fix (MCP server still loaded old code in memory). + +After CLI restart: +- Live `search_memory` will use fixed vector + BM25 pipeline +- RAG recall@5 = 1.000 confirmed + +**Other pending:** +- Curate 4 sub-agent MEMORY files (Implementer over threshold) +- Plan B-Wrap (Contract V2 test bundle BW1-BW7) +- Phase 9 UAT hard blockers (SMTP, rotate creds, SQL backup, win-acme cert) diff --git a/docs/gotchas.md b/docs/gotchas.md index 2e4644d..0dd38a7 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -921,6 +921,48 @@ const staticMap: Record = { - Original Plan CA Chunk B commit: `06a441c` (Implementer missed point 4) - Mirror: `fe-admin/src/components/Layout.tsx:33-53` +### 52. `qdrant-client` 1.18 xóa `search()` API — `except Exception: continue` nuốt lỗi silent → vector search luôn trả `[]` (Session 31 RAG eval diagnosis) + +**Triệu chứng:** `search_memory` MCP tool chỉ trả kết quả cho queries có BM25 exact-match tốt (có tất cả token trong cùng 1 chunk). Queries dùng ngữ nghĩa / multi-hop concept → **0 results** dù Qdrant `points_count=2949` green. + +**Root cause:** `qdrant-client 1.18.0` removed `QdrantClient.search()` method hoàn toàn. `retrieval.py` vẫn gọi `_qdrant.search(...)`: +```python +# OLD (broken in 1.18): +hits = _qdrant.search( + collection_name=c, + query_vector=query_vector, + limit=top_k, + with_payload=True, +) +# AttributeError: 'QdrantClient' object has no attribute 'search' +``` +Exception bị `except Exception: continue` nuốt silent → `vec_results = []` mọi lúc → pipeline chỉ có BM25. + +**Kết quả trước fix:** RAG eval v1.1 recall@5 = 0.455 (chỉ BM25 queries). v1.0 = 0.455 (same reason). BM25 strict AND-match: ALL tokens phải cùng chunk → multi-token query 8+ tokens fail. + +**Fix (retrieval.py):** +```python +# NEW (qdrant-client 1.12+ query_points API): +resp = _qdrant.query_points( + collection_name=c, + query=query_vector, # ← param renamed: query_vector → query + limit=top_k, + with_payload=True, +) +for h in resp.points: # ← .points không phải iterable trực tiếp +``` + +**Kết quả sau fix:** recall@5 = 1.000 (11/11), avg rerank = 0.847. PASS gate. + +**Phòng tránh:** Pin `qdrant-client` version trong deps (`==1.x.y`). Hoặc thêm health-check startup: `assert hasattr(_qdrant, 'query_points'), "upgrade qdrant-client"`. **KHÔNG dùng `except Exception: continue` che-mờ lỗi API** — ít nhất log warning. + +**References:** +- AI_INFRA: `claude-rag/lib/retrieval.py` `vector_search()` function — fixed 2026-05-26 S31 +- Eval run: `eval/runs/2026-05-26-baseline-v1.1-final.json` +- Diagnosis: Qdrant REST `/collections/proj_solution_erp` green (2949 points), `bm25.db` 2949 chunks → pipeline broken, not data. Confirmed via `python -c "from qdrant_client import QdrantClient; client.search(...)"` → AttributeError. + +--- + ## Checklist debug bug mới 1. Build pass không? → fail → check using + package version compat @@ -950,3 +992,4 @@ const staticMap: Record = { 25. Nếu page move cross-app (Implementer Case 2) nhưng menu leaf KHÔNG hiện sidebar dù BE trả permission OK → check `Layout.tsx` `resolvePath` staticMap miss key mapping → MenuLeaf null guard silent drop (#50). 4-place mirror checklist: page + Routes + menuKeys.ts + Layout.tsx staticMap 26. Nếu new Seed method KHÔNG chạy prod dù dotnet build PASS + deploy SUCCESS → check nested inside `if (!demoSeedDisabled)` gate (Plan T S23 t10 flag enabled prod) → INFRASTRUCTURE seed phải PROMOTE OUT of DemoSeed gate (#51). Decision tree: production cần để work end-to-end? YES → ungate 25. Nếu UI audit list show `Đã gửi duyệt → Đã gửi duyệt` lặp gây nhầm → drop dual-phase badge khi state machine self-loop, thay Decision badge + next-target hint parse từ comment (#49) +27. Nếu RAG `search_memory` trả 0 results dù Qdrant green + BM25 có data → `qdrant-client` upgrade xóa `search()` method, bị nuốt silent. Test: `python -c "from qdrant_client import QdrantClient; c=QdrantClient(url='http://127.0.0.1:6333'); c.search"`. Fix: dùng `query_points(query=...).points` (#52) diff --git a/eval/runs/2026-05-26-baseline-v1.1-final.json b/eval/runs/2026-05-26-baseline-v1.1-final.json new file mode 100644 index 0000000..20f3c33 --- /dev/null +++ b/eval/runs/2026-05-26-baseline-v1.1-final.json @@ -0,0 +1,36 @@ +{ + "run_date": "2026-05-26", + "run_label": "v1.1 FINAL — after retrieval.py fix (query_points)", + "golden_set_version": "v1.1", + "spec": "A", + "status": "COMPLETE", + "recall_at_5": 1.0, + "hits": 11, + "positive_queries": 11, + "pass_gate": true, + "avg_top1_rerank": 0.847, + "pipeline_fix": "retrieval.py vector_search(): search() → query_points() (qdrant-client 1.18 removed search())", + "results": [ + {"id":"q01","hit":true,"top1_source":"architecture.md","top1_rerank":0.8867,"note":"CI/CD gotcha #39 — BM25+vector hit"}, + {"id":"q02","hit":true,"top1_source":"architecture.md","top1_rerank":0.9102,"note":"CI/CD gotcha #41 — BM25+vector hit"}, + {"id":"q03","hit":true,"top1_source":"2026-05-22-s29-plan-ca-plan-b-contract-v2-wire.md","top1_rerank":0.8828,"note":"Session log — matches 'sessions' expected hint"}, + {"id":"q04","hit":true,"top1_source":"ef-core-migration/SKILL.md","top1_rerank":0.8672,"note":"EF migration SKILL — vector retrieval required"}, + {"id":"q05","hit":true,"top1_source":"gotchas.md","top1_rerank":0.9375,"note":"IIS gotcha #25 — vector retrieval required"}, + {"id":"q06","hit":true,"top1_source":"reviewer.md","top1_rerank":0.7422,"note":"CQRS MediatR — vector retrieval, marginal rerank"}, + {"id":"q07","hit":true,"top1_source":"reviewer.md","top1_rerank":0.7891,"note":"Smart Friend/Cognition — vector retrieval required"}, + {"id":"q08","hit":true,"top1_source":"project_solution_erp.md","top1_rerank":0.8398,"note":"PE V2 ApprovalWorkflow — BM25 hit (was working before)"}, + {"id":"q09","hit":true,"top1_source":"feedback_multi_agent_setup.md","top1_rerank":0.8047,"note":"Implementer worktree MAX_PATH — vector+memory hit"}, + {"id":"q10","hit":true,"top1_source":"feedback_subagent_setup_pitfalls.md","top1_rerank":0.8086,"note":"Sub-agent S27 fix — vector+memory hit"}, + {"id":"q11","hit":true,"top1_source":"gotchas.md","top1_rerank":0.8242,"note":"ApprovalWorkflow V1 V2 fallback — vector retrieval required"}, + {"id":"q12","hit":true,"note":"CORRECT EXCLUSION — rerank 0.4336 < 0.7 (GraphQL not in project)"}, + {"id":"q13","hit":true,"note":"CORRECT EXCLUSION — rerank 0.3789 < 0.7 (Redis not in project)"}, + {"id":"q14","hit":true,"note":"CORRECT EXCLUSION — rerank 0.4277 < 0.7 (Kubernetes not in project)"} + ], + "_fix_diagnosis": { + "root_cause": "qdrant-client 1.18.0 removed QdrantClient.search() method. retrieval.py used search() → AttributeError silently swallowed in except Exception clause → vec_results always empty. Only BM25 working, and BM25 strict FTS AND-match failed for multi-token queries.", + "fix": "retrieval.py vector_search(): _qdrant.search(query_vector=...) → _qdrant.query_points(query=...).points", + "impact": "Before fix: only BM25 queries worked (q01/q02/q03/q08). After fix: all 11 positive queries hit.", + "mcp_restart_required": "retrieval.py fix applied to AI_INFRA source. MCP server process still loaded old code — CLI restart required for live MCP to use fixed pipeline.", + "new_gotcha": "gotcha #52: qdrant-client 1.18 removed search() API — upgrade detection via 'except Exception: continue' masks error silently. Fix: use query_points(). Version bump: add dep pin or version check." + } +} diff --git a/eval/trial-state-lock.json b/eval/trial-state-lock.json index 92036dc..1ef923d 100644 --- a/eval/trial-state-lock.json +++ b/eval/trial-state-lock.json @@ -5,14 +5,14 @@ "governance_path": "docs/governance/README.md", "golden_set_version": "v1.1", "spec_chosen": "A", - "baseline_note": "v1.0 attempted 2026-05-26 recall@5=0.455 FAIL. v1.1 attempted same day — pending CLI restart for accurate numbers. Official baseline = after CLI restart + re-run.", + "baseline_note": "v1.0: 2026-05-26 recall@5=0.455 FAIL (vector broken). v1.1 FINAL: 2026-05-26 recall@5=1.000 PASS after fixing retrieval.py search()→query_points() (qdrant-client 1.18 removed search()).", "quality_gate": { - "baseline_recall_at_5": null, - "baseline_recall_at_5_note": "PENDING — use v1.0=0.455 as conservative estimate until v1.1 re-run post CLI restart", - "baseline_avg_top1_rerank": 0.870, + "baseline_recall_at_5": 1.0, + "baseline_recall_at_5_note": "FINAL — v1.1 run post CLI restart + retrieval.py fix. 11/11 positive queries hit, avg_top1_rerank=0.847", + "baseline_avg_top1_rerank": 0.847, "gate_threshold_recall": 0.7, "gate_threshold_avg_rerank": 0.65, - "pass": false + "pass": true }, "drift_monitor": { "chunk_count_baseline": 2949, @@ -22,11 +22,11 @@ "last_indexed_at_baseline": "2026-05-26T13:09:21.816262" }, "trial_milestones": [ - {"week": 0, "date": "2026-05-26", "status": "setup", "label": "Setup complete — pending CLI restart for v1.1 baseline"}, - {"week": 1, "date": "2026-06-02", "status": "pending", "label": "v1.1 re-run after CLI restart + triage 0-result queries"}, - {"week": 2, "date": "2026-06-09", "status": "pending", "label": "Triage Case C/D failures (q05 IIS 25 + q06 CQRS)"}, - {"week": 3, "date": "2026-06-16", "status": "pending", "label": "Empirical chunk 512 vs 1500 retest"}, - {"week": 4, "date": "2026-06-23", "status": "pending", "label": "Final trial evaluation + decide v1.3 stable OR v1.4"} + {"week": 0, "date": "2026-05-26", "status": "complete", "label": "Setup + v1.1 baseline PASS (11/11) — after fix qdrant-client search()→query_points()"}, + {"week": 1, "date": "2026-06-02", "status": "skipped", "label": "Skipped — baseline already achieved week 0 with fix"}, + {"week": 2, "date": "2026-06-09", "status": "pending", "label": "Monitor drift + verify MCP live after CLI restart"}, + {"week": 3, "date": "2026-06-16", "status": "pending", "label": "Empirical chunk 512 vs 1500 retest (optional — current 1.0 recall may not need)"}, + {"week": 4, "date": "2026-06-23", "status": "pending", "label": "Final trial evaluation + decide v1.3 stable confirm"} ], "_decision_log": { "spec_a_vs_b_resolution_chosen": "Spec A — Strict. SOLUTION_ERP chunks canonical + finite scope (51 gotchas, patterns, decisions) → strict retrieval test appropriate.", @@ -34,12 +34,15 @@ "anatomy_threshold_chosen": "6/6 STRICT per v1.3 §5.2 (corpus 2949 chunks mature)", "governance_path_b_reason": "Path B delegation stub — no local customize needed at Phase 9 UAT stable stage. AI_INFRA canonical sufficient.", "bootstrap_correct_command": "python D:\\Dropbox\\CONG_VIEC\\AI_INFRA\\claude-rag\\bootstrap.py --config D:\\Dropbox\\CONG_VIEC\\SOLUTION\\SOLUTION_ERP\\.claude\\rag.json", - "bootstrap_wrong_command": "python D:\\Dropbox\\CONG_VIEC\\AI_INFRA\\claude-rag\\bootstrap.py --project solution_erp (DO NOT USE — resolves from CWD, not project config)" + "bootstrap_wrong_command": "python D:\\Dropbox\\CONG_VIEC\\AI_INFRA\\claude-rag\\bootstrap.py --project solution_erp (DO NOT USE — resolves from CWD, not project config)", + "retrieval_fix_applied": "retrieval.py vector_search(): search() → query_points() (qdrant-client 1.18 removed search()). Fixed 2026-05-26 S31. CLI restart required for MCP to pick up fix.", + "new_gotcha_52": "qdrant-client 1.18 removed QdrantClient.search() — use query_points() instead. Silent AttributeError swallowed by except Exception → vec_results always []. Symptom: BM25-only queries work, vector queries fail silently." }, "_anti_patterns_observed": { "anti_24_registry_drift": "projects.json had root_path=AI_INFRA for solution_erp entry. Fixed 2026-05-26. Caused 2 bad bootstraps (1351 AI_INFRA chunks written to proj_solution_erp collection).", "anti_23_source_path": "Absolute Windows path D:\\Dropbox\\... in chunk payload. Low priority fix-forward.", - "mcp_reload_lesson": "Bootstrap.py clearing Qdrant collection + BM25 → MCP server must be restarted to pick up new data. Similar to agents/*.md hot-reload requiring CLI restart." + "mcp_reload_lesson": "Bootstrap.py clearing Qdrant collection + BM25 → MCP server must be restarted to pick up new data. Similar to agents/*.md hot-reload requiring CLI restart.", + "anti_qdrant_client_upgrade": "qdrant-client 1.18 silently removed search() method. retrieval.py had 'except Exception: continue' masking the AttributeError → vector search returned [] for ALL queries. Diagnosed by testing QdrantClient.search() directly and getting AttributeError. Fix: query_points().points. Lesson: pin qdrant-client version OR test search() in health check on startup." }, "_lessons": [ "CRITICAL: --project flag overrides only collection_name, NOT project root. Always use --config for cross-project bootstrap.",