diff --git a/STATUS.md b/STATUS.md index 4136c6f..d31bc7e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,321 +1,137 @@ -# KWCode 项目状态记录 +# KWCode Project Status -> 项目路径：D:\program\codeagent2604\kwcode -> GitHub：https://github.com/val1813/kwcode -> 启动日期：2026-04-26 -> 目标：本地模型 coding agent，通过确定性专家流水线让本地模型达到最高任务完成率 +> Path: D:\program\codeagent2604\kwcode +> GitHub: https://github.com/val1813/kwcode +> Started: 2026-04-26 +> Goal: Local-model coding agent — maximize task completion rate via deterministic expert pipeline --- -## 当前状态：v1.4.0 (2026-05-06) +## Current: v1.5.0 (2026-05-06) -424/424 测试全绿。v1.4.0 新增：多语言AST支持 + FastAPI Server + Textual TUI + VSCode插件。 +451/451 tests green. Architecture upgrade: isolated search + cross-file contracts + PENCIL compression. -### v1.4.0 新增：多语言 + TUI + IDE兼容（3 个模块） +### v1.5.0 — Isolated Search + Cross-File Contracts -理论来源：XRAY MCP Server(ast-grep选型) + OpenCode(client/server分离) + CodeCompass(工具采用率) +Theory: WarpGrep (isolated search subagent) + CGM (graph-injected attention) + PENCIL (erase intermediate state) + SWE-ContextBench (context quality > model size) -**模块A: 多语言AST支持** -- `ast_engine/language_detector.py`：7语言检测(Python/JS/TS/Go/Rust/Java/C#)，项目标记文件识别 -- `ast_engine/ast_grep_engine.py`：预定义查询模板(find_function/find_class/find_imports/find_method_call)，LLM只填参数不写pattern -- `ast_engine/parser.py`：TreeSitterParser扩展支持JS/TS/Go/Rust/Java（可选依赖，graceful fallback） -- `ast_engine/graph_builder.py`：SUPPORTED_EXTENSIONS动态扩展 + rig.json新增language_stats -- `experts/verifier.py`：多语言测试运行器(pytest/jest/go test/cargo test/mvn test) + 多语言语法检查 -- `builtin_experts/golang/SKILL.md`：Go领域知识（并发/错误处理/测试） -- `builtin_experts/typescript/SKILL.md`：TS领域知识（类型系统/React/async） -- `builtin_experts/rust/SKILL.md`：Rust领域知识（所有权/错误处理/tokio） -- `builtin_experts/java/SKILL.md`：Java领域知识（Spring Boot/异常/Maven） +**SearchSubagent** (`experts/search_subagent.py`) +- Independent context window — search noise never enters Generator +- Parallel file reads: ThreadPoolExecutor, 8 concurrent +- Returns only precise {file, start_line, end_line, content} +- Shadow TaskContext: Locator writes to shadow, main ctx stays clean -**模块B: FastAPI Server + Textual TUI** -- `server/app.py`：FastAPI + SSE事件流，端口7355 - - POST /api/task → 提交任务返回task_id - - GET /api/task/{id}/events → SSE事件流 - - GET /api/health, /api/status, /api/files, /api/file - - POST /api/rig/refresh → 重建rig.json -- `server/pipeline_factory.py`：共享pipeline构建（CLI和server复用） -- `server/models.py`：Pydantic模型(TaskRequest/TaskResponse/HealthResponse等) -- `tui/app.py`：Textual TUI（左文件树+右事件流+输入框），自动启动server -- CLI新增：`kwcode serve`命令 + `kwcode --tui`选项 +**UpstreamManifest** (`core/upstream_manifest.py`) +- Deterministic AST extraction: Python ast module, regex fallback for others +- Tracks: function signatures, constants, import dependencies +- get_constraints_for_file() → injected into Generator prompt +- check_consistency() → Verifier pre-check, catches arg count / constant mismatches +- Zero LLM calls -**模块C: VSCode插件** -- `extension/src/extension.ts`：插件入口，命令注册，文件保存触发RIG刷新 -- `extension/src/server-client.ts`：SSE客户端，连接localhost:7355 -- `extension/src/panel.ts`：Webview面板，事件渲染+任务输入 -- 薄客户端架构：不重复实现业务逻辑，所有计算在server端 +**PENCIL Compression + Contract Verification** +- task_compiler: _compact_subtask_result() keeps only signatures/constants/paths/test_status +- orchestrator: locator step uses SearchSubagent, verifier pre-checks contracts +- contract_violation error type triggers re-locate retry strategy +- Generator prompt receives upstream_constraints + retry_hint -**关键设计决策** -- ast-grep pattern绝对不让LLM生成，只用QUERY_TEMPLATES预定义模板 -- Server单例pipeline，每个任务asyncio.to_thread()隔离 -- EventBus事件直接推送到SSE Queue，TUI/VSCode/CLI共享同一事件流 -- 所有新依赖都是optional（multilang/server/tui），不影响现有安装 +**Code Quality (this release)** +- orchestrator.py run() split into 5 private methods (410→253 lines) +- Full type annotations: Optional[DebugSubagent], Callable[[str,str],None] +- __all__ added to 7 core modules +- pyproject.toml: license fixed, ruff + mypy configured +- TUI: 30+ event icons added (contract_violation, ab_test, replay, etc.) +- Server: /api/manifest endpoint, version 1.5.0 +### v1.4.0 — Multi-Language + TUI + IDE -理论来源：Dive into Claude Code(arXiv:2604.14228) + Wink(arXiv:2602.17037) + ARCS(arXiv:2504.20434) + SpecEyes(arXiv:2603.23483) + OPENDEV(arXiv:2603.05344) + Turn-Control(arXiv:2510.16786) +Theory: XRAY MCP Server + OpenCode + CodeCompass -**模块1: EventBus 统一事件总线** (`core/event_bus.py`) -- append-only 日志支持 replay/时间旅行调试 -- on/off/emit 三个核心方法，wildcard "*" 监听所有事件 -- CLI 接入追加式渲染（EVENT_ICONS 17种事件图标） +- 7-language AST (Python/JS/TS/Go/Rust/Java/C#) +- ast-grep with QUERY_TEMPLATES (LLM never writes patterns) +- FastAPI server (port 7355) + SSE streaming +- Textual TUI (file tree + event log + input) +- VSCode extension (thin client, all logic server-side) -**模块2: ToolGateway 工具权限层** (`tools/tool_gateway.py`) -- 专家权限白名单（generator只读不写，verifier可写可执行） -- 文件读缓存 + 脏标记（写后自动失效缓存） -- 所有工具调用通过 EventBus 可观测 +### v1.3.0 — EventBus + Error Strategy + Cognitive Gate -**模块3: 错误策略路由** (`core/orchestrator.py` RETRY_STRATEGIES) -- 按 error_type 切换重试序列（syntax/assertion/import/patch_apply/runtime/unknown） -- import 错误：确定性修复器 `tools/import_fixer.py`（不调LLM） -- _build_retry_hint() 按错误类型生成精准重试提示 -- _should_search() 按失败类型决定是否搜网络（不再统一 retry>=2 时搜） +Theory: Dive into Claude Code + Wink + ARCS + SpecEyes + OPENDEV + Turn-Control -**模块4: 认知门控 CognitiveGate** (`core/cognitive_gate.py`) -- patch 行数持续递减 → 边际收益递减 → 自动停止 -- 连续输出相同行数 → 原地打转 → 自动停止 -- 最后一次极小(≤3行) → 模型无从下手 → 自动停止 +- EventBus: append-only log, replay, wildcard listeners +- ToolGateway: per-expert permissions, file cache with dirty tracking +- Error strategy routing: 6 error types → different retry sequences +- CognitiveGate: diminishing returns detection → auto-stop +- GraduatedCompactor: 3-layer progressive context compression +- Plan auto-trigger for hard tasks +- Worktree isolation (git worktree / tempdir fallback) +- Speculative Prefetch: background file pre-read +- SearchRouter: intent-aware routing (arXiv/S2/GitHub/PyPI/Open-Meteo) +- Wink self-repair: scope_creep / repetitive_fix / patch_miss / empty_output -**模块5: 上下文渐进压缩 GraduatedCompactor** (`core/context_pruner.py`) -- Layer 1 (70%): 裁剪 tool 输出冗余（>500 token 提取关键词） -- Layer 2 (85%): 复用 ContextPruner 压缩中间轮次 -- Layer 3 (95%): 摘要化早期对话，只保留关键决策 +### v1.2.0 — RIG Project Map -**模块6: Plan 自动触发** (`core/orchestrator.py`) -- hard 任务自动生成执行计划（不打断用户） -- 低中风险直接执行，高风险暂停确认 +Theory: RIG + FastCode + CodeCompass -**模块7: Worktree 隔离** (`core/task_compiler.py` WorktreeManager) -- Git 项目：git worktree 隔离 -- 非 Git 项目：tempdir + copytree -- cleanup() 支持 merge 回主分支 +- export_rig(): full project index (exports/imports/routes/test coverage) +- upstream_summary structured dict for multi-task context passing +- ConsistencyChecker: deterministic frontend/backend API mismatch detection +- Gate/Locator prompt explicitly guided to query rig.json -**模块8: Speculative Prefetch** (`experts/locator.py`) -- Locator 完成后后台线程预读文件到内存 -- 减少 Generator 阶段 IO 等待 +### v1.1.0 — P0+P1+P2 Optimizations -**模块9: SearchRouter 意图感知搜索** (`search/search_router.py`) -- 零 key 默认可用：arXiv / Semantic Scholar / GitHub REST / PyPI JSON / Open-Meteo -- 按意图路由：research/code_solution/code_example/weather/library_doc/general -- 可选 Tavily key 提升通用搜索质量 -- 错误驱动搜索接入 orchestrator +- Verifier structured output (_classify_error: 5 error types) +- Circuit breakers (syntax 1x, import immediate, same-type 3x streak) +- Gate confidence scoring (0.92/0.75/0.55) +- Experience Replay (BM25 similar trajectory lookup) +- Session continuity (SessionState, 5-turn KWCODE.md re-injection) +- Locator minimal context (function boundaries, 60-line cap, gap markers) +- Watchdog 300s timeout +- Gate accuracy stats (/stats command) -**模块10: Wink 自修复监控** (`core/wink.py`) -- scope_creep: easy任务定位>5文件 → 纠正 -- repetitive_fix: 同类错误≥2次 → 换思路 -- patch_miss: patch_apply失败 → 重新读文件 -- empty_output: Generator无输出 → 简化任务 +### v0.9.0 — DAG Compiler + Debug Subagent -**搜索层网络保护补丁** -- duckduckgo.py: DDG为主，SearXNG为可选增强，不自动拉起Docker -- search_augmentor.py: 全局 try/except 保护，任何网络问题返回空 -- orchestrator.py: 搜索结果为空不阻塞，异常不中断重试流程 -- config.yaml: search_enabled 开关（环境变量 KWCODE_SEARCH_ENABLED） -- 内网/离线用户设置 false 永远不触发网络请求 +- TaskCompiler: DAG scheduler, ThreadPoolExecutor + Kahn topological sort +- Debug Subagent: sys.settrace variable capture on failure +- Prompt Optimizer: trajectory → experience rules → YAML system_prompt +- Cross-Encoder search reranking -### v1.2.0 新增：RIG侦察层（Project Map） +### Core Pipeline (v0.5.0–v0.8.0) -理论来源：RIG(arXiv:2601.10112) + FastCode(arXiv:2603.01012) + CodeCompass工具采用率研究 - -**RIG-1: export_rig() 仓库结构索引** (`ast_engine/graph_builder.py`) -- 扫描全项目Python文件提取exports/imports（regex，零LLM） -- 检测Flask/FastAPI路由装饰器 → api_routes -- 匹配test_foo.py → foo.py → test_coverage -- 扫描.js/.ts文件检测axios/fetch调用 → frontend_api_calls -- 双文件输出：.kaiwu/rig.json（完整索引）+ .kaiwu/rig_summary.json（精简骨架<5KB） -- 精简骨架动态截断文件列表，保证注入Gate/Locator不爆context - -**RIG-2: upstream_summary结构化** (`core/context.py` + `core/task_compiler.py`) -- upstream_summary从str改为dict: {modified_files, diffs, new_symbols, broken_interfaces} -- TaskCompiler自动提取上游patch的文件/diff/新符号，结构化传递给下游子任务 -- 新增_format_upstream_text()将结构化dict转为LLM可读文本注入prompt -- 下游子任务Locator可直接读取"哪些文件被改了"，不靠模型猜 - -**RIG-3: ConsistencyChecker前后端一致性检查** (`experts/consistency_checker.py`) -- 基于rig.json做确定性集合对比（不调LLM） -- 输出backend_only/frontend_only/matched不一致清单 -- check_with_details()带文件位置信息，可直接作为子任务输入 -- format_for_subtask()生成可注入Generator的文本 - -**RIG-4: Gate/Locator prompt显式引导查rig** (`core/gate.py` + `experts/locator.py`) -- Gate prompt新增显式引导："优先参考.kaiwu/rig.json理解项目结构" -- Locator prompt新增{rig_context}占位符 -- _load_rig_context()读取rig_summary.json（不是完整rig.json），注入路由/前端调用/测试覆盖摘要 -- 解决CodeCompass发现的工具采用率42%问题：模型不查图的根因是prompt没引导 - -### v1.1.0 新增：P0+P1+P2 全量优化 - -**P2-1: Watchdog任务超时** (`core/orchestrator.py`) -- threading.Timer 300秒硬超时，卡死任务自动终止 -- 重试循环每轮检查watchdog状态 - -**P2-2: Gate路由准确率统计** (`stats/value_tracker.py` + `cli/main.py`) -- `get_gate_accuracy()` 按expert_type统计成功率/平均耗时/平均重试 -- 新增 `/stats` CLI命令展示30天统计报告 - -**其他** -- 新增 CONTRIBUTING.md（架构红线、PR标准、贡献类型指南） -- README 贡献章节重写（精简+链接CONTRIBUTING.md） -- 删除 vision_expert.py.bak（vision_expert保留为辅助输入通道） +- Gate → 6 pipeline routes (locator_repair/codegen/refactor/doc/office/chat) +- BM25+AST call graph two-phase location (zero LLM, milliseconds) +- Generator: original from file, LLM only generates modified +- Verifier: syntax check + pytest +- 3-stage retry + Reflection root cause analysis +- 5 deterministic tools (read_file/write_file/run_bash/list_dir/git) +- KWCODE.md project rules + /plan + Checkpoint + DocReader +- Model capability tiers (SMALL/MEDIUM/LARGE) +- Expert flywheel (trajectory → pattern → backtest → AB test → production) +- 3-layer memory (PROJECT.md/EXPERT.md/PATTERN.md) +- Office document generation (Excel/PPT/Word) +- MCP Router, context compression, CJK BM25 --- -#### P0+P1 优化（7文件 +362行） +## Test Summary -**P0-1: Verifier结构化输出** (`experts/verifier.py`) -- `_classify_error()` 纯正则提取 error_type/error_file/error_line/error_message/failed_tests -- 5种错误类型：syntax/assertion/import/runtime/patch_apply -- 所有错误路径统一返回结构化字段，DebugSubagent不再需要自己解析 - -**P0-2: 熔断器+缩小scope** (`core/orchestrator.py`) -- syntax错误1次后直接熔断（重试无意义） -- import错误立即熔断+提示安装依赖 -- 同类error_type连续3次→硬熔断 -- 第2次失败自动缩小scope到第一个文件+函数 -- 低置信度(<0.6)任务自动减少重试预算 - -**P0-3: Gate置信度输出** (`core/gate.py`) -- `_estimate_confidence()` 关键词信号强度评分（0.92/0.75/0.55三档） -- 不覆盖expert_registry已有的confidence -- orchestrator消费：低置信度减少max_retries - -**P1-1: Experience Replay** (`flywheel/trajectory_collector.py` + `core/orchestrator.py` + `core/context.py`) -- `find_similar()` BM25检索历史成功轨迹（复用已有rank-bm25依赖） -- orchestrator.run()开头自动调用，结果存入ctx.similar_trajectories -- 飞轮闭环：同类任务不走冷启动 - -**P1-2: Session内多轮连贯** (`cli/main.py`) -- SessionState类：跟踪tasks/files_touched/turn_count -- `to_reminder()` 生成System Reminder注入Gate memory_context -- 每5轮重新注入KWCODE.md核心规则（注意力衰减对抗） - -**P1-3: Locator最小上下文裁剪** (`experts/locator.py`) -- 函数边界识别（indent-based，def到下一个同级def） -- 去掉纯注释行，docstring限3行 -- 60行/函数上限，gap marker标记不连续区域 -- 文件路径header + 行号前缀 - -### v0.9.0 新增 - -**DAG 任务编译器（TaskCompiler）** -- `core/task_compiler.py`：轻量 DAG 调度器，ThreadPoolExecutor + Kahn 拓扑排序 -- 支持串行（依赖链）和并行（独立任务）混合执行 -- 依赖上下文注入：前置任务结果自动追加到后续任务输入 -- 零新依赖，12 个测试覆盖（串行/并行/菱形DAG/环检测） - -**Debug Subagent（运行时调试子代理）** -- `experts/debug_subagent.py`：基于 Debug2Fix 论文（Microsoft 2026） -- verifier 失败后用 sys.settrace 非侵入式捕获目标行变量值 -- LLM 决定调试策略（断点位置+变量列表），fallback 到 pytest --tb=long -- 调试结果注入 generator retry prompt，让重试拿到真实运行时数据 -- 15 个测试覆盖 - -**Prompt Optimizer（飞轮优化 YAML system_prompt）** -- `flywheel/prompt_optimizer.py`：分析成功轨迹 → Opus/Sonnet API 生成经验规则 -- 规则追加到专家 YAML 的 system_prompt（`## 经验规则（自动生成）`） -- 替代已删除的 Python 代码优化器，方向正确：优化知识而非代码 - -**Reflexion 持久化**（保留） -- REFLECTION.md 结构化写入 + /plan 风险提示注入 - -**Cross-Encoder 搜索重排**（保留） -- BM25 后追加 Cross-Encoder 精排，FLEX-2 自动降级 - -### v0.8.0 已移除（方向错误） - -- ~~ExpertBase 继承体系~~（Python专家把知识和执行逻辑混在一起） -- ~~BugFixExpert.py~~（应该是 YAML 知识载体，不是 Python 类） -- ~~SelfImprovingOptimizer~~（优化 Python 代码 → 改为优化 YAML prompt） -- ~~Registry Python 专家加载~~（回退到纯 YAML） - -### 已完成功能清单 - -**MVP 核心流水线** -- Gate LLM分类 → 6种流水线路由（locator_repair/codegen/refactor/doc/office/chat） -- BM25+AST调用图两阶段定位（零LLM，毫秒级） -- Generator 从文件读original，LLM只生成modified -- Verifier 语法检查 + pytest 自动验证 -- 三阶段重试（正常→从错误出发→最小化修改）+ Reflection根因分析 -- 5个确定性工具（read_file/write_file/run_bash/list_dir/git） - -**P1 四大功能（v0.5.0）** -- KWCODE.md 项目规则文件（分段加载+按任务类型注入+token上限15%） -- /plan 计划模式 + 三档风险评估（High/Medium/Low，基于历史失败记录） -- Checkpoint 文件快照（git stash主路径+文件复制兜底+失败自动还原+降级建议） -- 非代码文件读取（PDF/Word/MD/TXT + BM25Plus段落匹配 + Locator自动注入） - -**P2 三大功能（v0.6.0）** -- 模型能力自适应（SMALL/MEDIUM/LARGE三档策略，自动检测，小模型强制plan） -- 飞轮可见性通知（专家投产Panel+积累进度+里程碑，不打断任务） -- 价值量化仪表盘（SQLite本地统计+kwcode stats命令+启动周报） - -**搜索模块重构（v0.6.1）** -- 四级内容提取管道（trafilatura→newspaper→readabilipy→soup，质量评分选最佳） -- SearXNG + DDG 并行搜索（ThreadPoolExecutor + URL去重合并） -- kwcode setup-search 一键安装SearXNG - -**意图感知搜索（v0.6.2）** -- 意图分类器增强（5类意图+关键词扩充+LLM fallback语义分类） -- ChatExpert搜索门控（follow-up/推理不搜索，实时数据始终搜索） -- QueryGenerator按意图生成更精准搜索词 -- BM25Plus搜索结果重排 - -**UI全面优化（v0.7.0）** -- 删掉所有机器内部信息（logger只写~/.kwcode/kwcode.log，warnings静默） -- 执行过程spinner动画（rich.progress，transient=True完成后消失） -- 完成后用户友好结果摘要（修改文件+改动bullet+测试结果） -- 重影大字KAIWU Header + 状态栏深色背景 -- kwcode setup-search 一键安装SearXNG搜索引擎 - -**其他** -- 专家注册表（15个预置专家YAML + .kwx导入导出） -- 专家飞轮三道门（轨迹→模式检测→回测→AB测试→投产） -- 3层记忆系统（PROJECT.md/EXPERT.md/PATTERN.md） -- Office文档生成（Excel/PPT/Word） -- MCP Router（kwcode serve-mcp） -- 上下文压缩（纯算法，头尾保留+中间关键词提取，<10ms） -- 中文分词BM25（DocReader CJK tokenizer） -- /api 命令（临时/永久切换任意OpenAI兼容API） - -### E2E 验收结果（2026-04-28，gemma3:4b） - -- P1: KWCODE.md注入✓ /plan风险评估✓ Checkpoint还原✓ DocReader注入✓ (8/8) -- P2: 模型自适应(4b→SMALL)✓ 飞轮通知✓ ValueTracker✓ (6/6) -- 集成: Gate分类✓ Chat流水线✓ Codegen流水线(2.9s)✓ (3/3) - -### 测试统计 - -| 类别 | 数量 | 状态 | -|------|------|------| -| 核心单元测试 | 38 | PASS | -| 回归测试 | 173 | PASS | -| P1 功能测试 | 33 | PASS | -| P2 功能测试 | 21 | PASS | -| 搜索重构测试 | 19 | PASS | -| 意图搜索测试 | 19 | PASS | -| E2E 真实模型 | 17 | PASS | -| RIG模块测试 | 29 | PASS | -| TaskCompiler测试 | 12 | PASS | -| 多语言模块测试 | 51 | PASS | -| Server/TUI测试 | 16 | PASS | -| **合计** | **424** | **全绿** | - -### 待做 - -1. SQLite 跨 session 查询（spec §7.1 kaiwu.db） -2. 12 个预置专家完整 benchmark（目前只跑了 BugFix+TestGen） -3. 实时数据API提示注入（codegen涉及天气/股价时，prompt注入免费API信息，避免模型编造假数据） -4. ~~多语言AST支持~~ ✅ v1.4.0 已完成 -5. pip publish 到 PyPI -6. install.ps1 / install.sh 一键安装脚本 - -### 已知问题 - -- qwen3-vl:8b 所有输出在thinking字段，content为空（已加thinking提取） -- reasoning模型Gate调用慢（8x multiplier） -- SearXNG需要Docker Desktop，无Docker时降级到DDG -- codegen生成网页时模型可能用Math.random()假数据（需实时数据API提示注入） +| Category | Count | Status | +|----------|-------|--------| +| Core unit tests | 38 | PASS | +| Regression tests | 173 | PASS | +| P1 feature tests | 33 | PASS | +| P2 feature tests | 21 | PASS | +| Search refactor | 19 | PASS | +| Intent search | 19 | PASS | +| E2E real model | 17 | PASS | +| RIG modules | 29 | PASS | +| TaskCompiler | 12 | PASS | +| Multi-language | 51 | PASS | +| Server/TUI | 16 | PASS | +| SearchSubagent+Manifest | 27 | PASS | +| **Total** | **451** | **All green** | --- -## 文件结构 +## File Structure ``` kwcode/ @@ -324,49 +140,64 @@ kwcode/ ├── STATUS.md └── kaiwu/ ├── cli/ - │ ├── main.py # REPL + EventBus追加式渲染 + spinner + 结果摘要 - │ ├── status_bar.py # 状态栏(4档自适应) + TokPerSecEstimator - │ └── onboarding.py # 首次启动引导 + │ ├── main.py # REPL + EventBus rendering + spinner + summary + │ ├── status_bar.py # Status bar (4-tier adaptive) + │ └── onboarding.py # First-run onboarding ├── core/ - │ ├── event_bus.py # [v1.3] 统一事件总线(append-only日志+replay) - │ ├── cognitive_gate.py # [v1.3] 认知门控(patch行数递减检测) - │ ├── wink.py # [v1.3] Wink自修复监控(偏离检测+纠正注入) - │ ├── gate.py # LLM任务分类 → 专家知识叠加 - │ ├── orchestrator.py # 确定性流水线 + 错误策略路由 + Plan自动触发 - │ ├── context.py # TaskContext数据类 - │ ├── task_compiler.py # DAG调度器 + WorktreeManager隔离 - │ ├── planner.py # /plan计划模式 + 风险评估 - │ ├── checkpoint.py # 文件快照(git stash/文件复制) - │ ├── kwcode_md.py # KWCODE.md分段加载+注入 - │ ├── model_capability.py # 模型三档自适应(SMALL/MEDIUM/LARGE) - │ ├── context_pruner.py # 上下文压缩 + GraduatedCompactor 3层渐进压缩 - │ ├── network.py # 网络探测+代理配置 - │ └── sysinfo.py # 系统信息+VRAM监控 + │ ├── event_bus.py # Unified event bus (append-only + replay) + │ ├── cognitive_gate.py # Diminishing returns detection + │ ├── wink.py # Self-repair monitor + │ ├── gate.py # LLM task classification → expert routing + │ ├── orchestrator.py # Deterministic pipeline + error strategy routing + │ ├── context.py # TaskContext dataclass + │ ├── task_compiler.py # DAG scheduler + WorktreeManager + │ ├── upstream_manifest.py # [v1.5] Cross-file contract tracking (zero LLM) + │ ├── planner.py # /plan mode + risk assessment + │ ├── checkpoint.py # File snapshot (git stash / file copy) + │ ├── kwcode_md.py # KWCODE.md segmented loading + │ ├── model_capability.py # Model tier detection (SMALL/MEDIUM/LARGE) + │ ├── context_pruner.py # Context compression + GraduatedCompactor + │ ├── network.py # Network detection + proxy config + │ └── sysinfo.py # System info + VRAM monitoring ├── experts/ - │ ├── locator.py # BM25+调用图定位 + DocReader注入 + Speculative Prefetch - │ ├── generator.py # 代码生成(original从文件读，LLM只写modified) - │ ├── verifier.py # 语法检查 + pytest - │ ├── search_augmentor.py # 搜索增强 + BM25重排 + 网络保护 - │ ├── consistency_checker.py # 前后端接口一致性检查(确定性，不调LLM) - │ ├── chat_expert.py # 聊天(搜索门控：follow-up/推理不搜) - │ └── office_handler.py # Office文档生成 - ├── search/ - │ ├── search_router.py # [v1.3] 意图感知搜索路由(arxiv/S2/GitHub/PyPI/Open-Meteo) - │ ├── duckduckgo.py # DDG主+SearXNG可选 + search_enabled开关 - │ ├── extraction_pipeline.py # 四级内容提取 - │ ├── intent_classifier.py # 意图感知(5类+LLM fallback) - │ ├── query_generator.py # 按意图生成搜索词 - │ ├── content_fetcher.py # 薄封装→extraction_pipeline - │ └── quality_filter.py # 域名黑白名单 - ├── knowledge/doc_reader.py # PDF/Word/MD读取 + CJK分词BM25 - ├── flywheel/ # 轨迹→模式→生成→AB测试→投产 - ├── registry/ # 专家注册表 + .kwx打包 - ├── notification/ # 飞轮通知(expert_born/progress/milestone) - ├── stats/ # 价值量化(SQLite) - ├── memory/ # 三层记忆(PROJECT/EXPERT/PATTERN) - ├── ast_engine/ # tree-sitter AST + 调用图(SQLite) + │ ├── locator.py # BM25+graph location + DocReader + Prefetch + │ ├── search_subagent.py # [v1.5] Isolated search (independent context) + │ ├── generator.py # Code generation (original from file, LLM writes modified) + │ ├── verifier.py # Syntax + pytest + cross-file contract check + │ ├── search_augmentor.py # Search augmentation + BM25 rerank + │ ├── consistency_checker.py # Frontend/backend API consistency (deterministic) + │ ├── chat_expert.py # Chat (search gating) + │ └── office_handler.py # Office document generation + ├── search/ # Intent-aware search routing + ├── knowledge/ # PDF/Word/MD reader + CJK BM25 + ├── flywheel/ # Trajectory → pattern → generation → AB test + ├── registry/ # Expert registry + .kwx packaging + ├── notification/ # Flywheel notifications + ├── stats/ # Value tracking (SQLite) + ├── memory/ # 3-layer memory system + ├── ast_engine/ # tree-sitter AST + call graph + ├── server/ # FastAPI + SSE (port 7355) + ├── tui/ # Textual TUI ├── mcp/ # MCP Router - ├── llm/ # Ollama + llama.cpp双后端 - ├── tools/ # 5个确定性工具 + ToolGateway + import_fixer - └── tests/ # 357个测试 + ├── llm/ # Ollama + llama.cpp backends + ├── tools/ # 5 deterministic tools + ToolGateway + └── tests/ # 451 tests ``` + +--- + +## TODO + +1. CLI refactor: split main.py (1861 lines) into cli/commands/ +2. Comments: unify to English across all modules +3. Expert-level EventBus emit (file reads, function locations, test results) +4. SQLite cross-session queries +5. Full expert benchmark (12 presets) +6. pip publish to PyPI +7. install.ps1 / install.sh one-click install + +## Known Issues + +- qwen3-vl:8b outputs in thinking field, content empty (thinking extraction added) +- Reasoning models slow on Gate (8x multiplier) +- SearXNG requires Docker Desktop, degrades to DDG without it diff --git a/kaiwu/core/context.py b/kaiwu/core/context.py index 8c84e69..b20d295 100644 --- a/kaiwu/core/context.py +++ b/kaiwu/core/context.py @@ -6,6 +6,8 @@ Each expert reads from and writes to specific fields only. from dataclasses import dataclass, field from typing import Optional +__all__ = ["TaskContext"] + @dataclass class TaskContext: @@ -71,3 +73,9 @@ class TaskContext: # Experience Replay: similar successful trajectories from history similar_trajectories: list = field(default_factory=list) + + # SearchSubagent: cross-file constraints injected into Generator prompt + upstream_constraints: str = "" + + # Retry hint: error-specific guidance injected into Generator prompt + retry_hint: str = "" diff --git a/kaiwu/core/event_bus.py b/kaiwu/core/event_bus.py index 87ef600..799d728 100644 --- a/kaiwu/core/event_bus.py +++ b/kaiwu/core/event_bus.py @@ -15,6 +15,8 @@ import logging logger = logging.getLogger(__name__) +__all__ = ["EventBus"] + class EventBus: """ diff --git a/kaiwu/core/gate.py b/kaiwu/core/gate.py index 1285670..824d0c9 100644 --- a/kaiwu/core/gate.py +++ b/kaiwu/core/gate.py @@ -16,6 +16,8 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +__all__ = ["Gate"] + GATE_SYSTEM = "你是任务分类器。只返回JSON，不要有其他内容。" GATE_PROMPT = """分析用户输入，返回分类JSON。 diff --git a/kaiwu/core/orchestrator.py b/kaiwu/core/orchestrator.py index c74aeff..8abcdd1 100644 --- a/kaiwu/core/orchestrator.py +++ b/kaiwu/core/orchestrator.py @@ -7,7 +7,11 @@ RED-5: Max 3 retries, hardcoded. import logging import time import threading -from typing import Optional +from typing import Callable, Optional, TYPE_CHECKING + +if TYPE_CHECKING: + from kaiwu.experts.debug_subagent import DebugSubagent + from kaiwu.experts.vision_expert import VisionExpert from kaiwu.core.context import TaskContext from kaiwu.core.event_bus import EventBus @@ -24,6 +28,8 @@ from kaiwu.memory.kaiwu_md import KaiwuMemory from kaiwu.registry.expert_registry import ExpertRegistry from kaiwu.tools.executor import ToolExecutor from kaiwu.flywheel.trajectory_collector import TrajectoryCollector + +__all__ = ["PipelineOrchestrator"] from kaiwu.flywheel.pattern_detector import PatternDetector from kaiwu.flywheel.ab_tester import ABTester from kaiwu.core.checkpoint import Checkpoint @@ -97,13 +103,13 @@ class PipelineOrchestrator: office_handler: OfficeHandlerExpert, tool_executor: ToolExecutor, memory: KaiwuMemory, - registry: ExpertRegistry | None = None, - trajectory_collector: TrajectoryCollector | None = None, - ab_tester: ABTester | None = None, - chat_expert: ChatExpert | None = None, - debug_subagent=None, - vision_expert=None, - bus: EventBus | None = None, + registry: Optional[ExpertRegistry] = None, + trajectory_collector: Optional[TrajectoryCollector] = None, + ab_tester: Optional[ABTester] = None, + chat_expert: Optional[ChatExpert] = None, + debug_subagent: Optional["DebugSubagent"] = None, + vision_expert: Optional["VisionExpert"] = None, + bus: Optional[EventBus] = None, ): self.locator = locator self.generator = generator @@ -124,9 +130,7 @@ class PipelineOrchestrator: self.bus = bus or EventBus() self._wink = WinkMonitor() self._cognitive_gate = CognitiveGate() - # SearchSubagent: isolated context window for code search self._search_subagent = SearchSubagent(locator, tool_executor) - # UpstreamManifest: cross-file contract tracking (per-task lifecycle) self._manifest = UpstreamManifest() def run( @@ -134,11 +138,11 @@ class PipelineOrchestrator: user_input: str, gate_result: dict, project_root: str, - on_status=None, + on_status: "Optional[Callable[[str, str], None]]" = None, no_search: bool = False, skip_checkpoint: bool = False, pre_search_results: str = "", - image_paths: list = None, + image_paths: Optional[list[str]] = None, ) -> dict: """ Execute the expert pipeline. @@ -169,6 +173,9 @@ class PipelineOrchestrator: expert_system_prompt=gate_result.get("system_prompt", ""), ) + # Reset manifest for each new top-level task + self._manifest.clear() + # 处理图片路径 if image_paths: ctx.image_paths = list(image_paths) @@ -194,78 +201,13 @@ class PipelineOrchestrator: else: ctx.expert_system_prompt = kwcode_rules - # chat类型：直接回复，不走AB测试/搜索/重试 - if expert_type == "chat": - self._emit(on_status, "chat", "聊天模式") - if self.chat_expert: - result = self.chat_expert.run(ctx) - else: - ctx.generator_output = {"explanation": "我是KWCode，专注于代码任务。", "patches": []} - result = {"passed": True} - elapsed = time.time() - start_time - return { - "success": True, - "context": ctx, - "error": None, - "elapsed": elapsed, - } + # chat/vision类型：早期返回 + simple_result = self._handle_simple_type(ctx, expert_type, start_time, on_status) + if simple_result is not None: + return simple_result - # vision类型：图片处理任务 - if expert_type == "vision": - self._emit(on_status, "vision", "图片处理模式") - if self.vision_expert and ctx.image_paths: - result = self.vision_expert.run(ctx) - explanation = result.get("output", "").strip() - ctx.generator_output = { - "explanation": explanation, - "patches": [], - "metadata": {"vision": result.get("metadata", {})}, - } - elapsed = time.time() - start_time - success = result.get("success", False) - return { - "success": success, - "context": ctx, - "error": None if success else explanation or "图片处理失败", - "elapsed": elapsed, - } - else: - ctx.generator_output = {"explanation": "图片处理功能需要配置Vision专家", "patches": []} - elapsed = time.time() - start_time - return { - "success": False, - "context": ctx, - "error": "Vision专家未配置或未提供图片", - "elapsed": elapsed, - } - - # Gate 3: AB test — check if a candidate expert should be used for this task - ab_candidate_name = None - ab_used_new = False - if self.ab_tester and expert_type != "chat": - candidate_def = self.ab_tester.should_use_candidate(expert_type) - if candidate_def: - ab_candidate_name = candidate_def["name"] - ab_used_new = True - # Override gate_result to use the candidate expert's pipeline - gate_result = { - **gate_result, - "expert_name": ab_candidate_name, - "route_type": "expert_registry", - "pipeline": candidate_def.get("pipeline", []), - "system_prompt": candidate_def.get("system_prompt", ""), - } - self._emit(on_status, "ab_test", f"AB测试：使用候选专家 {ab_candidate_name}") - else: - # Check if any candidate is in AB testing for this type (baseline run) - for name, info in self.ab_tester._candidates.items(): - if (info["status"] == "ab_testing" - and info["expert_def"].get("type") == expert_type - and len(info["ab_results"]) < 10): - ab_candidate_name = name - ab_used_new = False - self._emit(on_status, "ab_test", f"AB测试：基线对照（候选 {name}）") - break + # Gate 3: AB test + ab_candidate_name, ab_used_new, gate_result = self._setup_ab_test(gate_result, expert_type, on_status) # Use custom pipeline from expert registry if available, else default if gate_result.get("route_type") == "expert_registry" and "pipeline" in gate_result: @@ -275,49 +217,8 @@ class PipelineOrchestrator: self._emit(on_status, "gate", f"任务类型：{expert_type} | 难度：{gate_result.get('difficulty', '?')}") - # ── Experience Replay: find similar successful trajectories ── - if self.trajectory_collector and expert_type not in ("chat", "office", "vision"): - try: - similar = self.trajectory_collector.find_similar(user_input, expert_type, k=3) - if similar: - ctx.similar_trajectories = similar - best = similar[0] - self._emit(on_status, "replay", - f"发现相似成功案例：{best.get('user_input', '')[:40]}") - except Exception as e: - logger.debug("Experience replay failed (non-blocking): %s", e) - - # codegen任务如果涉及实时数据，首次就触发搜索（不等失败重试） - if expert_type == "codegen" and not no_search and self._needs_realtime_data(user_input): - try: - self._emit(on_status, "search", "检测到实时数据需求，预搜索...") - results = self.search_augmentor.search(ctx) - if results: - ctx.search_results = results - ctx.search_triggered = True - self._emit(on_status, "search_done", f"搜索完成，注入{len(results)}字参考信息") - except Exception as e: - logger.debug("Pre-search failed (网络保护，不阻塞): %s", e) - - # ── Plan 自动触发：hard 任务自动生成计划（不打断用户）── - if (gate_result.get("difficulty") == "hard" - and expert_type not in ("chat", "office", "vision") - and not ctx.subtask_results): - try: - from kaiwu.core.planner import Planner - from kaiwu.memory import pattern_md - planner = Planner( - locator=self.locator, - pattern_md_module=pattern_md, - llm=self.generator.llm, - ) - plan = planner.generate_plan_steps(user_input, gate_result, project_root) - if plan and len(plan) > 1: - ctx.execution_plan = plan - self._emit(on_status, "plan_generated", f"自动生成 {len(plan)} 步计划") - self.bus.emit("plan_generated", {"steps": len(plan), "msg": f"生成 {len(plan)} 步计划"}) - except Exception as e: - logger.debug("Auto-plan failed (non-blocking): %s", e) + # Experience replay + pre-search + plan + self._prepare_context(ctx, gate_result, expert_type, user_input, project_root, no_search, on_status) # ── Checkpoint: snapshot before execution (skip in multi-task to avoid race) ── checkpoint = Checkpoint(project_root) @@ -354,33 +255,9 @@ class PipelineOrchestrator: if success: elapsed = time.time() - start_time - checkpoint.discard() # Clean up snapshot on success - - # Reviewer: 需求对齐审查（非阻塞，不影响成功判定） - review_result = self._do_review(ctx, on_status) - - # Save to memory on success (with elapsed for expert/pattern tracking) - self.memory.save(project_root, ctx, elapsed=elapsed) - # Update expert registry stats - expert_name = gate_result.get("expert_name") - if expert_name and self.registry: - self.registry.update_stats(expert_name, success=True, latency=elapsed) - # Flywheel: record trajectory and detect patterns (non-blocking) - self._record_trajectory(ctx, True, elapsed, on_status) - # Gate 3: record AB test result if this task is part of an AB test - self._record_ab_result(ab_candidate_name, ab_used_new, True, elapsed, on_status) - # P2: Value tracking (local SQLite) - self._record_value(project_root, gate_result, True, elapsed, ctx) - # P2: Milestone check - self._check_milestone(on_status) - # Reflexion持久化：成功时也记录注意事项 - self._persist_reflection(project_root, ctx, gate_result, success=True) - return { - "success": True, - "context": ctx, - "error": None, - "elapsed": elapsed, - } + return self._record_success(ctx, project_root, gate_result, + ab_candidate_name, ab_used_new, elapsed, + checkpoint, on_status) ctx.retry_count += 1 error_detail = "" @@ -506,6 +383,173 @@ class PipelineOrchestrator: _watchdog.cancel() # Clean up watchdog timer elapsed = time.time() - start_time + return self._record_failure_result(ctx, project_root, gate_result, + ab_candidate_name, ab_used_new, max_retries, + elapsed, checkpoint, checkpoint_saved, on_status) + + def _handle_simple_type(self, ctx: TaskContext, expert_type: str, start_time: float, on_status) -> Optional[dict]: + """Handle chat and vision early returns. Returns result dict or None to continue.""" + # chat类型：直接回复，不走AB测试/搜索/重试 + if expert_type == "chat": + self._emit(on_status, "chat", "聊天模式") + if self.chat_expert: + result = self.chat_expert.run(ctx) + else: + ctx.generator_output = {"explanation": "我是KWCode，专注于代码任务。", "patches": []} + result = {"passed": True} + elapsed = time.time() - start_time + return { + "success": True, + "context": ctx, + "error": None, + "elapsed": elapsed, + } + + # vision类型：图片处理任务 + if expert_type == "vision": + self._emit(on_status, "vision", "图片处理模式") + if self.vision_expert and ctx.image_paths: + result = self.vision_expert.run(ctx) + explanation = result.get("output", "").strip() + ctx.generator_output = { + "explanation": explanation, + "patches": [], + "metadata": {"vision": result.get("metadata", {})}, + } + elapsed = time.time() - start_time + success = result.get("success", False) + return { + "success": success, + "context": ctx, + "error": None if success else explanation or "图片处理失败", + "elapsed": elapsed, + } + else: + ctx.generator_output = {"explanation": "图片处理功能需要配置Vision专家", "patches": []} + elapsed = time.time() - start_time + return { + "success": False, + "context": ctx, + "error": "Vision专家未配置或未提供图片", + "elapsed": elapsed, + } + + return None + + def _setup_ab_test(self, gate_result: dict, expert_type: str, on_status) -> tuple: + """Setup AB test. Returns (ab_candidate_name, ab_used_new, gate_result).""" + ab_candidate_name = None + ab_used_new = False + if self.ab_tester and expert_type != "chat": + candidate_def = self.ab_tester.should_use_candidate(expert_type) + if candidate_def: + ab_candidate_name = candidate_def["name"] + ab_used_new = True + # Override gate_result to use the candidate expert's pipeline + gate_result = { + **gate_result, + "expert_name": ab_candidate_name, + "route_type": "expert_registry", + "pipeline": candidate_def.get("pipeline", []), + "system_prompt": candidate_def.get("system_prompt", ""), + } + self._emit(on_status, "ab_test", f"AB测试：使用候选专家 {ab_candidate_name}") + else: + # Check if any candidate is in AB testing for this type (baseline run) + for name, info in self.ab_tester._candidates.items(): + if (info["status"] == "ab_testing" + and info["expert_def"].get("type") == expert_type + and len(info["ab_results"]) < 10): + ab_candidate_name = name + ab_used_new = False + self._emit(on_status, "ab_test", f"AB测试：基线对照（候选 {name}）") + break + return (ab_candidate_name, ab_used_new, gate_result) + + def _prepare_context(self, ctx: TaskContext, gate_result: dict, expert_type: str, + user_input: str, project_root: str, no_search: bool, on_status) -> None: + """Experience replay + pre-search + plan generation.""" + # ── Experience Replay: find similar successful trajectories ── + if self.trajectory_collector and expert_type not in ("chat", "office", "vision"): + try: + similar = self.trajectory_collector.find_similar(user_input, expert_type, k=3) + if similar: + ctx.similar_trajectories = similar + best = similar[0] + self._emit(on_status, "replay", + f"发现相似成功案例：{best.get('user_input', '')[:40]}") + except Exception as e: + logger.debug("Experience replay failed (non-blocking): %s", e) + + # codegen任务如果涉及实时数据，首次就触发搜索（不等失败重试） + if expert_type == "codegen" and not no_search and self._needs_realtime_data(user_input): + try: + self._emit(on_status, "search", "检测到实时数据需求，预搜索...") + results = self.search_augmentor.search(ctx) + if results: + ctx.search_results = results + ctx.search_triggered = True + self._emit(on_status, "search_done", f"搜索完成，注入{len(results)}字参考信息") + except Exception as e: + logger.debug("Pre-search failed (网络保护，不阻塞): %s", e) + + # ── Plan 自动触发：hard 任务自动生成计划（不打断用户）── + if (gate_result.get("difficulty") == "hard" + and expert_type not in ("chat", "office", "vision") + and not ctx.subtask_results): + try: + from kaiwu.core.planner import Planner + from kaiwu.memory import pattern_md + planner = Planner( + locator=self.locator, + pattern_md_module=pattern_md, + llm=self.generator.llm, + ) + plan = planner.generate_plan_steps(user_input, gate_result, project_root) + if plan and len(plan) > 1: + ctx.execution_plan = plan + self._emit(on_status, "plan_generated", f"自动生成 {len(plan)} 步计划") + self.bus.emit("plan_generated", {"steps": len(plan), "msg": f"生成 {len(plan)} 步计划"}) + except Exception as e: + logger.debug("Auto-plan failed (non-blocking): %s", e) + + def _record_success(self, ctx: TaskContext, project_root: str, gate_result: dict, + ab_candidate_name, ab_used_new: bool, elapsed: float, + checkpoint, on_status) -> dict: + """Record success: memory, registry, trajectory, AB, value, milestone, reflection.""" + checkpoint.discard() # Clean up snapshot on success + + # Reviewer: 需求对齐审查（非阻塞，不影响成功判定） + review_result = self._do_review(ctx, on_status) + + # Save to memory on success (with elapsed for expert/pattern tracking) + self.memory.save(project_root, ctx, elapsed=elapsed) + # Update expert registry stats + expert_name = gate_result.get("expert_name") + if expert_name and self.registry: + self.registry.update_stats(expert_name, success=True, latency=elapsed) + # Flywheel: record trajectory and detect patterns (non-blocking) + self._record_trajectory(ctx, True, elapsed, on_status) + # Gate 3: record AB test result if this task is part of an AB test + self._record_ab_result(ab_candidate_name, ab_used_new, True, elapsed, on_status) + # P2: Value tracking (local SQLite) + self._record_value(project_root, gate_result, True, elapsed, ctx) + # P2: Milestone check + self._check_milestone(on_status) + # Reflexion持久化：成功时也记录注意事项 + self._persist_reflection(project_root, ctx, gate_result, success=True) + return { + "success": True, + "context": ctx, + "error": None, + "elapsed": elapsed, + } + + def _record_failure_result(self, ctx: TaskContext, project_root: str, gate_result: dict, + ab_candidate_name, ab_used_new: bool, max_retries: int, + elapsed: float, checkpoint, checkpoint_saved: bool, + on_status) -> dict: + """Record failure: checkpoint restore, memory, registry, trajectory, AB, value, reflection.""" # ── Checkpoint: restore on failure ── if checkpoint_saved: restored = checkpoint.restore() @@ -557,10 +601,11 @@ class PipelineOrchestrator: ctx.relevant_code_snippets = search_result["code_snippets"] # Inject upstream constraints for Generator if search_result.get("upstream_constraints"): - ctx._upstream_constraints = search_result["upstream_constraints"] + ctx.upstream_constraints = search_result["upstream_constraints"] files = search_result["relevant_files"] funcs = search_result["relevant_functions"] - self._emit(on_status, "locator_done", f"文件：{', '.join(files[:3])} | 函数：{', '.join(funcs[:3])}") + func_str = ', '.join(funcs[:3]) if funcs else "（文件级修改）" + self._emit(on_status, "locator_done", f"文件：{', '.join(files[:3])} | 函数：{func_str}") elif step == "generator": self._emit(on_status, "generator", "生成patch...") diff --git a/kaiwu/core/task_compiler.py b/kaiwu/core/task_compiler.py index 6a71113..195e48b 100644 --- a/kaiwu/core/task_compiler.py +++ b/kaiwu/core/task_compiler.py @@ -20,6 +20,8 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +__all__ = ["TaskCompiler", "WorktreeManager", "CycleError"] + MAX_PARALLEL_WORKERS = 4 diff --git a/kaiwu/core/upstream_manifest.py b/kaiwu/core/upstream_manifest.py index 2fc5e99..466a712 100644 --- a/kaiwu/core/upstream_manifest.py +++ b/kaiwu/core/upstream_manifest.py @@ -16,6 +16,8 @@ from typing import Optional logger = logging.getLogger(__name__) +__all__ = ["UpstreamManifest"] + class UpstreamManifest: """ diff --git a/kaiwu/experts/generator.py b/kaiwu/experts/generator.py index a236f5e..6db305b 100644 --- a/kaiwu/experts/generator.py +++ b/kaiwu/experts/generator.py @@ -318,7 +318,7 @@ class GeneratorExpert: prompt = self._build_retry_prompt(ctx, fpath, original, task_desc, search_ctx) # Inject upstream constraints from SearchSubagent (cross-file contracts) - upstream_constraints = getattr(ctx, '_upstream_constraints', "") + upstream_constraints = ctx.upstream_constraints if upstream_constraints: prompt += f"\n\n## 跨文件契约（必须遵守）\n{upstream_constraints}" @@ -327,9 +327,8 @@ class GeneratorExpert: prompt += f"\n\n## 相关文档参考\n{ctx.doc_context[:800]}" # Inject retry_hint if available - retry_hint = getattr(ctx, 'retry_hint', "") - if retry_hint: - prompt += f"\n\n## 重试提示\n{retry_hint}" + if ctx.retry_hint: + prompt += f"\n\n## 重试提示\n{ctx.retry_hint}" system = self._build_system(ctx) diff --git a/kaiwu/experts/search_subagent.py b/kaiwu/experts/search_subagent.py index d76387d..88c3172 100644 --- a/kaiwu/experts/search_subagent.py +++ b/kaiwu/experts/search_subagent.py @@ -26,6 +26,8 @@ from kaiwu.tools.executor import ToolExecutor logger = logging.getLogger(__name__) +__all__ = ["SearchSubagent", "SearchResult"] + # Max parallel file reads per batch MAX_PARALLEL_READS = 8 # Max lines per snippet returned to Generator diff --git a/kaiwu/server/app.py b/kaiwu/server/app.py index a468c05..2006cc6 100644 --- a/kaiwu/server/app.py +++ b/kaiwu/server/app.py @@ -34,13 +34,13 @@ def create_app( from kaiwu.server.models import ( TaskRequest, TaskResponse, HealthResponse, - StatusResponse, FileContent, + StatusResponse, FileContent, ManifestResponse, ) from kaiwu.server.pipeline_factory import build_pipeline app = FastAPI( title="KwCode Server", - version="1.3.0", + version="1.5.0", description="KwCode coding agent HTTP API with SSE streaming", ) @@ -78,7 +78,7 @@ def create_app( async def health(): return HealthResponse( status="ok", - version="1.3.0", + version="1.5.0", model=ollama_model or "local", project_root=project_root, ) @@ -286,4 +286,16 @@ def create_app( except Exception as e: raise HTTPException(status_code=500, detail=str(e)) + @app.get("/api/manifest", response_model=ManifestResponse) + async def get_manifest(): + """Get current UpstreamManifest state (cross-file contracts).""" + manifest = orchestrator._manifest + sigs = manifest.get_all_signatures() + consts = manifest.get_all_constants() + return ManifestResponse( + signatures=sigs, + constants=consts, + file_count=len(sigs), + ) + return app diff --git a/kaiwu/server/models.py b/kaiwu/server/models.py index 4e846e6..94d494a 100644 --- a/kaiwu/server/models.py +++ b/kaiwu/server/models.py @@ -33,7 +33,7 @@ class TaskResult(BaseModel): class HealthResponse(BaseModel): """Health check response.""" status: str = "ok" - version: str = "1.3.0" + version: str = "1.5.0" model: str = "" project_root: str = "" @@ -61,3 +61,10 @@ class FileContent(BaseModel): content: str language: str = "" lines: int = 0 + + +class ManifestResponse(BaseModel): + """UpstreamManifest state response.""" + signatures: dict[str, dict[str, str]] = Field(default_factory=dict) + constants: dict[str, dict[str, str]] = Field(default_factory=dict) + file_count: int = 0 diff --git a/kaiwu/tests/test_search_subagent.py b/kaiwu/tests/test_search_subagent.py new file mode 100644 index 0000000..eb42036 --- /dev/null +++ b/kaiwu/tests/test_search_subagent.py @@ -0,0 +1,363 @@ +""" +Tests for SearchSubagent and UpstreamManifest. +Covers: isolated context, parallel reads, manifest extraction, contract checking. +""" + +import pytest +from unittest.mock import MagicMock, patch +from kaiwu.core.upstream_manifest import UpstreamManifest +from kaiwu.experts.search_subagent import SearchSubagent, SearchResult +from kaiwu.core.context import TaskContext + + +# ══════════════════════════════════════════════════════════════════════ +# UpstreamManifest Tests +# ══════════════════════════════════════════════════════════════════════ + + +class TestUpstreamManifest: + def setup_method(self): + self.manifest = UpstreamManifest() + + def test_extract_python_signatures(self): + patches = [{ + "file": "utils.py", + "original": "", + "modified": "def reset_password(token: str, new_pass: str) -> bool:\n return True\n", + }] + self.manifest.update(patches) + sigs = self.manifest.get_all_signatures() + assert "utils.py" in sigs + assert "reset_password" in sigs["utils.py"] + assert "token: str" in sigs["utils.py"]["reset_password"] + assert "-> bool" in sigs["utils.py"]["reset_password"] + + def test_extract_constants(self): + patches = [{ + "file": "config.py", + "original": "", + "modified": "API_BASE = 'https://api.example.com/v2'\nMAX_RETRIES = 3\n", + }] + self.manifest.update(patches) + consts = self.manifest.get_all_constants() + assert "config.py" in consts + assert "API_BASE" in consts["config.py"] + assert "MAX_RETRIES" in consts["config.py"] + + def test_extract_imports(self): + patches = [{ + "file": "views.py", + "original": "", + "modified": "from kaiwu.core.context import TaskContext\nimport os\n\ndef handler():\n pass\n", + }] + self.manifest.update(patches) + # Should track dependency + assert "views.py" in self.manifest._imports + assert any("TaskContext" in imp for imp in self.manifest._imports["views.py"]) + + def test_get_constraints_for_file(self): + # First, register upstream signatures + patches_upstream = [{ + "file": "kaiwu/core/context.py", + "original": "", + "modified": "class TaskContext:\n def __init__(self, user_input: str = ''):\n self.user_input = user_input\n", + }] + self.manifest.update(patches_upstream) + + # Then register a file that imports from upstream + patches_downstream = [{ + "file": "views.py", + "original": "", + "modified": "from kaiwu.core.context import TaskContext\n\ndef handler():\n ctx = TaskContext(user_input='test')\n", + }] + self.manifest.update(patches_downstream) + + constraints = self.manifest.get_constraints_for_file("views.py") + # Should mention the upstream contract + assert "kaiwu/core/context.py" in constraints or constraints == "" + + def test_check_consistency_no_violations(self): + # Register a function signature + self.manifest._signatures["utils.py"] = { + "process": "def process(data: list, mode: str)" + } + self.manifest._dependency_graph["main.py"] = ["utils.py"] + + # Code that calls process correctly + code = "result = process(my_data, 'fast')\n" + violations = self.manifest.check_consistency("main.py", code) + assert violations == [] + + def test_check_consistency_too_many_args(self): + # Register: process takes 2 params (data, mode) + self.manifest._signatures["utils.py"] = { + "process": "def process(data: list, mode: str)" + } + self.manifest._dependency_graph["main.py"] = ["utils.py"] + + # Code that calls process with 3 args + code = "result = process(my_data, 'fast', True)\n" + violations = self.manifest.check_consistency("main.py", code) + assert len(violations) == 1 + assert "3" in violations[0] and "2" in violations[0] + + def test_check_consistency_constant_redefinition(self): + self.manifest._constants["config.py"] = {"API_BASE": "'https://api.example.com'"} + self.manifest._dependency_graph["main.py"] = ["config.py"] + + # Code redefines constant with different value + code = "API_BASE = 'https://wrong-url.com'\nprint(API_BASE)\n" + violations = self.manifest.check_consistency("main.py", code) + assert len(violations) >= 1 + assert "API_BASE" in violations[0] + + def test_to_compact_summary(self): + patches = [{ + "file": "a.py", + "original": "", + "modified": "MAX_SIZE = 100\ndef foo(x: int) -> str:\n return str(x)\n", + }] + self.manifest.update(patches) + summary = self.manifest.to_compact_summary() + assert "signatures" in summary + assert "constants" in summary + assert "imports" in summary + + def test_clear(self): + self.manifest._signatures["a.py"] = {"foo": "def foo()"} + self.manifest.clear() + assert self.manifest._signatures == {} + assert self.manifest._constants == {} + + def test_async_function_extraction(self): + patches = [{ + "file": "api.py", + "original": "", + "modified": "async def fetch_data(url: str, timeout: int = 30) -> dict:\n return {}\n", + }] + self.manifest.update(patches) + sigs = self.manifest.get_all_signatures() + assert "fetch_data" in sigs["api.py"] + assert "async def" in sigs["api.py"]["fetch_data"] + + def test_regex_fallback_for_non_python(self): + patches = [{ + "file": "main.go", + "original": "", + "modified": "func ProcessData(input []byte, mode string) error {\n return nil\n}\n\nMAX_BUFFER = 4096\n", + }] + self.manifest.update(patches) + sigs = self.manifest.get_all_signatures() + assert "main.go" in sigs + assert "ProcessData" in sigs["main.go"] + consts = self.manifest.get_all_constants() + assert "MAX_BUFFER" in consts.get("main.go", {}) + + def test_multiple_patches_accumulate(self): + self.manifest.update([{ + "file": "a.py", + "original": "", + "modified": "def foo() -> int:\n return 1\n", + }]) + self.manifest.update([{ + "file": "b.py", + "original": "", + "modified": "def bar(x: str) -> str:\n return x\n", + }]) + sigs = self.manifest.get_all_signatures() + assert "foo" in sigs["a.py"] + assert "bar" in sigs["b.py"] + + def test_count_params_with_self(self): + """self/cls should not count as parameters.""" + count = UpstreamManifest._count_params("def method(self, x: int, y: int)") + assert count == 2 + + def test_count_params_with_kwargs(self): + """*args/**kwargs means we can't determine exact count.""" + count = UpstreamManifest._count_params("def func(x, *args, **kwargs)") + assert count is None + + def test_count_args_nested(self): + """Nested function calls shouldn't split on inner commas.""" + count = UpstreamManifest._count_args("foo(1, 2), bar(3)") + assert count == 2 # Two top-level args + + +# ══════════════════════════════════════════════════════════════════════ +# SearchSubagent Tests +# ══════════════════════════════════════════════════════════════════════ + + +class TestSearchSubagent: + def setup_method(self): + self.mock_locator = MagicMock() + self.mock_tools = MagicMock() + self.subagent = SearchSubagent(self.mock_locator, self.mock_tools) + + def test_search_returns_clean_results(self): + """SearchSubagent should return structured results, not raw locator state.""" + self.mock_locator.run.return_value = { + "relevant_files": ["src/main.py", "src/utils.py"], + "relevant_functions": ["process", "validate"], + "edit_locations": ["src/main.py:L10-20"], + "method": "bm25_graph", + } + self.mock_tools.read_file.return_value = "def process(data):\n return data\n" + + ctx = TaskContext(user_input="fix the process function", project_root="/tmp/proj") + result = self.subagent.search(ctx) + + assert result["relevant_files"] == ["src/main.py", "src/utils.py"] + assert result["relevant_functions"] == ["process", "validate"] + assert result["method"] == "bm25_graph" + assert "code_snippets" in result + + def test_search_does_not_modify_original_ctx(self): + """SearchSubagent must not pollute the original TaskContext.""" + self.mock_locator.run.return_value = { + "relevant_files": ["a.py"], + "relevant_functions": ["foo"], + "edit_locations": [], + "method": "bm25_graph", + } + self.mock_tools.read_file.return_value = "def foo():\n pass\n" + + ctx = TaskContext(user_input="test", project_root="/tmp") + # These should remain None after search + assert ctx.locator_output is None + + self.subagent.search(ctx) + + # Original ctx should NOT be modified by search + assert ctx.locator_output is None + + def test_search_with_manifest_constraints(self): + """Should include upstream constraints when manifest is provided.""" + self.mock_locator.run.return_value = { + "relevant_files": ["views.py"], + "relevant_functions": ["handler"], + "edit_locations": [], + "method": "llm_fallback", + } + self.mock_tools.read_file.return_value = "def handler():\n pass\n" + + manifest = UpstreamManifest() + manifest._signatures["utils.py"] = {"process": "def process(data: list) -> dict"} + manifest._dependency_graph["views.py"] = ["utils.py"] + + ctx = TaskContext(user_input="fix handler", project_root="/tmp") + result = self.subagent.search(ctx, manifest) + + assert "upstream_constraints" in result + assert "process" in result["upstream_constraints"] + + def test_search_locator_returns_nothing(self): + """Should return empty results gracefully when locator finds nothing.""" + self.mock_locator.run.return_value = None + + ctx = TaskContext(user_input="fix something", project_root="/tmp") + result = self.subagent.search(ctx) + + assert result["relevant_files"] == [] + assert result["code_snippets"] == {} + assert result["method"] == "none" + + def test_parallel_read_handles_errors(self): + """File read errors should not crash the subagent.""" + self.mock_locator.run.return_value = { + "relevant_files": ["good.py", "bad.py"], + "relevant_functions": ["foo"], + "edit_locations": [], + "method": "bm25_graph", + } + + def side_effect(path): + if "bad" in path: + return "[ERROR] File not found" + return "def foo():\n return 42\n" + + self.mock_tools.read_file.side_effect = side_effect + + ctx = TaskContext(user_input="test", project_root="/tmp") + result = self.subagent.search(ctx) + + # Should have snippet for good.py but not bad.py + assert "good.py" in result["code_snippets"] + assert "bad.py" not in result["code_snippets"] + + def test_extract_precise_snippet_with_functions(self): + content = """import os + +def helper(): + return 1 + +def target_func(x, y): + result = x + y + if result > 10: + return result * 2 + return result + +def another(): + pass +""" + snippet = self.subagent._extract_precise_snippet( + content, ["target_func"], "test.py" + ) + assert "target_func" in snippet + assert "x + y" in snippet + # Should NOT include unrelated functions in full + assert "def another" not in snippet or "..." in snippet + + def test_extract_precise_snippet_no_functions(self): + """When no functions specified, return first N lines.""" + content = "line1\nline2\nline3\n" + snippet = self.subagent._extract_precise_snippet(content, [], "test.py") + assert "line1" in snippet + + def test_merge_ranges(self): + ranges = [(1, 10), (8, 20), (25, 30)] + merged = SearchSubagent._merge_ranges(ranges) + assert merged == [(1, 20), (25, 30)] + + def test_merge_ranges_with_gap(self): + """Ranges within 3 lines should merge.""" + ranges = [(1, 10), (12, 20)] # gap of 2 + merged = SearchSubagent._merge_ranges(ranges) + assert merged == [(1, 20)] + + def test_find_function_range(self): + lines = [ + "import os", + "", + "def foo(x):", + " return x + 1", + "", + "def bar():", + " pass", + ] + start, end = SearchSubagent._find_function_range(lines, "foo") + assert start == 2 + assert end == 5 # up to but not including def bar + + def test_find_function_range_not_found(self): + lines = ["def other():", " pass"] + start, end = SearchSubagent._find_function_range(lines, "nonexistent") + assert start == -1 + assert end == -1 + + +class TestSearchResult: + def test_to_dict(self): + r = SearchResult( + file="src/main.py", + start_line=10, + end_line=25, + content="def foo():\n pass", + function_name="foo", + ) + d = r.to_dict() + assert d["file"] == "src/main.py" + assert d["start_line"] == 10 + assert d["end_line"] == 25 + assert d["function_name"] == "foo" diff --git a/kaiwu/tests/test_server.py b/kaiwu/tests/test_server.py index 5085204..f489694 100644 --- a/kaiwu/tests/test_server.py +++ b/kaiwu/tests/test_server.py @@ -57,7 +57,7 @@ class TestServerHealth: assert resp.status_code == 200 data = resp.json() assert data["status"] == "ok" - assert data["version"] == "1.3.0" + assert data["version"] == "1.5.0" assert data["model"] == "test-model" asyncio.run(_test()) @@ -222,7 +222,7 @@ class TestServerModels: from kaiwu.server.models import HealthResponse resp = HealthResponse(model="qwen3-8b", project_root="/tmp") assert resp.status == "ok" - assert resp.version == "1.3.0" + assert resp.version == "1.5.0" def test_file_content(self): from kaiwu.server.models import FileContent diff --git a/kaiwu/tui/app.py b/kaiwu/tui/app.py index 92d6927..ba76149 100644 --- a/kaiwu/tui/app.py +++ b/kaiwu/tui/app.py @@ -51,6 +51,42 @@ EVENT_ICONS = { "task_completed": "✅", "task_error": "❌", "keepalive": "", + # v1.5: SearchSubagent + UpstreamManifest events + "contract_violation": "⚠️", + "locator": "🔍", + "locator_done": "🔍", + "locator_fail": "🔍", + "generator": "⚙️", + "generator_done": "⚙️", + "generator_fail": "⚙️", + "verifier": "✔️", + "verifier_done": "✔️", + "verifier_fail": "✔️", + "reflection": "💭", + "debug": "🐛", + "debug_done": "🐛", + "replay": "📼", + "ab_test": "🧪", + "ab_test_record": "🧪", + "ab_graduated": "🎓", + "ab_archived": "📦", + "flywheel": "⚡", + "checkpoint": "💾", + "import_fix": "🔧", + "office": "📄", + "office_done": "📄", + "office_fail": "📄", + "search": "🌐", + "search_done": "🌐", + "suggest": "💡", + "warning": "⚠️", + "watchdog": "⏱️", + "low_confidence": "⚠️", + "review": "👁️", + "review_done": "👁️", + "review_gap": "⚠️", + "chat": "💬", + "vision": "👁️", } DEFAULT_SERVER_URL = "http://127.0.0.1:7355" diff --git a/pyproject.toml b/pyproject.toml index 02e3f37..7fcacfe 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,7 +8,7 @@ version = "1.4.0" description = "KwCode - Local-model coding agent with MoE expert pipeline" requires-python = ">=3.10" readme = "README.md" -license = "MIT" +license = {text = "MIT"} keywords = ["coding-agent", "local-llm", "ollama", "code-generation"] classifiers = [ "Development Status :: 4 - Beta", @@ -81,3 +81,17 @@ include = ["kaiwu*"] [tool.setuptools.package-data] "kaiwu.builtin_experts" = ["*.yaml"] + +[tool.ruff] +line-length = 100 +target-version = "py311" + +[tool.ruff.lint] +select = ["E", "F", "W", "I"] +ignore = ["E501"] + +[tool.mypy] +python_version = "3.11" +warn_return_any = true +warn_unused_configs = true +ignore_missing_imports = true