Repository navigation
🧪 Test(e2e): migrate the ext-knowledge mock into the formal framework and repair the daily knowledge cases - #4090
Merged
Merged
Conversation
cj2026-bit
requested review from
Dallas98,
WMC001,
YehongPan,
hhhhsc701 and
jeffwu-1999
as code owners
October 14, 2026 02:01
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
… and repair the daily knowledge cases - Move the ext-knowledge mock contract into test-e2e/infra/mock-services/ext-knowledge-mock (single-source build context, runner/product dual compose overlays, per-case attach tooling with identity check, endpoint lock, full reset and evidence export, suite-worker integration, validate_execution mock-service enum + profile selection, publisher evidence whitelist). - Migrate the eight formal mock consumers (AGT-001/002, CTR-016/017/046/047, PERF-09, REL-02) onto the framework-managed instance; API-AUTO-FC03F18034CB0D51 is repaired separately below. - Repair the four daily knowledge cases: the D2 delete signature and audit isolation, PW-KB-LOCAL-01 (agent-card configure action + AddKnowledgeDrawer autosave), PW-FILE-01 (composer/attachment and refreshed-chat drift), and the new PW-KB-AIDP-01 journey with explicit mock/real_smoke profile selection. - Add the fidelity suite test/ext_components/ext_knowledge (57 original + 6 regressions = 63 tests) driving the real adapters against the mock; CI collects it through test/ext_components. - Repair shared D4 helper drift found under live execution (collapsible agent editor sections, tool-config autosave evidence, high-fidelity conversation guide, collaborator drawer, guardrail modal, agent publish before a new conversation). - No product code changed.
…ployment with D-001 Review-fix pass on the framework migration: - Bind RESULT_DIR in the worker process itself and adopt any prerequisite receipt (not only preparation-step failures), so attach and dependency failures report BLOCKED_BY_DEPENDENCY with their concrete reason instead of a bare AUTOMATION_ERROR. Regression tests cover the classification and the worker wiring (the previous tests set RESULT_DIR themselves and thus masked the gap). - Publish the mock port on loopback only per design D-001; product-side callers keep using the compose-network DNS name. The registration test now pins the loopback binding. - Decide the instance's released/unknown state before releasing the attach lock (removes the unlock-vs-rewrite race); the schema description now matches the allowed host-mode equality of runner_url/product_url. - Fidelity conftest clears the wire log per test and restores the identity environment variables at teardown. - Control plane validates the whole /_reset service list before executing and checks the AIDP sub-app response codes on fail-next/fail-reset. - PW-KB-AIDP-01 syncs the contract text with the implemented evidence model and gains acceptance obligations and bindings; drop an unused variable and a stray blank line.
cj2026-bit
force-pushed
the
test-ext-knowledge-mock-v2
branch
from
October 14, 2026 06:19
7b15fc5 to
2e7ca27
Compare
Minimum-regression re-run against a fresh one-off deployment on the review-fix code: PW-TASK-01 and AGT-001 passed, PW-KB-AIDP-01 passed its full mock-profile journey, and the repaired helpers were exercised successfully wherever reached (PW-CHAT-01 step 1, PW-CHAT-05 steps 1-3, PW-PROMPT-02 preconditions, PW-AGENT-01 precondition 3). The remaining cases fail on their own pre-existing post-#4058 drift (AgentPage.create() flow, retired 请输入智能体描述 placeholder, 智能体角色 tab, 执行 mode button, 知识库： scope button), recorded per case and left untouched. The negative attach path was verified live: a foreign identity yields BLOCKED_BY_DEPENDENCY with the concrete reason written to the case runtime receipt.
… mock README Documents what a human enters when binding the mocked services as agent tools (fields, product-side base URLs, credential shapes, runtime input names incl. iData's question + required rerank_model_id) and the full seed catalog the pickers list per service, including the AIDP startup rename and the permission-table registration note. Markdown only; no behavior or test change.
…e files Satisfies develop's new copyright-check workflow (#4087), generated with its official fixer (scripts/check_copyright.py --fix): one header comment line plus a separator per file, after any shebang/coding line. Comments only — no behavior change; fidelity suite (63), attach/worker tests (31) and mock-services tests (26) re-run green after the insertion.
jeffwu-1999
reviewed
Oct 15, 2026
…-mock-v2 # Conflicts: # test-e2e/infra/tools/suite_publish.py # test-e2e/infra/tools/tests/test_suite_publish.py
…lock on an unusable profile contract Review fixes (jeffwu-1999): - _attach_locked: the exception path released the lock before rewriting the released state, the reverse of the release() rule fixed earlier — a concurrent attacher could read the pre-failure released=False in that window and false-block with 'previous attach state was not released cleanly'. The state is now decided while the lock is still held; a failed reset still keeps the unknown state. - resolve_profile_selection: an unreadable case.yaml or a missing execution.profiles entry for the selected profile silently produced an empty dependency set, letting attach() skip the mock identity check, lock and reset while the batch report looked clean. Both paths now raise a typed dependency failure with the concrete reason; a legitimately empty required_services entry (real_smoke) is unchanged. - Four regression tests: state-at-unlock snapshots for both failure orderings, plus both blocking paths of the profile contract. Offline: attach/worker 35 passed, fidelity 63 passed, mock-services 26 passed, asset validator green.
jeffwu-1999
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
API-AUTO-FC03F18034CB0D51raisedTypeErroron a stale delete signature,PW-KB-LOCAL-01andPW-FILE-01still targeted the retired agent-card / composer controls, andPW-KB-AIDP-01had no execution binding, script or page object at all.Fix
test-e2e/infra/mock-services/ext-knowledge-mock/(single-source package + seeds, Docker context builder with a manifest hash, dual compose overlays for the runner and product networks, README and deployment descriptor).test-e2e/infra/tools/ext_attach.pyverifies the mock identity, guards one instance per endpoint (file lock), probes both sides, resets per case and exports evidence; the suite worker attaches it before any other preparation.validate_executiongained therequired_mock_servicesenum and the newprofile_selectionfield, and the publisher whitelists the four new evidence files.http_requestand isolates the audit sink at its actual lookup site, keeping every original redis / pending / fallback / permission assertion;PW-KB-LOCAL-01uses the current agent-card 配置 action and the AddKnowledgeDrawer autosave;PW-FILE-01uses the current composer / attachment panel and the refreshed-chat sidebar selector;PW-KB-AIDP-01is implemented end to end (case-local spec, AidpKnowledgePage, mock / real_smoke profile selection through the machine-local config) proving branch exclusivity, KB selection, the realaidp_searchtool-config autosave, Agent Debug recall with the oracle text and source citation, refresh persistence, and a released-version conversation recall.test/ext_components/ext_knowledge/runs the original 57 fidelity tests plus 6 new regressions (63 total) that drive the real adapters (DataMate / Dify / iData / RAGFlow / Haotian / AIDP / fault) against a session-private live mock; CI collects it viatest/ext_components.Verification
test/ext_components/ext_knowledge, runs in CI throughtest/run_all_test.py).local-latestdeployment built from this branch (one-off data root, ext-knowledge mock attached over the product network): a single local batch passed 11/11 (the eight migrated consumers + PW-FILE-01 + PW-KB-LOCAL-01);PW-KB-AIDP-01passed its full mock-profile journey (4 preconditions / 7 steps / 4 assertions, clean cleanup); the shared-helper regressionPW-AGENT-02passed with the repaired helpers.BLOCKED_BY_DEPENDENCYwith the concrete reason on disk — verified live, not only by unit tests); the mock port is loopback-only per D-001; the attach release state is decided before the lock drops; the fidelity conftest clears the wire log per test and restores its identity environment; the control plane validates the whole/_resetlist first and checks AIDP sub-app response codes. CI and codecov/patch re-ran green on both pushes.PW-TASK-01andAGT-001passed;PW-KB-AIDP-01passed its full mock-profile journey again (acceptance obligations bound). The repaired helpers were exercised successfully wherever reached (PW-CHAT-01step 1,PW-CHAT-05steps 1–3,PW-PROMPT-02preconditions,PW-AGENT-01precondition 3).Notes
aidp-journey.example.yaml): the controlled KB must be registered for the journey tenant (aidp_kb_permission_tACTIVE row owned by the tenant admin) because the product intersects the AIDP catalog with local permission rows and has no HTTP flow to register an existing remote KB; the deployment must seed it once.PW-AGENT-01andPW-VERSION-01in the unchangedAgentPage.create()flow (old 创建智能体 dialog / 新建 entry),PW-PROMPT-01on the retired 请输入智能体描述 placeholder,PW-PROMPT-02on the retired 智能体角色 tab,PW-CHAT-01on the retired 执行 mode button,PW-CHAT-05on the retired 知识库： scope button. These locators are outside this PR's changed surface and remain recorded as known drift.PW-CHAT-02/PW-MEDIA-STT-01still inline the retired发送消息...composer placeholder, and the tab-basedbindTool/bindSkill/openToolsSkillshelpers plus their four callers still target the pre-redesign tools/skills tab.