Three changes:
1. Fix the bench harness's set_pou_code case. The previous version
passed POU_PATH / DECLARATION_CODE / IMPLEMENTATION_CODE, but the
script template expects POU_FULL_PATH / DECLARATION_CONTENT /
IMPLEMENTATION_CONTENT. Plus the recently-added SET_DECLARATION /
SET_IMPLEMENTATION boolean flags from 35abc8c. With these
corrections, set_pou_code passes for the first time in the bench.
2. Add bench-results-v5.json from a fresh persistent-mode run on
MCPTest2 v1.3.4.0. All 10 cases PASS.
3. Update TEST_OVERVIEW.md with the v5 numbers alongside the v1
historical numbers. v5 is ~2x faster across the board than v1
(likely the cumulative effect of the ScriptManager cache removal +
SP22 Patch 1 IPC improvements). first-cold open_project is slower
than v1 (10.6 s vs 7.7 s) -- one-shot, noise probably; warm
already-open call dropped from 740 ms to 314 ms.
The v5 set_pou_code pass is the third end-to-end signal that the
omitted-decl wipe fix is good (after the e2e regression test and the
live PLC_PRG restoration earlier today).
TEST_OVERVIEW.md: complete tool inventory (37 tools), each tagged
working / broken with status notes and per-mode timing characteristics.
Mode primer up top explains the per-call vs first-call cost asymmetry.
Deep-dive on every broken tool with proposed fixes:
- create_folder: parent_object.create_folder() not exposed in SP21+;
fall back to create_object(typeUuid=...) or types.IecFolder.
- compile_project / get_compile_messages: IronPython 2.7 json.dumps
can't serialize System.Int64 (line_number / position fields).
Fix is a _coerce_int helper applied uniformly.
- connect_to_device: SP21+ may expose the login enum as LoginMode
instead of OnlineChangeOption. Extend the candidate sweep over
multiple enum sources, with priority on TryOnlineChange-equivalents.
- open_project (cross-project switch): ensure_project_open has the
"close prior project" branch commented out; uncomment with a
save+close+delay sequence and silent-mode guard.
list_project_libraries is flagged as ✅ working in current SP22
(historical entries in the project memory should be cleared).
bench.mjs: standalone benchmark harness driving HeadlessExecutor and
CodesysLauncher directly (no MCP server in the loop). Copies the source
.project to a temp dir so write tools don't mutate the original. Covers
9 tools (read-only + write-revertible) with configurable iterations,
emits markdown to stdout + JSON to --out.
Run with:
node tests/bench.mjs --modes headless,persistent --iterations 2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
