Three changes:
1. Fix the bench harness's set_pou_code case. The previous version
passed POU_PATH / DECLARATION_CODE / IMPLEMENTATION_CODE, but the
script template expects POU_FULL_PATH / DECLARATION_CONTENT /
IMPLEMENTATION_CONTENT. Plus the recently-added SET_DECLARATION /
SET_IMPLEMENTATION boolean flags from 35abc8c. With these
corrections, set_pou_code passes for the first time in the bench.
2. Add bench-results-v5.json from a fresh persistent-mode run on
MCPTest2 v1.3.4.0. All 10 cases PASS.
3. Update TEST_OVERVIEW.md with the v5 numbers alongside the v1
historical numbers. v5 is ~2x faster across the board than v1
(likely the cumulative effect of the ScriptManager cache removal +
SP22 Patch 1 IPC improvements). first-cold open_project is slower
than v1 (10.6 s vs 7.7 s) -- one-shot, noise probably; warm
already-open call dropped from 740 ms to 314 ms.
The v5 set_pou_code pass is the third end-to-end signal that the
omitted-decl wipe fix is good (after the e2e regression test and the
live PLC_PRG restoration earlier today).
Bug: calling set_pou_code with implementationCode only (declarationCode
omitted) would wipe the POU's PROGRAM/VAR...END_VAR block in the binary.
After such a call mirror_export classified the POU as 'UNKNOWN' (no
PROGRAM/FUNCTION_BLOCK keyword in the empty declaration), and the var
block disappeared from the .st mirror file.
Root cause: the TS wrapper substituted '' (empty string) into the Python
template when declarationCode was undefined, giving DECLARATION_CONTENT
= "". The Python script then took the truthy-ish branch (empty string
is not None) and called decl_obj.replace('') -- wiping textual_decl.
Fix: pass explicit SET_DECLARATION / SET_IMPLEMENTATION boolean flags
from the TS wrapper, gate the replace() calls on those flags. Empty
string remains a valid intentional value (caller wants to wipe).
- Reproduced on MCPTest2: PLC_PRG declaration block was wiped between
v1.3.0.0 and v1.3.1.0 by exactly this code path.
- Regression test added in tests/integration/e2e.test.ts covering the
omitted-declarationCode path.
- Existing set_pou_code test extended to assert SET_DECLARATION /
SET_IMPLEMENTATION are emitted in the rendered script.
- Bumped status from "fixed (factory fallback)" to "fixed in c87f3a9
(4-iteration debug saga)" with the v1->v4 iteration table:
* v1 2607063: name= kwarg -- FAIL (SP22 wants foldername)
* v2 e07f281: positional + foldername= -- FAIL (silent None return)
* v3 32e6120: project-level + cache removal -- FAIL (still None=fail)
* v4 c87f3a9: walk children to detect side-effect success -- PASS
- Real root cause documented: SP22's create_folder returns void; the
fix verifies by walking parent.get_children(False).
- Side benefit from v3 (ScriptManager cache dropped) noted as the
hot-reload mechanism that made v4 reachable in one debug session.
- "Lesson for future fork work" callout: probably applies to other
create_* methods too, audit pending.
- Inventory table row for create_folder updated with the c87f3a9
link + a more accurate description of the strategy chain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
create_folder v2 (positional foldername) returned None silently against
the SP22 Application object -- no exception raised, no folder created.
Investigation showed:
- ScriptObject.create_folder(foldername) is documented to "create a
folder in the structured view of the parent node", but on Application
specifically it's a silent no-op (the structured view isn't pinned to
POU view there).
- ScriptProject.create_folder(foldername, structured_view=None) on the
project itself with explicit SV_POU GUID
({21AF5390-2942-461a-BF89-951AAF6999F1}) is the documented and
reliable pathway -- the resulting folder appears under Application
in the IDE tree because that's where SV_POU lives.
v3 fix: try strategies in order until one returns non-None:
(1) primary_project.create_folder(name, SV_POU_GUID) -- new, primary
(2) parent.create_folder(name) positional -- pre-SP21 path
(3) parent.create_folder(foldername=name) -- alt keyword
(4) primary_project.create_folder(name) -- default view
(5) parent.create_object(typeUuid='85d1215e-...') -- alt factory
(6) parent.add(script_engine.types.IecFolder, name=name) -- legacy
Each strategy guards on hasattr + return-value-not-None, so a silent
no-op falls through instead of being mistaken for success.
ScriptManager: dropped the in-memory template cache. Each loadTemplate
call now reads the .py from disk fresh. Cost: ~1ms per call vs ~1.5s
of CODESYS execution time -- invisible. Win: edits to dist/scripts/
take effect without an MCP restart, which makes iterating on script-
side fixes (like this very create_folder loop) much faster. Existing
"cache hit" unit test rewritten as "two loads return equal content".
tests/test-fixes.mjs: standalone harness that drives a single persistent
CODESYS through HeadlessExecutor + CodesysLauncher to verify the four
broken-tool fixes end-to-end. Useful for regression-testing without
needing a vsc reboot loop. Currently only smoke-tests
create_folder + compile + cross-project; expand as more fixes need
verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Headline result table at the top: persistent is 15-24x faster than
headless across all CODESYS-roundtripping tools. Numbers are from a
fresh bench-results.json run on MCPTest2.
- Updated mode primer with measured first-call vs warm-call costs:
persistent first launch ~14.6s, subsequent calls ~1.5s; headless
first cold call ~58s, warm calls ~22s.
- Inventory table: replaced "typical" estimates with measured numbers
for the 9 tools the bench covers.
- Deep-dive section: each broken tool now marked "fixed in 2607063"
with concise summary of the fix that landed (instead of "proposed
fix"). Removed the long pre-fix code blocks since they're in the
commit history now.
- Status legend: create_folder, compile_project, get_compile_messages,
connect_to_device, open_project all flipped from broken to fixed
(with caveats: some need PLC or runtime verification).
- Footnote about set_pou_code bench-harness failure (multi-line code
through triple-quoted-string interpolation -- not a tool bug).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
create_folder SP21+ fallback, ensure_project_open cross-project switch)
+ bench results
compile_project.py + get_compile_messages.py:
- IronPython 2.7's json.dumps cannot serialize System.Int64-backed `long`
values, which is what CODESYS's compile-message objects expose as
line_number / position. Added _coerce_int + _coerce_str helpers and a
shared _build_message_entry function. Three duplicated message-building
blocks collapsed into single helper calls.
- Defensive `try: json.dumps(...) except TypeError: json.dumps(default=str)`
so a stray field that slips past the helpers doesn't kill the emit.
connect_to_device.py:
- SP21+ may expose the login enum as LoginMode rather than
OnlineChangeOption. Extended the candidate sweep to probe both
script_engine.LoginMode and script_engine.OnlineChangeOption AND
online_app.LoginMode/OnlineChangeOption (some builds attach it to the
app object). Added "OnlineChange" + "Login" + "Download" to the
preferred-priority list. Added a 3-arg call shape variant for SPs
that take (mode, secondary-mode, force-bool).
create_folder.py:
- parent_object.create_folder() is no longer exposed on every parent type
in SP21+. Added two fallback factories tried in order:
1. parent.create_object(typeUuid='85d1215e-6520-4983-9a55-2d39d1f24cb4', name=...)
2. parent.add(script_engine.types.IecFolder, name=...)
with detailed warnings when each path fails. Final TypeError now lists
every factory tried so a future SP rotation surfaces clearly.
ensure_project_open.py:
- Uncommented the close-prior-project branch (was a TODO since the
initial fork). Cross-project switches in a persistent CODESYS now do
save() -> close() -> 500ms pump -> open(target). Without this,
projects.open against a different already-primary project fails
intermittently on file lock contention or pops a "project in use"
modal that freezes the IDE thread.
- save() is best-effort: if it raises (transient lock, save-as required)
we still proceed with close + open rather than getting stuck in a
half-switched state forever.
tests/bench-results.json:
- Captured timings from a clean run on MCPTest2 (PLCWinNT, 5 lib refs,
~12 POUs). 9 tools x 2 modes x iterations.
Headers (mean ms): persistent vs headless --
open_project 7700 vs 40041 (~5x; first call cold)
mirror_export 1547 vs 23723 (~15x)
list_project_libraries 1565 vs 23322 (~15x)
get_all_pou_code 1607 vs 23376 (~15x)
save_project 2095 vs 23321 (~11x)
create_pou (FB) 1540 vs 23903 (~16x)
delete_object 1544 vs 27420 (~18x)
bump_project_version 1540 vs 30678 (~20x)
bump_project_version #2 1556 vs 37769 (~24x)
- set_pou_code FAILED in both modes -- bench harness param-shape issue
(multi-line code passed verbatim to triple-quoted-string interpolation
doesn't survive the round-trip). Tool itself works fine through the
MCP tool call path; bench needs to escape newlines / use the same
prepareScriptWithHelpers shape the server uses. Filed for follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TEST_OVERVIEW.md: complete tool inventory (37 tools), each tagged
working / broken with status notes and per-mode timing characteristics.
Mode primer up top explains the per-call vs first-call cost asymmetry.
Deep-dive on every broken tool with proposed fixes:
- create_folder: parent_object.create_folder() not exposed in SP21+;
fall back to create_object(typeUuid=...) or types.IecFolder.
- compile_project / get_compile_messages: IronPython 2.7 json.dumps
can't serialize System.Int64 (line_number / position fields).
Fix is a _coerce_int helper applied uniformly.
- connect_to_device: SP21+ may expose the login enum as LoginMode
instead of OnlineChangeOption. Extend the candidate sweep over
multiple enum sources, with priority on TryOnlineChange-equivalents.
- open_project (cross-project switch): ensure_project_open has the
"close prior project" branch commented out; uncomment with a
save+close+delay sequence and silent-mode guard.
list_project_libraries is flagged as ✅ working in current SP22
(historical entries in the project memory should be cleared).
bench.mjs: standalone benchmark harness driving HeadlessExecutor and
CodesysLauncher directly (no MCP server in the loop). Copies the source
.project to a temp dir so write tools don't mutate the original. Covers
9 tools (read-only + write-revertible) with configurable iterations,
emits markdown to stdout + JSON to --out.
Run with:
node tests/bench.mjs --modes headless,persistent --iterations 2
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.3.0 returning-watcher architecture — background thread polls for commands
and marshals execution onto the CODESYS UI thread, keeping the IDE fully
responsive between operations. File-based IPC with atomic writes, async mutex
command serialization, headless fallback, and 35 passing tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
