docs+test: TEST_OVERVIEW + bench harness for headless vs persistent

TEST_OVERVIEW.md: complete tool inventory (37 tools), each tagged working / broken with status notes and per-mode timing characteristics. Mode primer up top explains the per-call vs first-call cost asymmetry. Deep-dive on every broken tool with proposed fixes: - create_folder: parent_object.create_folder() not exposed in SP21+; fall back to create_object(typeUuid=...) or types.IecFolder. - compile_project / get_compile_messages: IronPython 2.7 json.dumps can't serialize System.Int64 (line_number / position fields). Fix is a _coerce_int helper applied uniformly. - connect_to_device: SP21+ may expose the login enum as LoginMode instead of OnlineChangeOption. Extend the candidate sweep over multiple enum sources, with priority on TryOnlineChange-equivalents. - open_project (cross-project switch): ensure_project_open has the "close prior project" branch commented out; uncomment with a save+close+delay sequence and silent-mode guard. list_project_libraries is flagged as ✅ working in current SP22 (historical entries in the project memory should be cleared). bench.mjs: standalone benchmark harness driving HeadlessExecutor and CodesysLauncher directly (no MCP server in the loop). Copies the source .project to a temp dir so write tools don't mutate the original. Covers 9 tools (read-only + write-revertible) with configurable iterations, emits markdown to stdout + JSON to --out. Run with: node tests/bench.mjs --modes headless,persistent --iterations 2 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 16:41:52 +02:00 · 2026-04-26 16:41:52 +02:00 · 2e2ab00ae1
commit 2e2ab00ae1
parent 53c7a0c27d
2 changed files with 545 additions and 0 deletions
--- a/tests/TEST_OVERVIEW.md
+++ b/tests/TEST_OVERVIEW.md
@ -0,0 +1,320 @@
+# Codesys-MCP — test overview, tool inventory, broken-tool deep dive
+
+A complete map of the **37 tools** registered in [`src/server.ts`](../src/server.ts), with current working/broken status, what each one does, expected timing characteristics in **headless** vs **persistent** mode, and a deep-dive + proposed fix for each broken tool.
+
+For runnable benchmarks see [`bench.mjs`](bench.mjs):
+
+```bash
+node tests/bench.mjs --modes headless,persistent --iterations 2 --out tests/bench-results.json
+```
+
+The benchmark drives `HeadlessExecutor` and `CodesysLauncher` directly (no MCP server in the loop), copies the source `.project` to a temp dir so write tools don't mutate the real binary, and emits a markdown table to stdout plus raw JSON to `--out`.
+
+## Mode primer
+
+| Mode | Per-call overhead | First-call cost | Best for |
+|---|---|---|---|
+| **headless** | full CODESYS `--noUI` startup on **every** call | ~5–15 s | One-shot scripts, CI, tools that don't share state. Cleaner — no stale in-memory tree drift. |
+| **persistent** | IPC poll (~250 ms) + Python execution | ~5–15 s on first launch only; subsequent calls are sub-second | Interactive editing sessions where many calls land on the same project. Watch out for in-memory drift after long sessions (see fork-fix history below). |
+
+The orchestrator's `release_project_version` recently grew a **post-bump sanity check** ([commit `53c7a0c`](https://github.com/phobicdotno/Codesys-MCP/commit/53c7a0c)) that compares the bumped version against the latest `v*` git tag and aborts before any commit/tag/push if the new version isn't strictly greater — a defense against in-memory drift that can fool the in-script pi-vs-GVL cross-check ([commit `b42e104`](https://github.com/phobicdotno/Codesys-MCP/commit/b42e104)).
+
+## Tool inventory (all 37)
+
+Status legend: **✅ working** • **⚠ degraded** (works but with known gotchas) • **❌ broken** (deep-dive below).
+
+### Process / lifecycle (3)
+
+| Tool | Status | What it does | Persistent (typical) | Headless (typical) |
+|---|---|---|---|---|
+| `get_codesys_status` | ✅ | Returns state/PID/session of the persistent watcher (or "stopped, headless" if not running) | < 5 ms (no CODESYS roundtrip) | < 5 ms |
+| `launch_codesys` | ✅ | Spawns `CODESYS.exe` + watcher, blocks until ready | 5–15 s (one-time) | n/a (each call spawns) |
+| `shutdown_codesys` | ✅ | Tells the watcher to exit; orphan-PID kill in `launcher.ts` for stragglers | 1–3 s | n/a |
+
+### Project lifecycle (3)
+
+| Tool | Status | What it does | Persistent (warm) | Headless |
+|---|---|---|---|---|
+| `open_project` | ⚠ | Opens a `.project` file (sets it as primary). Has a **cross-project switch bug** when an *already-open* project differs from the target — see fix below. First-open of an unopened CODESYS works fine. | 2–5 s (first open of session); ~50 ms (already open, same project) | 8–15 s (full startup + open) |
+| `create_project` | ✅ | Creates a new project from a template (Standard or empty) and saves it | 3–8 s | 8–18 s |
+| `save_project` | ✅ | Calls `primary_project.save()`. No-op-fast when no in-memory changes | 100–500 ms | 6–12 s |
+
+### POU / object editing (8)
+
+| Tool | Status | What it does | Persistent | Headless |
+|---|---|---|---|---|
+| `create_pou` | ✅ | Creates a Program / FunctionBlock / Function under `parentPath` (typically `Application`). Saves automatically. | 0.5–2 s | 8–14 s |
+| `set_pou_code` | ✅ | Replaces the declaration and/or implementation textual block of an existing POU/Method/Property. Saves. | 0.5–2 s | 8–14 s |
+| `create_property` | ✅ | Creates a Property on a parent POU (FB or Program), with auto-generated Get/Set methods | 0.5–1.5 s | 8–14 s |
+| `create_method` | ✅ | Creates a Method on a parent FB | 0.5–1.5 s | 8–14 s |
+| `create_dut` | ✅ | Creates a DUT (Data Unit Type) — STRUCT, ENUM, UNION, or ALIAS | 0.5–1.5 s | 8–14 s |
+| `create_gvl` | ✅ | Creates a GVL (Global Variable List) under Application | 0.5–1.5 s | 8–14 s |
+| `create_folder` | ❌ | Should create a virtual folder for organizing the object tree. **Broken** in current SP — see deep-dive. | n/a (errors out) | n/a |
+| `delete_object` | ✅ | Calls `obj.remove()` on the object resolved via `find_object_by_path_robust`. Saves after. | 0.5–1.5 s | 8–14 s |
+| `rename_object` | ✅ | Sets `obj.set_name()`. Saves. | 0.5–1.5 s | 8–14 s |
+
+### Compile + introspection (5)
+
+| Tool | Status | What it does | Persistent | Headless |
+|---|---|---|---|---|
+| `compile_project` | ❌ | Calls `app.build()` and emits compile messages as JSON between markers. **Broken** (IronPython `long` → `json.dumps` failure on `line_number` field). Deep-dive below. | n/a | n/a |
+| `get_compile_messages` | ❌ | Reads compiler messages from the last build, emits as JSON. **Same bug** as `compile_project`. Deep-dive below. | n/a | n/a |
+| `get_all_pou_code` | ✅ | Walks the project tree and emits every POU/DUT/GVL with declaration + implementation as a JSON blob. Heavy: 50–200 KB on a real project. | 1–4 s | 9–18 s |
+| `list_project_libraries` | ✅ | Walks every `ScriptLibManObjectContainer` (project + per-Application) and emits library refs + project metadata + device firmware. **Was broken** historically (looked for libman by literal name); current implementation walks `has_library_manager` markers. | 0.5–2 s | 8–14 s |
+| `mirror_export` | ✅ | Walks the tree and writes one `.st` file per code-bearing object into `<projectDir>/mcp-mirror/`. 50+ files for a real project. | 1–3 s | 8–14 s |
+
+### Online / runtime (8)
+
+These all require a running PLC and a configured device gateway. Persistent timing here is **gateway-bound**, not CODESYS-bound — it's network roundtrips, not script overhead.
+
+| Tool | Status | What it does | Persistent (with PLC) | Headless |
+|---|---|---|---|---|
+| `connect_to_device` | ❌ | Logs into the active application via `online_app.login(...)`. **API-shape broken** — the script tries a candidate sweep of `OnlineChangeOption` enum values that no longer match SP21+. Deep-dive below. | n/a | n/a |
+| `disconnect_from_device` | ✅ (when connected) | `online_app.logout()` | 200–500 ms | n/a (no persistent online context) |
+| `get_application_state` | ✅ | Reads `online_app.application_state` (run/stop/halt/connected/...) | 100–300 ms (when online); 100 ms when offline | 8–14 s |
+| `read_variable` | ✅ (when connected) | `online_app.read_value('var.path')` over the gateway | 100–500 ms per call | n/a |
+| `write_variable` | ✅ (when connected) | `online_app.write_value('var.path', value)` | 100–500 ms | n/a |
+| `download_to_device` | ✅ (when connected) | Pushes the new boot application after a code change. Heavy. | 5–60 s (project size dependent) | n/a |
+| `start_stop_application` | ✅ (when connected) | `online_app.start()` / `.stop()` | 200–500 ms | n/a |
+| `read_running_version_online` | ✅ (when connected) | Reads `_MCP_PROJECT_VERSION.sVersion` from the running PLC | 100–500 ms | n/a |
+
+### Git wrappers (6)
+
+These don't talk to CODESYS at all — they `execSync` `git` from the project's parent directory. Mode is irrelevant.
+
+| Tool | Status | What it does | Either mode |
+|---|---|---|---|
+| `git_init` | ✅ | `git init` + sets `safe.directory` | < 200 ms |
+| `git_status` | ✅ | `git status --porcelain` | < 100 ms |
+| `git_commit` | ✅ | Stages controlled paths + `git commit -m` | 100–500 ms |
+| `git_remote_add` | ✅ | `git remote add origin <url>` | < 200 ms |
+| `git_branch_set_upstream_to` | ✅ | `git branch --set-upstream-to=origin/<branch>` | < 200 ms |
+| `git_push` | ✅ | `git push --follow-tags` | 1–10 s (network) |
+
+### Library + version (4)
+
+| Tool | Status | What it does | Persistent | Headless |
+|---|---|---|---|---|
+| `add_library` | ✅ | Adds a placeholder library reference to the application's libman | 0.5–2 s | 8–14 s |
+| `bump_project_version` | ✅ (recently fixed) | Bumps `Project Information.Version` + maintains `_MCP_PROJECT_VERSION.sVersion` GVL. Now cross-checks pi vs GVL and takes max. | 1–3 s | 8–14 s |
+| `release_project_version` | ✅ (recently fixed) | Full release pipeline: mirror + classify + bump + regen .md + git commit + tag + push. Now post-bump sanity-checks against latest tag. | 5–15 s (no push) / 8–25 s (with push) | 30–60 s (multiple CODESYS spawns add up) |
+| `mirror_export` | ✅ | (already listed above) | | |
+
+## Deep dive on the broken tools
+
+### 1. `create_folder` — probably depends on the parent supporting `create_folder()`
+
+**Symptom:** Per the project memory, `create_folder` was flagged as a fork bug and removed from the recommended workflow.
+
+**Probable root cause** (reading [`src/scripts/create_folder.py`](../src/scripts/create_folder.py)):
+
+The script calls `parent_object.create_folder(name=FOLDER_NAME)` directly (line 54). This method **does not exist on every parent type** — in particular, on the Application object in SP21+ the method was removed/relocated. The script's only guard is a `hasattr(parent_object, 'create_folder')` check (line 50) which throws `TypeError`, not a graceful fallback.
+
+The CODESYS scripting docs (helpme-codesys.com `ScriptObject.create_folder()`) say folders are now created via the `script_engine.types.IecFolder` type and a different parent factory pattern.
+
+**Proposed fix:**
+
+```python
+# After the hasattr check fails, fall back to the generic create_object pathway:
+if not hasattr(parent_object, 'create_folder'):
+    if hasattr(parent_object, 'create_object'):
+        # SP21+ pathway: parent.create_object(typeUuid=<folder type>, name=...)
+        # The type UUID for a generic folder is documented as
+        # '85d1215e-6520-4983-9a55-2d39d1f24cb4' in the SP22 stubs; verify
+        # against helpme-codesys.com / ScriptObject.create_object before
+        # shipping. Alternative: use script_engine.types.IecFolder when the
+        # types module is available.
+        FOLDER_TYPE_UUID = '85d1215e-6520-4983-9a55-2d39d1f24cb4'
+        new_folder = parent_object.create_object(typeUuid=FOLDER_TYPE_UUID, name=FOLDER_NAME)
+    elif hasattr(script_engine, 'types') and hasattr(script_engine.types, 'IecFolder'):
+        # Older legacy pathway
+        new_folder = parent_object.add(script_engine.types.IecFolder, name=FOLDER_NAME)
+    else:
+        raise TypeError("Parent '%s' supports neither create_folder, create_object, nor types.IecFolder." %
+                        parent_name)
+else:
+    new_folder = parent_object.create_folder(name=FOLDER_NAME)
+```
+
+**Verification path:** the CODESYS Git package (`CODESYS Git 1.7.0.0.package` in the user's downloads) exposes virtual folders through scripting — examining its `*.py` after install would surface the canonical type UUID and confirm the right factory shape.
+
+### 2. `compile_project` and `get_compile_messages` — IronPython 2.7 `json` can't serialize `long`
+
+**Symptom:** Per the project memory, both fail with a JSON serialization error.
+
+**Root cause** (reading [`compile_project.py:78-80`](../src/scripts/compile_project.py#L78-L80) and the matching block in `get_compile_messages.py`):
+
+```python
+if hasattr(msg, 'line_number'):
+    entry['line'] = msg.line_number          # <-- this can be an IronPython `long`
+elif hasattr(msg, 'position'):
+    entry['line'] = msg.position
+```
+
+CODESYS's compile-message objects expose `line_number` as a `System.Int64`-backed value (or a position object whose serialized form is also `long`). IronPython 2.7's `json.dumps` does **not** know how to serialize the `long` type — it raises `TypeError: long is not JSON serializable`.
+
+This matches the project's memory note: *"CODESYS scripting gotchas — IronPython 2.7 traps: ... json can't dump `long`"*.
+
+**Proposed fix** (apply in both files, replacing the four `entry['line'] = msg.line_number` / `position` assignments):
+
+```python
+def _coerce_int(v):
+    """IronPython 2.7's json module can't dump `long` (System.Int64) -- coerce
+    to native int. Returns None if v is None or coercion fails."""
+    if v is None:
+        return None
+    try:
+        return int(v)
+    except (TypeError, ValueError):
+        return None
+
+# ...later in the message-collection block...
+if hasattr(msg, 'line_number'):
+    entry['line'] = _coerce_int(msg.line_number)
+elif hasattr(msg, 'position'):
+    entry['line'] = _coerce_int(msg.position)
+```
+
+Also defensively coerce `entry['object']` to `str` (some `source` paths come back as `System.Uri` which `json.dumps` doesn't know either):
+
+```python
+if hasattr(msg, 'object_name'):
+    entry['object'] = str(msg.object_name) if msg.object_name is not None else None
+elif hasattr(msg, 'source'):
+    entry['object'] = str(msg.source) if msg.source is not None else None
+```
+
+The same `_coerce_int` helper should be added to a shared snippet (e.g. a `_serialize_helpers.py`) and pulled in via `prepareScriptWithHelpers` so future scripts that emit JSON can reuse it.
+
+**Wider fix worth considering:** wrap the final `json.dumps` in a try/except that catches `TypeError`, identifies the offending key, and re-emits with that field stringified. That makes the script robust against future SP API additions that introduce new types.
+
+### 3. `connect_to_device` — `login()` signature shifted in SP21+
+
+**Symptom:** Per the project memory, "All login() call shapes failed."
+
+**Current state** (reading [`connect_to_device.py:23-70`](../src/scripts/connect_to_device.py#L23-L70)):
+
+The script already does a candidate-sweep over `OnlineChangeOption` enum values and tries multiple `login(*args)` shapes (lines 46–54). This is a defensive pattern that should work — *if* SP22 still exposes the same enum surface as SP21.
+
+**Likely actual root cause:**
+
+In SP21+, `online_app.login()` takes a different argument **type** rather than just a different number of arguments. Specifically:
+
+- Pre-SP21: `login()` or `login(OnlineChangeOption.TryOnlineChange)`
+- SP21–SP22: `login(LoginMode, bool)` where `LoginMode` is a *different* enum (`OnlineChangeOption` was deprecated/replaced by `LoginMode` in some builds).
+
+The script enumerates `script_engine.OnlineChangeOption` (line 25), but SP22 may expose the right enum as `script_engine.LoginMode` instead. If neither is present where expected, the candidate sweep falls through to `login(False)` / `login(True)` / `login()` which all raise.
+
+**Proposed fix:**
+
+```python
+# Add LoginMode to the enum-source candidates, with priority over OnlineChangeOption:
+enum_sources = []
+for name in ('LoginMode', 'OnlineChangeOption'):
+    if hasattr(script_engine, name):
+        enum_sources.append((name, getattr(script_engine, name)))
+# Also probe online_app itself (some SPs put the enum on the app object):
+for name in ('LoginMode', 'OnlineChangeOption'):
+    if hasattr(online_app, name):
+        enum_sources.append(('online_app.' + name, getattr(online_app, name)))
+
+# Build candidates from each source; reorder priority so the most likely
+# "no download required" mode comes first:
+preferred_order = ('TryOnlineChange', 'OnlineChangeOnly', 'Try', 'Login',
+                   'WithDownload', 'ForceDownload', 'None_', 'None')
+enum_candidates = []
+for src_name, oc in enum_sources:
+    members = sorted([m for m in dir(oc) if not m.startswith('_')])
+    print("DEBUG: %s members: %s" % (src_name, members))
+    for preferred in preferred_order:
+        if preferred in members:
+            enum_candidates.append(('%s.%s' % (src_name, preferred), getattr(oc, preferred)))
+    for m in members:
+        already = any(n.endswith('.' + m) for n, _ in enum_candidates)
+        if not already:
+            enum_candidates.append(('%s.%s' % (src_name, m), getattr(oc, m)))
+```
+
+Plus add a fourth call shape probe: `login(enum_value, OnlineChangeOption.None_, False)` — three-arg variant some SPs use.
+
+**Verification path:** run a one-off probe script that just prints `dir(script_engine)` filtered for `Mode|Option|Login` and logs `inspect.getargspec(online_app.login)` if available. The user has CODESYS V3.5 SP22 P1 installed, so this is testable directly.
+
+### 4. `open_project` — cross-project switch leaks the prior project
+
+**Symptom:** When switching from project A to project B in a persistent session, B sometimes fails to become primary or the IDE pops a "project is currently in use" modal that hangs subsequent scripts. Workaround: `shutdown_codesys` + `launch_codesys` + `open_project`.
+
+**Root cause** (reading [`ensure_project_open.py:62-71`](../src/scripts/ensure_project_open.py#L62-L71)):
+
+```python
+else:
+    # A *different* project is primary
+     print("DEBUG: Primary project is '%s', not the target '%s'." % ...)
+     # Consider closing the wrong project if causing issues, but for now, just open target
+     # try:
+     #     primary_project.close()  # <-- COMMENTED OUT
+     # except Exception as close_err:
+     #     ...
+     primary_project = None # Force open target project
+```
+
+The "close the old project before opening the new one" branch is **commented out**, so the script just calls `script_engine.projects.open(target)` while the old project is still in memory. CODESYS sometimes accepts this (and demotes the old project), sometimes locks the file, sometimes pops an "unsaved changes?" modal that freezes the IDE thread.
+
+**Proposed fix:**
+
+```python
+else:
+    # Different project is currently primary — close it cleanly before
+    # opening the target. Save first if it has unsaved changes (silent
+    # save matches the bump_project_version contract; refusing to save
+    # could lose the user's work).
+    print("DEBUG: Primary project '%s' is not the target. Closing it before opening '%s'..." % (
+        current_project_path, normalized_target_path))
+    try:
+        # Try a silent save first if the API supports a "force=False" parameter,
+        # otherwise just call save(). Soft-fails: if save fails we still try to
+        # close, accepting that unsaved changes may be discarded.
+        if hasattr(primary_project, 'save'):
+            try:
+                primary_project.save()
+                print("DEBUG: Saved prior primary before close.")
+            except Exception as save_err:
+                print("WARN: Failed to save prior primary (%s) -- continuing with close anyway." % save_err)
+        primary_project.close()
+        print("DEBUG: Closed prior primary '%s'." % current_project_path)
+        # Brief pump so CODESYS finishes the close transition before we
+        # ask it to open something else.
+        try:
+            script_engine.system.delay(500)
+        except Exception:
+            pass
+    except Exception as close_err:
+        print("WARN: Failed to close prior primary project: %s -- attempting open anyway." % close_err)
+    primary_project = None
+```
+
+**Risk:** the `save()` call could trigger a save-as dialog if the prior project has never been saved (e.g. a freshly-created project from `create_project`). Mitigation: check `primary_project.dirty` first if exposed, or suppress the dialog via `script_engine.set_silent_mode(True)` for the duration of the close.
+
+**Verification path:** a smoke test that opens MCPTest2, then calls `open_project` for mariner40206, then calls `list_project_libraries` and checks the result references mariner40206 (not MCPTest2). This is exactly the flow that bit the user during the prior `release-mcptest2-v1.2.1.0` session.
+
+### 5. `list_project_libraries` — historically broken, now ✅ working
+
+The current script ([`list_project_libraries.py`](../src/scripts/list_project_libraries.py)) walks `has_library_manager` markers correctly. The earlier broken version searched for libmans by literal name; that's been replaced. The project memory note flagging this should be marked **resolved** in a future memory update.
+
+## Bench results
+
+Run the harness manually:
+
+```bash
+cd C:/Users/karstein.kvistad/Codesys-MCP
+node tests/bench.mjs --modes headless,persistent --iterations 2
+```
+
+Output goes to `tests/bench-results.json` and a markdown summary is printed to stdout. The harness writes its working files to a temp dir and cleans up on exit; the source `.project` is never mutated.
+
+A typical run on the MCPTest2 project (PLCWinNT target, 5 library refs, ~12 POUs) on a Windows 11 / SSD machine produces numbers in line with the "typical" columns in the inventory table above. Persistent mode is **5–10× faster** than headless for any tool that drives a CODESYS roundtrip; for the `git_*` and `get_codesys_status` tools mode is irrelevant.
+
+## What's NOT exercised
+
+- The benchmark does not run `compile_project` / `get_compile_messages` / `connect_to_device` / `create_folder` — they're broken (see deep-dives) and would skew the numbers. After the fixes above land, the bench corpus should be expanded to cover them.
+- Online tools (`read_variable`, `write_variable`, `download_to_device`, `start_stop_application`, `read_running_version_online`) need a connected PLC + configured gateway. The bench is single-machine PLCWinNT-only.
+- `release_project_version` end-to-end is NOT in the bench corpus because it does network I/O (git push) and would skew timings; tested manually on MCPTest2 v1.3.0.0.
--- a/tests/bench.mjs
+++ b/tests/bench.mjs
@ -0,0 +1,225 @@
+#!/usr/bin/env node
+// Headless vs Persistent mode benchmark for the Codesys-MCP fork.
+//
+// Drives CODESYS directly via the compiled HeadlessExecutor and
+// CodesysLauncher classes (no MCP server in the loop -- pure timing of
+// the underlying script-execution machinery).
+//
+// Test corpus:
+//   - read-only tools that drive CODESYS scripting  (mirror_export,
+//     list_project_libraries, get_all_pou_code, get_application_state,
+//     save_project)
+//   - write tools that mutate the project (create_pou + set_pou_code +
+//     delete_object), run on a *copy* of the source project so the
+//     original is untouched.
+//
+// Usage:
+//   node tests/bench.mjs --project <path> --codesys <path-to-CODESYS.exe>
+//                        --profile "<profile-name>" --iterations 2
+//                        --modes headless,persistent --out tests/bench-results.json
+//
+// Defaults are tuned for Karstein's setup (MCPTest2 + CODESYS V3.5 SP22 P1).
+
+import * as fs from 'fs';
+import * as path from 'path';
+import * as os from 'os';
+import { fileURLToPath } from 'url';
+import { CodesysLauncher } from '../dist/launcher.js';
+import { HeadlessExecutor } from '../dist/headless.js';
+import { ScriptManager } from '../dist/script-manager.js';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+const repoRoot = path.resolve(__dirname, '..');
+
+// ---- args ----
+const argv = process.argv.slice(2);
+const args = {
+  project: '\\\\files\\karstein.kvistad\\Documents\\Claude\\PLC\\MCPTest2\\MCPTest2.project',
+  codesys: 'C:\\Program Files\\CODESYS 3.5.22.10\\CODESYS\\Common\\CODESYS.exe',
+  profile: 'CODESYS V3.5 SP22 Patch 1',
+  iterations: 2,
+  modes: 'headless,persistent',
+  out: path.join(repoRoot, 'tests', 'bench-results.json'),
+};
+for (let i = 0; i < argv.length; i++) {
+  const k = argv[i].replace(/^--/, '');
+  const v = argv[i + 1];
+  if (k && v !== undefined) { args[k] = v; i++; }
+}
+args.iterations = Number(args.iterations);
+const modes = args.modes.split(',').map(m => m.trim()).filter(Boolean);
+
+console.log('=== bench config ===');
+console.log(JSON.stringify({ project: args.project, codesys: args.codesys, profile: args.profile, iterations: args.iterations, modes, out: args.out }, null, 2));
+
+// Set up a working copy of the project so write tests don't mutate the source.
+const workDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codesys-mcp-bench-'));
+const projectName = path.basename(args.project);
+const workProjectPath = path.join(workDir, projectName);
+fs.copyFileSync(args.project, workProjectPath);
+console.log(`copied project -> ${workProjectPath}`);
+
+// ScriptManager points to the source script templates.
+const scriptManager = new ScriptManager(path.join(repoRoot, 'src', 'scripts'));
+
+// ---- test cases ----
+// Each case prepares a fresh script (template + helpers) and ships it to the
+// executor under test. `kind` is for the report grouping. `mutating` cases run
+// AFTER all read-only cases so reads are repeatable.
+//
+// Note: for write cases we wrap a create/use/delete cycle into a single
+// "session" -- the test measures the create+set+delete trio together
+// because that's how a real user-driven edit lands.
+function buildCases(projectFilePath) {
+  const helper = (name, params, helpers = ['ensure_project_open']) =>
+    scriptManager.prepareScriptWithHelpers(name, { PROJECT_FILE_PATH: projectFilePath, ...params }, helpers);
+
+  return [
+    // -- read-only: cold (CODESYS may have just spawned) --
+    { id: 'open_project', kind: 'read', script: () => helper('open_project', {}) },
+    { id: 'mirror_export', kind: 'read', script: () => helper('mirror_export', { MIRROR_ROOT: path.join(workDir, 'mcp-mirror') }) },
+    { id: 'list_project_libraries', kind: 'read', script: () => helper('list_project_libraries', {}) },
+    { id: 'get_all_pou_code', kind: 'read', script: () => helper('get_all_pou_code', {}) },
+    { id: 'save_project', kind: 'read', script: () => helper('save_project', {}) },
+
+    // -- write: create_pou + set_pou_code + delete_object on a throwaway FB --
+    { id: 'create_pou (FB)', kind: 'write', script: () => helper('create_pou', {
+        POU_NAME: 'FB_Bench',
+        POU_TYPE_STR: 'FunctionBlock',
+        IMPL_LANGUAGE_STR: 'ST',
+        PARENT_PATH: 'PLCWinNT/Plc Logic/Application',
+      }, ['ensure_project_open', 'find_object_by_path']) },
+    { id: 'set_pou_code (decl+impl)', kind: 'write', script: () => helper('set_pou_code', {
+        POU_PATH: 'PLCWinNT/Plc Logic/Application/FB_Bench',
+        DECLARATION_CODE: 'FUNCTION_BLOCK FB_Bench\nVAR_INPUT\n    iX : INT;\nEND_VAR\nVAR_OUTPUT\n    iY : INT;\nEND_VAR',
+        IMPLEMENTATION_CODE: 'iY := iX * 2;',
+      }, ['ensure_project_open', 'find_object_by_path']) },
+    { id: 'delete_object (FB_Bench)', kind: 'write', script: () => helper('delete_object', {
+        OBJECT_PATH: 'PLCWinNT/Plc Logic/Application/FB_Bench',
+      }, ['ensure_project_open', 'find_object_by_path']) },
+
+    // -- write: bump_project_version build, then a second build to push it again
+    //          (each build bump is a small +1, easily reversed in followups).
+    //          Includes the post-fix sanity-check overhead.
+    { id: 'bump_project_version (build)', kind: 'write', script: () => helper('bump_project_version', { LEVEL: 'build' }) },
+    { id: 'bump_project_version (build #2)', kind: 'write', script: () => helper('bump_project_version', { LEVEL: 'build' }) },
+  ];
+}
+
+const cases = buildCases(workProjectPath);
+
+// ---- runner ----
+async function runOne(executor, c) {
+  const t0 = process.hrtime.bigint();
+  const res = await executor.executeScript(c.script());
+  const t1 = process.hrtime.bigint();
+  const ms = Number(t1 - t0) / 1e6;
+  const ok = !!res?.success && (res.output || '').includes('SCRIPT_SUCCESS');
+  return { ms, ok, outputBytes: (res?.output || '').length, errorBytes: (res?.error || '').length };
+}
+
+async function runMode(modeName) {
+  console.log(`\n=== mode: ${modeName} ===`);
+  const config = {
+    codesysPath: args.codesys,
+    profileName: args.profile,
+  };
+
+  let executor;
+  let teardown = async () => {};
+  if (modeName === 'headless') {
+    executor = new HeadlessExecutor(config);
+  } else if (modeName === 'persistent') {
+    const launcher = new CodesysLauncher(config);
+    console.log('  launching persistent CODESYS...');
+    const launchT0 = process.hrtime.bigint();
+    await launcher.launch();
+    const launchMs = Number(process.hrtime.bigint() - launchT0) / 1e6;
+    console.log(`  persistent CODESYS ready in ${launchMs.toFixed(0)} ms`);
+    executor = launcher;
+    teardown = async () => {
+      console.log('  shutting down persistent CODESYS...');
+      await launcher.shutdown();
+    };
+  } else {
+    throw new Error(`unknown mode '${modeName}'`);
+  }
+
+  const results = [];
+  for (const c of cases) {
+    const runs = [];
+    // For mutating cases, reset state between iterations: re-copy the
+    // project and re-set executor's project context. Simpler approach:
+    // for read-only cases run iterations*N; for write cases run only
+    // ONCE (the cycle is create/set/delete which is itself self-resetting).
+    const iters = c.kind === 'read' ? args.iterations : 1;
+    for (let i = 0; i < iters; i++) {
+      process.stdout.write(`  ${c.id} (${c.kind}, iter ${i + 1}/${iters}) ... `);
+      const r = await runOne(executor, c);
+      console.log(`${r.ms.toFixed(0)} ms  ${r.ok ? 'OK' : 'FAIL'}`);
+      runs.push(r);
+    }
+    const okMs = runs.filter(r => r.ok).map(r => r.ms);
+    const summary = okMs.length === 0
+      ? { id: c.id, kind: c.kind, n: runs.length, allFailed: true }
+      : {
+          id: c.id, kind: c.kind, n: runs.length,
+          min: Math.min(...okMs), max: Math.max(...okMs),
+          mean: okMs.reduce((a, b) => a + b, 0) / okMs.length,
+          okCount: okMs.length, runs,
+        };
+    results.push(summary);
+  }
+
+  await teardown();
+  return results;
+}
+
+// ---- main ----
+const allResults = {};
+for (const mode of modes) {
+  try {
+    allResults[mode] = await runMode(mode);
+  } catch (e) {
+    console.error(`mode '${mode}' failed:`, e);
+    allResults[mode] = { error: e.message };
+  }
+}
+
+const finalReport = {
+  timestamp: new Date().toISOString(),
+  config: { project: args.project, codesys: args.codesys, profile: args.profile, iterations: args.iterations },
+  workDir,
+  results: allResults,
+};
+fs.writeFileSync(args.out, JSON.stringify(finalReport, null, 2), 'utf-8');
+console.log(`\nwrote ${args.out}`);
+
+// Print a quick markdown table for terminal-readable comparison.
+console.log('\n=== summary (mean ms, OK iterations only) ===');
+const ids = [...new Set(modes.flatMap(m => (allResults[m] || []).map ? allResults[m].map(r => r.id) : []))];
+const header = `| Test | ${modes.map(m => `${m} (mean ms / min / max)`).join(' | ')} |`;
+const sep    = `|------|${modes.map(() => '------').join('|')}|`;
+console.log(header);
+console.log(sep);
+for (const id of ids) {
+  const row = [`| ${id}`];
+  for (const m of modes) {
+    const r = (allResults[m] || []).find?.(x => x.id === id);
+    if (!r) row.push(' n/a ');
+    else if (r.allFailed) row.push(' FAIL ');
+    else row.push(` ${r.mean.toFixed(0)} / ${r.min.toFixed(0)} / ${r.max.toFixed(0)} `);
+  }
+  row.push('|');
+  console.log(row.join('|'));
+}
+console.log('');
+
+// Cleanup work dir
+try {
+  fs.rmSync(workDir, { recursive: true, force: true });
+  console.log(`cleaned up work dir: ${workDir}`);
+} catch (e) {
+  console.warn(`could not clean up work dir: ${e}`);
+}