nikon/Codesys-MCP-SP21-plus
0
0
Fork 0
Code Activity Actions
Codesys-MCP-SP21-plus/tests/bench-results.json
Karstein Kvistad 2607063306 fix: 4 broken-tool fixes (compile json long, connect_to_device LoginMode, 
create_folder SP21+ fallback, ensure_project_open cross-project switch)
+ bench results

compile_project.py + get_compile_messages.py:
- IronPython 2.7's json.dumps cannot serialize System.Int64-backed `long`
  values, which is what CODESYS's compile-message objects expose as
  line_number / position. Added _coerce_int + _coerce_str helpers and a
  shared _build_message_entry function. Three duplicated message-building
  blocks collapsed into single helper calls.
- Defensive `try: json.dumps(...) except TypeError: json.dumps(default=str)`
  so a stray field that slips past the helpers doesn't kill the emit.

connect_to_device.py:
- SP21+ may expose the login enum as LoginMode rather than
  OnlineChangeOption. Extended the candidate sweep to probe both
  script_engine.LoginMode and script_engine.OnlineChangeOption AND
  online_app.LoginMode/OnlineChangeOption (some builds attach it to the
  app object). Added "OnlineChange" + "Login" + "Download" to the
  preferred-priority list. Added a 3-arg call shape variant for SPs
  that take (mode, secondary-mode, force-bool).

create_folder.py:
- parent_object.create_folder() is no longer exposed on every parent type
  in SP21+. Added two fallback factories tried in order:
    1. parent.create_object(typeUuid='85d1215e-6520-4983-9a55-2d39d1f24cb4', name=...)
    2. parent.add(script_engine.types.IecFolder, name=...)
  with detailed warnings when each path fails. Final TypeError now lists
  every factory tried so a future SP rotation surfaces clearly.

ensure_project_open.py:
- Uncommented the close-prior-project branch (was a TODO since the
  initial fork). Cross-project switches in a persistent CODESYS now do
  save() -> close() -> 500ms pump -> open(target). Without this,
  projects.open against a different already-primary project fails
  intermittently on file lock contention or pops a "project in use"
  modal that freezes the IDE thread.
- save() is best-effort: if it raises (transient lock, save-as required)
  we still proceed with close + open rather than getting stuck in a
  half-switched state forever.

tests/bench-results.json:
- Captured timings from a clean run on MCPTest2 (PLCWinNT, 5 lib refs,
  ~12 POUs). 9 tools x 2 modes x iterations.
  Headers (mean ms): persistent vs headless --
    open_project              7700  vs   40041   (~5x; first call cold)
    mirror_export             1547  vs   23723   (~15x)
    list_project_libraries    1565  vs   23322   (~15x)
    get_all_pou_code          1607  vs   23376   (~15x)
    save_project              2095  vs   23321   (~11x)
    create_pou (FB)           1540  vs   23903   (~16x)
    delete_object             1544  vs   27420   (~18x)
    bump_project_version      1540  vs   30678   (~20x)
    bump_project_version #2   1556  vs   37769   (~24x)
- set_pou_code FAILED in both modes -- bench harness param-shape issue
  (multi-line code passed verbatim to triple-quoted-string interpolation
  doesn't survive the round-trip). Tool itself works fine through the
  MCP tool call path; bench needs to escape newlines / use the same
  prepareScriptWithHelpers shape the server uses. Filed for follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 16:53:25 +02:00

394 lines
No EOL
8.5 KiB
JSON
Raw Blame History

{
"timestamp": "2026-04-26T14:52:18.324Z",
"config": {
"project": "\\\\files\\karstein.kvistad\\Documents\\Claude\\PLC\\MCPTest2\\MCPTest2.project",
"codesys": "C:\\Program Files\\CODESYS 3.5.22.10\\CODESYS\\Common\\CODESYS.exe",
"profile": "CODESYS V3.5 SP22 Patch 1",
"iterations": 2
},
"workDir": "C:\\Users\\KARSTE~1.KVI\\AppData\\Local\\Temp\\codesys-mcp-bench-HBzAUg",
"results": {
"headless": [
{
"id": "open_project",
"kind": "read",
"n": 2,
"min": 22012.8208,
"max": 58068.5602,
"mean": 40040.6905,
"okCount": 2,
"runs": [
{
"ms": 58068.5602,
"ok": true,
"outputBytes": 1131,
"errorBytes": 0
},
{
"ms": 22012.8208,
"ok": true,
"outputBytes": 1131,
"errorBytes": 0
}
]
},
{
"id": "mirror_export",
"kind": "read",
"n": 2,
"min": 23526.9205,
"max": 23919.1101,
"mean": 23723.0153,
"okCount": 2,
"runs": [
{
"ms": 23919.1101,
"ok": true,
"outputBytes": 1516,
"errorBytes": 0
},
{
"ms": 23526.9205,
"ok": true,
"outputBytes": 1516,
"errorBytes": 0
}
]
},
{
"id": "list_project_libraries",
"kind": "read",
"n": 2,
"min": 23288.2306,
"max": 23356.7295,
"mean": 23322.48005,
"okCount": 2,
"runs": [
{
"ms": 23356.7295,
"ok": true,
"outputBytes": 3981,
"errorBytes": 0
},
{
"ms": 23288.2306,
"ok": true,
"outputBytes": 3979,
"errorBytes": 0
}
]
},
{
"id": "get_all_pou_code",
"kind": "read",
"n": 2,
"min": 23359.3891,
"max": 23392.8083,
"mean": 23376.098700000002,
"okCount": 2,
"runs": [
{
"ms": 23392.8083,
"ok": true,
"outputBytes": 10895,
"errorBytes": 0
},
{
"ms": 23359.3891,
"ok": true,
"outputBytes": 10895,
"errorBytes": 0
}
]
},
{
"id": "save_project",
"kind": "read",
"n": 2,
"min": 23089.9258,
"max": 23551.2732,
"mean": 23320.5995,
"okCount": 2,
"runs": [
{
"ms": 23551.2732,
"ok": true,
"outputBytes": 1312,
"errorBytes": 0
},
{
"ms": 23089.9258,
"ok": true,
"outputBytes": 1312,
"errorBytes": 0
}
]
},
{
"id": "create_pou (FB)",
"kind": "write",
"n": 1,
"min": 23902.8924,
"max": 23902.8924,
"mean": 23902.8924,
"okCount": 1,
"runs": [
{
"ms": 23902.8924,
"ok": true,
"outputBytes": 3475,
"errorBytes": 0
}
]
},
{
"id": "set_pou_code (decl+impl)",
"kind": "write",
"n": 1,
"allFailed": true
},
{
"id": "delete_object (FB_Bench)",
"kind": "write",
"n": 1,
"min": 27419.9285,
"max": 27419.9285,
"mean": 27419.9285,
"okCount": 1,
"runs": [
{
"ms": 27419.9285,
"ok": true,
"outputBytes": 2872,
"errorBytes": 0
}
]
},
{
"id": "bump_project_version (build)",
"kind": "write",
"n": 1,
"min": 30678.0223,
"max": 30678.0223,
"mean": 30678.0223,
"okCount": 1,
"runs": [
{
"ms": 30678.0223,
"ok": true,
"outputBytes": 1746,
"errorBytes": 0
}
]
},
{
"id": "bump_project_version (build #2)",
"kind": "write",
"n": 1,
"min": 37769.477,
"max": 37769.477,
"mean": 37769.477,
"okCount": 1,
"runs": [
{
"ms": 37769.477,
"ok": true,
"outputBytes": 1746,
"errorBytes": 0
}
]
}
],
"persistent": [
{
"id": "open_project",
"kind": "read",
"n": 2,
"min": 740.7234,
"max": 14660.0126,
"mean": 7700.368,
"okCount": 2,
"runs": [
{
"ms": 14660.0126,
"ok": true,
"outputBytes": 1119,
"errorBytes": 0
},
{
"ms": 740.7234,
"ok": true,
"outputBytes": 770,
"errorBytes": 0
}
]
},
{
"id": "mirror_export",
"kind": "read",
"n": 2,
"min": 1545.8997,
"max": 1548.8912,
"mean": 1547.39545,
"okCount": 2,
"runs": [
{
"ms": 1548.8912,
"ok": true,
"outputBytes": 1155,
"errorBytes": 0
},
{
"ms": 1545.8997,
"ok": true,
"outputBytes": 1155,
"errorBytes": 0
}
]
},
{
"id": "list_project_libraries",
"kind": "read",
"n": 2,
"min": 1545.7068,
"max": 1583.9429,
"mean": 1564.82485,
"okCount": 2,
"runs": [
{
"ms": 1545.7068,
"ok": true,
"outputBytes": 3612,
"errorBytes": 0
},
{
"ms": 1583.9429,
"ok": true,
"outputBytes": 3612,
"errorBytes": 0
}
]
},
{
"id": "get_all_pou_code",
"kind": "read",
"n": 2,
"min": 1570.3402,
"max": 1642.7852,
"mean": 1606.5627,
"okCount": 2,
"runs": [
{
"ms": 1570.3402,
"ok": true,
"outputBytes": 10530,
"errorBytes": 0
},
{
"ms": 1642.7852,
"ok": true,
"outputBytes": 10530,
"errorBytes": 0
}
]
},
{
"id": "save_project",
"kind": "read",
"n": 2,
"min": 1538.7077,
"max": 2651.2456,
"mean": 2094.97665,
"okCount": 2,
"runs": [
{
"ms": 2651.2456,
"ok": true,
"outputBytes": 949,
"errorBytes": 0
},
{
"ms": 1538.7077,
"ok": true,
"outputBytes": 949,
"errorBytes": 0
}
]
},
{
"id": "create_pou (FB)",
"kind": "write",
"n": 1,
"min": 1539.7142,
"max": 1539.7142,
"mean": 1539.7142,
"okCount": 1,
"runs": [
{
"ms": 1539.7142,
"ok": true,
"outputBytes": 3088,
"errorBytes": 0
}
]
},
{
"id": "set_pou_code (decl+impl)",
"kind": "write",
"n": 1,
"allFailed": true
},
{
"id": "delete_object (FB_Bench)",
"kind": "write",
"n": 1,
"min": 1543.7014,
"max": 1543.7014,
"mean": 1543.7014,
"okCount": 1,
"runs": [
{
"ms": 1543.7014,
"ok": true,
"outputBytes": 2484,
"errorBytes": 0
}
]
},
{
"id": "bump_project_version (build)",
"kind": "write",
"n": 1,
"min": 1540.0761,
"max": 1540.0761,
"mean": 1540.0761,
"okCount": 1,
"runs": [
{
"ms": 1540.0761,
"ok": true,
"outputBytes": 1379,
"errorBytes": 0
}
]
},
{
"id": "bump_project_version (build #2)",
"kind": "write",
"n": 1,
"min": 1555.9909,
"max": 1555.9909,
"mean": 1555.9909,
"okCount": 1,
"runs": [
{
"ms": 1555.9909,
"ok": true,
"outputBytes": 1379,
"errorBytes": 0
}
]
}
]
}
}
View git blame Copy permalink