EscapistsReintroduced/docs/performance-comparison.md

Equivalent-workload performance comparison

tools/compare_performance.py validates and summarizes supplied CSV recordings. It does not collect frames, run an engine, authenticate measurements, demonstrate a faster successor, or grant acceptance. No real measurements accompany this tool. Its regression inputs are explicitly synthetic and exist only in temporary test directories.

Run:

python3 tools/compare_performance.py path/to/comparison.json > path/to/comparison-report.json
python3 tests/test_performance_comparison.py

Invalid or incomplete input exits with status 2 and a reason on stderr. A valid arithmetic report exits 0, with acceptance: "not_evaluated" regardless of measured/synthetic origin or positive/negative differences. Do not overwrite your manifest or CSV inputs by redirecting output to them.

Collect real recordings later

Use the same hardware, engine version, capture tool/version, renderer/display mode, settings, map, timed route, player count, physical AI count, vehicle count and total entity count for both variants. Record the original scenario and successor's addon IDs and exact revision/build identity separately. Keep other workloads stable and document relevant drivers, CPU/power settings, resolution, graphics quality, VSync, frame cap, simulation settings, network topology and server settings in the settings object. Version the route specification, camera path and timing; use that stable route ID in metadata. A shared route name without a reproducible route is insufficient evidence.

Collect at least three independently executed paired repeats, alternating which variant runs first. Each capture must include at least 30 seconds of warm-up and 60 seconds of measurement, with identical declared durations across all runs. Hold declared populations constant during warm-up and measurement. Record every frame/tick, elapsed interval, current populations and AI simulation duration using the same profiler/instrumentation. Preserve raw capture files, full logs and collection notes next to the normalized CSVs. If the available profiler cannot export AI load and populations with each interval, add or choose suitable instrumentation before using this schema; do not fill the columns with guesses, interpolate missing frames, or manufacture stable counts.

Client frame-time comparisons and dedicated server tick comparisons are separate datasets. Their interval rates are both expressed per second in the report, but server tick rate is not rendered client FPS. Headless Workbench may only be compared against identically configured headless Workbench; such results cannot establish ordinary-client performance. The current scripted hidden fixtures teleport actors and use a capped renderer, so their existing logs are not suitable benchmark inputs.

An offline metadata check cannot prove that two captures are authentic or that all meaningful conditions were held constant. Review raw captures, population traces, route adherence, variance and repeatability before any performance acceptance decision. Positive deltas alone do not establish statistical significance. Keep the performance acceptance gate pending until real equivalent workloads have been measured and reviewed.

Manifest schema, version 1

The top-level JSON object has exactly schema_version (integer 1), sample_origin ("synthetic" or "measured") and runs (array). Each repeat ID must appear exactly twice: once as baseline, once as successor; at least three paired IDs are required. Use "measured" only for actual captured intervals. Duplicate JSON keys, non-finite values, missing/extra schema fields, reused CSV paths, changing build/addon identities within a variant and unequal conditions are rejected.

Each entry in runs has this shape. This is an illustrative record, not a measurement or a complete accepted manifest; replace placeholders with collection records and supply all six or more independent run entries:

{
  "variant": "baseline",
  "repeat_id": "pair-01",
  "artifact_id": "RECORD_EXACT_BUILD_OR_REVISION",
  "addon_ids": ["RECORD_LOADED_ADDON_IDS"],
  "engine_version": "RECORD_ENGINE_VERSION",
  "hardware": {
    "cpu": "RECORD_CPU",
    "gpu": "RECORD_GPU_OR_NO_GPU_FOR_SERVER",
    "ram_gib": 32,
    "os": "RECORD_OS_BUILD",
    "driver": "RECORD_DRIVER_OR_SERVER_NOT_APPLICABLE"
  },
  "workload": {
    "map": "RECORD_MAP_RESOURCE_ID",
    "route": "RECORD_VERSIONED_ROUTE_SPECIFICATION",
    "players": 12,
    "ai": 36,
    "vehicles": 4,
    "entities": 1000,
    "settings": {
      "RECORD_ALL_RELEVANT_SETTINGS": "replace this placeholder"
    },
    "warmup_seconds": 30,
    "duration_seconds": 60
  },
  "capture": {
    "kind": "client_frame",
    "environment": "client",
    "tool": "RECORD_PROFILER_OR_INSTRUMENTATION",
    "version": "RECORD_CAPTURE_TOOL_VERSION"
  },
  "sample_file": "baseline-pair-01.csv"
}

Paths resolve relative to the manifest (absolute paths also work). artifact_id and addon_ids identify the intentional difference between baseline and successor; each variant keeps its own identity fixed through all repeats. engine_version, hardware, workload and capture must match exactly across every recording, including warm-up/duration. settings is a nonempty JSON object whose internal field names are collector-defined; document all relevant settings rather than relying on this nonempty-object check for completeness. players is a positive integer, the other populations are nonnegative integers, and total entities must cover at least declared players + AI + vehicles. Use a consistent entity-count definition excluding no populations silently.

Capture kinds are client_frame and server_tick. Environments are client, dedicated_server and workbench_headless. A dedicated server cannot provide client frames. Hardware fields must be recorded even if one is explicitly not applicable. JSON whitespace and object-key order do not affect equality.

CSV schema

The exact header and column order are:

frame_index,elapsed_ms,frame_time_ms,players,ai,vehicles,entities,ai_simulation_ms

• frame_index starts at 1 and increases by one without gaps or duplicates.
• elapsed_ms is the cumulative endpoint of the frame/tick, relative to the start of capture at 0; it must strictly increase.
• frame_time_ms is that entire interval's duration, finite and at least 0.001 ms. Its difference from adjacent elapsed endpoints, and cumulative total versus elapsed, must agree within 0.1 ms. Preserve sufficient precision when normalizing raw exports.
• Population columns are integer observations and must match declared counts in every row, including warm-up. Entity count includes the actors/vehicles.
• ai_simulation_ms is finite nonnegative AI profiling time for that interval. It may exceed wall interval time if the documented profiler sums concurrent worker durations. Baseline and successor must use the same definition/instrumentation.

Record through the first frame endpoint at or beyond warmup_seconds + duration_seconds; include that final boundary-crossing frame and stop there. Ending early or adding rows after that endpoint is rejected. Only whole frames starting at/after warm-up and ending at/before the measurement boundary contribute metrics. Boundary-crossing intervals remain in the raw capture but are excluded symmetrically from metrics. At least 120 complete measured intervals must cover at least 99% of the declared measurement duration. A stall crossing either warm-up or the final boundary that leaves lower coverage rejects the entire comparison; a short fragment cannot stand in for the declared workload. The report exposes measurement_coverage_fraction and each excluded boundary interval's full duration and overlap with the measurement window, alongside included sample counts and seconds. Preserve rejected captures as evidence of the stall; do not trim or move boundaries after seeing results merely to make a recording qualify.

Report interpretation

Each repeated run reports median, p95 and p99 frame/tick duration in milliseconds; median, p05 and p01 interval FPS/rate; throughput (included interval count / included interval seconds); and median/p95 AI profiling milliseconds. FPS/rate percentiles are computed directly from 1000 / frame_time_ms, not the reciprocal of mean duration. Percentiles use nearest rank (ceil(percentile × sample count)); medians use the conventional midpoint average for even counts. Thus reciprocal upper duration and lower FPS ranks may differ by one observation. Throughput is a different statistic from median instantaneous rate.

Variant aggregates are the median of each per-run metric, giving repeats equal weight; they are not pooled-frame percentiles. Paired repeat deltas report median-frame reduction, p99-frame reduction, and throughput-rate increase as percentages of baseline. Positive values indicate the named numerical reduction/increase within those supplied samples, negative values indicate regression. The final delta summary is the median of paired deltas. No confidence interval, significance test or automatic performance pass is produced. Input manifest and CSV SHA-256 hashes retain provenance for review, but hashes do not authenticate the origin of recordings.

Acquisition boundary from the installed APIs

The 2026-09-19 read-only audit found a supported telemetry entry point in Game.OnUpdate(BaseWorld world, float timeslice), with native world frame numbers and System.GetProfileData(). A future neutral observer could record that update cadence and physical population without editing either mission. It must first validate interval semantics and collection overhead. System.GetFPS() and System.GetFrameTimeS() explicitly average ten frames; they are not raw slow-tail samples. ProfileData.SimulationTimeMs is total simulation time, not AI time; no AI-only timing getter/export was established. Unknown AI cost must remain unavailable, never zero or renamed total simulation cost, so the present strict CSV format cannot yet be populated completely from those APIs alone.

Supported -logStats already produced native timing/population summaries in the archived empty-server dedicated-2026-09-19-084040-vanilla run. These window summaries cannot reconstruct individual frames or AI cost. Existing private runners cap display/server cadence at 30 Hz/FPS; their records cannot demonstrate ordinary-client improvement. No new collector, benchmark or performance acceptance resulted from this audit.

The native server statistics exporter now extracts real coarse -logStats records from existing console logs, preserving source hashes and raw lines. It explicitly leaves unavailable AI-only duration and raw-frame percentiles unknown. Its output is useful for native monitoring, but cannot substitute for the matched frame/tick datasets required above.

HQ planning scan budget (2 October 2026)

The final pursuit selection pass now counts global active/pending commitments once after cleanup, retasking and missing-patrol investigation, then increments that local count only when a previously idle unit receives a pending order. Hunting occupancy remains separate from global occupancy; investigations and other existing orders still consume the global limit. Nothing is cached across ticks, and scheduling, radio delivery, reservations and force eligibility are unchanged.

An actual-method regression executes the entire final selection/commit loop across 360 deterministic scenarios. It compares selected orders, roles, destinations, bearings, deadlines, home reservations and hunt cooldowns against the previous repeated-count implementation. A 64-belief/128-unit no-dispatch case reduces the repeated Assigned() scans inside that selection pass from over 10,000 unit visits to 128. Earlier cleanup scans, eligibility checks and candidate scoring are outside that count. The old source fails the scan-budget assertion; an independently compiled mutation removing the increment fails behavior equivalence.

These are operation counts with controlled policy/native doubles, not measured frame time or FPS. Equivalent-workload engine performance acceptance remains pending. Current native integration and full-suite results are recorded in the change review.