92 lines
9.1 KiB
Markdown
92 lines
9.1 KiB
Markdown
# Equivalent-workload performance comparison
|
||
|
||
`tools/compare_performance.py` validates and summarizes supplied CSV recordings. It does not collect frames, run an engine, authenticate measurements, demonstrate a faster successor, or grant acceptance. No real measurements accompany this tool. Its regression inputs are explicitly synthetic and exist only in temporary test directories.
|
||
|
||
Run:
|
||
|
||
```sh
|
||
python3 tools/compare_performance.py path/to/comparison.json > path/to/comparison-report.json
|
||
python3 tests/test_performance_comparison.py
|
||
```
|
||
|
||
Invalid or incomplete input exits with status 2 and a reason on stderr. A valid arithmetic report exits 0, with `acceptance: "not_evaluated"` regardless of measured/synthetic origin or positive/negative differences. Do not overwrite your manifest or CSV inputs by redirecting output to them.
|
||
|
||
## Collect real recordings later
|
||
|
||
Use the same hardware, engine version, capture tool/version, renderer/display mode, settings, map, timed route, player count, physical AI count, vehicle count and total entity count for both variants. Record the original scenario and successor's addon IDs and exact revision/build identity separately. Keep other workloads stable and document relevant drivers, CPU/power settings, resolution, graphics quality, VSync, frame cap, simulation settings, network topology and server settings in the settings object. Version the route specification, camera path and timing; use that stable route ID in metadata. A shared route name without a reproducible route is insufficient evidence.
|
||
|
||
Collect at least three independently executed paired repeats, alternating which variant runs first. Each capture must include at least 30 seconds of warm-up and 60 seconds of measurement, with identical declared durations across all runs. Hold declared populations constant during warm-up and measurement. Record every frame/tick, elapsed interval, current populations and AI simulation duration using the same profiler/instrumentation. Preserve raw capture files, full logs and collection notes next to the normalized CSVs. If the available profiler cannot export AI load and populations with each interval, add or choose suitable instrumentation before using this schema; do not fill the columns with guesses, interpolate missing frames, or manufacture stable counts.
|
||
|
||
Client frame-time comparisons and dedicated server tick comparisons are separate datasets. Their interval rates are both expressed per second in the report, but server tick rate is not rendered client FPS. Headless Workbench may only be compared against identically configured headless Workbench; such results cannot establish ordinary-client performance. The current scripted hidden fixtures teleport actors and use a capped renderer, so their existing logs are not suitable benchmark inputs.
|
||
|
||
An offline metadata check cannot prove that two captures are authentic or that all meaningful conditions were held constant. Review raw captures, population traces, route adherence, variance and repeatability before any performance acceptance decision. Positive deltas alone do not establish statistical significance. Keep the performance acceptance gate pending until real equivalent workloads have been measured and reviewed.
|
||
|
||
## Manifest schema, version 1
|
||
|
||
The top-level JSON object has exactly `schema_version` (integer 1), `sample_origin` (`"synthetic"` or `"measured"`) and `runs` (array). Each repeat ID must appear exactly twice: once as `baseline`, once as `successor`; at least three paired IDs are required. Use `"measured"` only for actual captured intervals. Duplicate JSON keys, non-finite values, missing/extra schema fields, reused CSV paths, changing build/addon identities within a variant and unequal conditions are rejected.
|
||
|
||
Each entry in `runs` has this shape. This is an illustrative record, **not a measurement or a complete accepted manifest**; replace placeholders with collection records and supply all six or more independent run entries:
|
||
|
||
```json
|
||
{
|
||
"variant": "baseline",
|
||
"repeat_id": "pair-01",
|
||
"artifact_id": "RECORD_EXACT_BUILD_OR_REVISION",
|
||
"addon_ids": ["RECORD_LOADED_ADDON_IDS"],
|
||
"engine_version": "RECORD_ENGINE_VERSION",
|
||
"hardware": {
|
||
"cpu": "RECORD_CPU",
|
||
"gpu": "RECORD_GPU_OR_NO_GPU_FOR_SERVER",
|
||
"ram_gib": 32,
|
||
"os": "RECORD_OS_BUILD",
|
||
"driver": "RECORD_DRIVER_OR_SERVER_NOT_APPLICABLE"
|
||
},
|
||
"workload": {
|
||
"map": "RECORD_MAP_RESOURCE_ID",
|
||
"route": "RECORD_VERSIONED_ROUTE_SPECIFICATION",
|
||
"players": 12,
|
||
"ai": 36,
|
||
"vehicles": 4,
|
||
"entities": 1000,
|
||
"settings": {
|
||
"RECORD_ALL_RELEVANT_SETTINGS": "replace this placeholder"
|
||
},
|
||
"warmup_seconds": 30,
|
||
"duration_seconds": 60
|
||
},
|
||
"capture": {
|
||
"kind": "client_frame",
|
||
"environment": "client",
|
||
"tool": "RECORD_PROFILER_OR_INSTRUMENTATION",
|
||
"version": "RECORD_CAPTURE_TOOL_VERSION"
|
||
},
|
||
"sample_file": "baseline-pair-01.csv"
|
||
}
|
||
```
|
||
|
||
Paths resolve relative to the manifest (absolute paths also work). `artifact_id` and `addon_ids` identify the intentional difference between baseline and successor; each variant keeps its own identity fixed through all repeats. `engine_version`, `hardware`, `workload` and `capture` must match exactly across every recording, including warm-up/duration. `settings` is a nonempty JSON object whose internal field names are collector-defined; document all relevant settings rather than relying on this nonempty-object check for completeness. `players` is a positive integer, the other populations are nonnegative integers, and total entities must cover at least declared players + AI + vehicles. Use a consistent entity-count definition excluding no populations silently.
|
||
|
||
Capture kinds are `client_frame` and `server_tick`. Environments are `client`, `dedicated_server` and `workbench_headless`. A dedicated server cannot provide client frames. Hardware fields must be recorded even if one is explicitly not applicable. JSON whitespace and object-key order do not affect equality.
|
||
|
||
## CSV schema
|
||
|
||
The exact header and column order are:
|
||
|
||
```text
|
||
frame_index,elapsed_ms,frame_time_ms,players,ai,vehicles,entities,ai_simulation_ms
|
||
```
|
||
|
||
- `frame_index` starts at 1 and increases by one without gaps or duplicates.
|
||
- `elapsed_ms` is the cumulative endpoint of the frame/tick, relative to the start of capture at 0; it must strictly increase.
|
||
- `frame_time_ms` is that entire interval's duration, finite and at least 0.001 ms. Its difference from adjacent elapsed endpoints, and cumulative total versus elapsed, must agree within 0.1 ms. Preserve sufficient precision when normalizing raw exports.
|
||
- Population columns are integer observations and must match declared counts in every row, including warm-up. Entity count includes the actors/vehicles.
|
||
- `ai_simulation_ms` is finite nonnegative AI profiling time for that interval. It may exceed wall interval time if the documented profiler sums concurrent worker durations. Baseline and successor must use the same definition/instrumentation.
|
||
|
||
Record through the first frame endpoint at or beyond `warmup_seconds + duration_seconds`; include that final boundary-crossing frame and stop there. Ending early or adding rows after that endpoint is rejected. Only whole frames starting at/after warm-up and ending at/before the measurement boundary contribute metrics. Boundary-crossing intervals remain in the raw capture but are excluded symmetrically from metrics. At least 120 complete measured intervals must cover **at least 99% of the declared measurement duration**. A stall crossing either warm-up or the final boundary that leaves lower coverage rejects the entire comparison; a short fragment cannot stand in for the declared workload. The report exposes `measurement_coverage_fraction` and each excluded boundary interval's full duration and overlap with the measurement window, alongside included sample counts and seconds. Preserve rejected captures as evidence of the stall; do not trim or move boundaries after seeing results merely to make a recording qualify.
|
||
|
||
## Report interpretation
|
||
|
||
Each repeated run reports median, p95 and p99 frame/tick duration in milliseconds; median, p05 and p01 interval FPS/rate; throughput (`included interval count / included interval seconds`); and median/p95 AI profiling milliseconds. FPS/rate percentiles are computed directly from `1000 / frame_time_ms`, not the reciprocal of mean duration. Percentiles use nearest rank (`ceil(percentile × sample count)`); medians use the conventional midpoint average for even counts. Thus reciprocal upper duration and lower FPS ranks may differ by one observation. Throughput is a different statistic from median instantaneous rate.
|
||
|
||
Variant aggregates are the **median of each per-run metric**, giving repeats equal weight; they are not pooled-frame percentiles. Paired repeat deltas report median-frame reduction, p99-frame reduction, and throughput-rate increase as percentages of baseline. Positive values indicate the named numerical reduction/increase within those supplied samples, negative values indicate regression. The final delta summary is the median of paired deltas. No confidence interval, significance test or automatic performance pass is produced. Input manifest and CSV SHA-256 hashes retain provenance for review, but hashes do not authenticate the origin of recordings.
|