74 lines
10 KiB
Markdown
74 lines
10 KiB
Markdown
# Runtime testing
|
||
|
||
## Keep the desktop unobstructed
|
||
|
||
Use terminal/file checks by default. Do not repeatedly launch Workbench or game windows on the main monitor. A visible editor should use the user's selected spare output. The user authorized the right-hand display (HDMI-A-1, 1920×1080 on this machine). Keep primary DP-3 free of test windows. A KDE rule matching `steam_app_1874910` places both editor and in-editor game windows on screen 2 before mapping and disables fullscreen. The previous configuration was backed up to `~/.config/kwinrulesrc.erri-backup`. The rule is named `erri-workbench-right-monitor` and can be removed independently in KDE Window Rules.
|
||
|
||
A later test found that Workbench can restore geometry after placement and still extend onto the main display. Automated local tests therefore now use a dedicated headless Gamescope display and a cloned disposable Wine prefix. The graphical application remains native Workbench; this is not a dedicated-server test. No performance comparison may use these headless results as equivalent to an ordinary client workload.
|
||
|
||
The additional session guard is `tools/desktop/kwin-right-monitor.js`; it minimizes matching test windows if HDMI-A-1 is unavailable. The original title-only guard missed the separate game window, so use the application-class match and initial-placement rule together. Do not claim placement is verified merely because the editor window moved. Verify every new game/dialog window. The API is documented by [KDE](https://develop.kde.org/docs/plasma/kwin/api/).
|
||
|
||
Existing missions and the original workspace are read-only references. Never save their open editor documents during successor validation. All authored successor content and fixes belong in this repository.
|
||
|
||
## Native setup
|
||
|
||
Use matching stable Arma Reforger and Arma Reforger Tools installations. Keep original Escapists and its extensions unloaded. Register `addons` as an addon search directory, and open `addons/chat/addon.gproj`. Open `Worlds/ERR_Everon.ent` in the World Editor. `ERR_ValidateWorldPlugin` logs the authored locations' terrain heights; `-errPlay 1` allows that diagnostic plugin to enter play mode when deliberately invoked.
|
||
|
||
The command-line conventions follow the [official startup parameter documentation](https://community.bistudio.com/wiki/Arma_Reforger:Startup_Parameters): Workbench uses `-gproj`, `-addonsDir`, `-load` and `-plugin`. Workbench's `-load` is distinct from the game's `-world`. Launching a visible editor is an explicit test step, not part of `tools/check.py`.
|
||
|
||
## Acceptance sessions
|
||
|
||
Record the game/Tools versions, repository tree hash, addon list, player count, AI/vehicle counts, settings, map positions and complete logs for every run. Start from an isolated profile.
|
||
|
||
1. Join with 1, 2 and 12 players; verify one controlled prisoner each. Join late during every phase. Reconnect with the same identity both within and after native reconnect reservation expiry. Voluntary departure must not create a free extra life. Test reused transient player IDs.
|
||
2. Test conscious, unconscious, revived and dead players; forbidden respawn requests; spectators; nobody connected; all survivors dead; partial extraction; defeat; restart. Voluntary reconnect reservation and spectator following are implemented; actual remote-client verification remains required.
|
||
3. Pick the prison lock, leave the enclosure, scavenge physical folders, reveal/identify the radio, hold its area for the full continuous defence period, locate the helicopter and cross the escape boundary aboard it. Test every interrupted stage and a destroyed extraction vehicle. Check that moving or collecting objective objects cannot move the objective.
|
||
4. Observe a player, interrupt a radio hop, remove/capture a relay, let a contact expire, lose a support group and deplete support ammunition. Verify orders reach only existing forces, are not duplicated and do not use undelivered knowledge. Aircraft/mortar/logistics cases remain blocked until their controllers are migrated.
|
||
5. Repeat restarts and long empty/occupied server periods. Compare entity, AI group, callback, waypoint, inventory and journal counts. Check JIP marker state and shared-map ownership/visibility.
|
||
|
||
## Performance
|
||
|
||
Use equivalent player/AI/vehicle workloads, settings, routes and hardware for old and successor runs. Include warm-up, repeat runs and collect frame times/server FPS with populations held constant. Record median and lower-percentile FPS plus AI simulation load and entity counts. Reducing AI population is not evidence of a faster equivalent system. No FPS improvement is claimed before those measurements exist.
|
||
|
||
Do not publish while an acceptance entry is pending or failed. Source-policy regression passes are not multiplayer or performance evidence.
|
||
|
||
## Opt-in scripted integration smoke
|
||
|
||
In local Workbench only, `-errSmoke 1` enables `ERR_NativeSmoke` alongside the validation plugin and `-errPlay 1`. Use a disposable profile: the test changes its intel and defence settings, makes the test player invulnerable, takes a scene screenshot and teleports the player and helicopter. It never runs in a release game build or dedicated server. It exercises real spawn, prison action, guaranteed physical cache documents, inventory-driven discovery, radio range, continuous occupancy, interrupted defence and physical helicopter seating. Automatic round restart reruns it.
|
||
|
||
Run `python3 tools/native_report.py <console.log> --require-completions 2` to summarize assertions and retain script errors as failures. Stop the isolated test process after the required cycles. Do not interpret completion as manual piloting, multiplayer or performance acceptance.
|
||
|
||
`tools/reference_guard.py snapshot` and `verify` can compare the original workspace's source/resource contents around a validation session. Keep the baseline outside the original workspace.
|
||
|
||
Use `tools/native_command.py workbench --exe <Workbench> --profile <disposable-profile> --smoke victory` to print the test command, or `--smoke lost-helicopter` for extraction-vehicle loss. `--smoke player-death` kills the real prisoner, checks one-life identity rules and camera activation, then waits for all-dead defeat and restart. The helicopter-loss case deletes the empty test helicopter during extraction and expects native mission defeat. `tools/archive_native.py <session-directory> <new-evidence-name>` retains raw logs, screenshots and current source hashes without overwriting earlier evidence. Stop the test before archiving and archive before changing the tested addon sources. Its nonzero exit preserves native failures.
|
||
|
||
## Windowless local runner on Linux
|
||
|
||
Clone a working Tools compatdata directory into a disposable path outside Steam, preserving symlinks (`cp -a --reflink=auto`). Never point the runner at the user's live Wine prefix. With Gamescope and Proton Experimental installed:
|
||
|
||
```sh
|
||
python3 tools/headless_workbench.py --compat /tmp/erri-headless-compat --profile /tmp/erri-workbench-profile --smoke radio-network --rounds 2
|
||
```
|
||
|
||
The runner uses Gamescope's headless backend, retains full GPU rendering on a private display, and terminates only its new process group. It rejects a concurrent editor using the same test profile, stops after failure/timeout/completion, and returns a nonzero status for any recorded native errors. `radio-network` exercises real radio equipment and interrupted delivery before the normal death/spectator test.
|
||
|
||
The user requested muted test audio. The runner creates/reuses only `erri_tests_silent`, a muted null output, sets a separate PulseAudio application ID, and verifies/reroutes/mutes test-owned streams during execution. It does not change the system default sink or other application streams.
|
||
|
||
## Core isolation
|
||
|
||
Pass `--project core --smoke victory` to `headless_workbench.py` to compile only Core and vanilla and run the separate base fixture. The full integration fixture still defaults to `--project chat`. Core now owns document recovery and objective markers; optional Intel distribution and the shared field map must not be required to advance the mission. Inspect the **Loaded addons** log section, not **Available addons**, when checking actual isolation.
|
||
|
||
## Deliberate startup failure
|
||
|
||
Use `--project core --smoke startup-failure --rounds 1` only for the negative-path fixture. It changes the test's in-memory faction catalog to an invalid combination, verifies that services stop and no players/sites/stock spawn, checks the administrator-visible reason and rejects further progression. The normal startup ERROR is expected in this test and is preserved in the native report. No source configuration or external mission is modified. This fixture does not restart, so exactly one round is required.
|
||
|
||
The Core victory fixture also holds its actual player character unconscious for 35 seconds at the radio, longer than the all-dead grace period. It checks that the physical survivor still prevents defeat, cannot advance radio progress, retains control/camera, and resumes defence after consciousness is restored without receiving another life. This uses the native controller's consciousness transition; it does not validate treatment animations, medical item consumption, bleeding or a remote revive action.
|
||
|
||
## Alternate scavenging route and visual discovery
|
||
|
||
Use `--project core --smoke cache-loss --rounds 2 --timeout 480` with the hidden runner. The fixture deletes the southern supply crate before taking any documents, then completes escape using the western cache. It verifies that neither deleted nor depleted stock is recreated, including across the usual preparation query. Both repeated rounds must independently start with fresh configured sites.
|
||
|
||
Core victory and cache-loss fixtures also inspect the three outpost barriers for upright placement and actual collision. At the medical stash they test unconsciousness, distance, a temporary physical wall, clear visibility and one discovery event. Only the fixture adds/removes that isolated wall; production discovery changes knowledge without changing scenery. The cache-loss variant uses the alternate documents, while the normal victory variant uses the original southern cache.
|
||
|
||
`--project core --smoke site-survey --rounds 1` prints a bounded terrain and obstacle survey for three areas without changing existing scenery. Candidate grid checks are placement diagnostics, not navigability or encounter-balance acceptance. Existing authored successor objects can now appear as obstacles in a repeated survey.
|