Files
jakub 6a667d8deb Capture current mission, AI, map and administration development
Include the current addon, terrain integrations, tools, regression tests and recorded development evidence.
2026-10-13 22:52:53 +02:00

8.8 KiB
Raw Permalink Blame History

TTS playback hitch investigation — 2026-10-13

The native benchmark reproduced a long synchronous preparation step before speech playback. Decoding a synthetic seven-second clip took 274 ms in one callback; a thirty-second clip took 1,833 ms. After spreading preparation across updates, the largest measured preparation call was 3 ms, with all 36 native checks passing in a04. These are native component timings measured with System.GetTickCount(), not inferred from download latency or a guarantee that every game frame is hitch-free.

Native baseline

Run a02 completed all three input sizes using the original production ERR_WelcomeVoiceRules.Decode, FileHandle.WriteArray and AudioSystem.PlaySound calls. The fixture used PCM mono, 24,000 Hz, 16-bit WAV files filled with silence. It ran locally through Workbench in a headless 1280×720/30 Hz session with a silent audio sink; it did not contact the speech service.

Synthetic clip Decoded WAV Decode Write PlaySound
1 second 48,044 bytes 41 ms 1 ms 0 ms
7 seconds 336,044 bytes 274 ms 2 ms 1 ms
30 seconds 1,440,044 bytes 1,833 ms 10 ms 0 ms

The timer has 1 ms resolution, so a zero measurement does not prove zero cost. The source calls the complete decoder from the REST callback before the next update can play the sound. A full synthetic clip was reconstructed correctly in each case: all three native byte-count assertions passed, and the preserved native output files exactly match the decoded inputs. The baseline fixture terminates each sound immediately after starting it; this does not qualify audible speech or playback over the full utterance.

The same fixture separately measured 2,048 direct ToAscii reads on the full encoded string. On the thirty-second input these took 893 ms at the start and 885 ms at the end. Those diagnostic loops run after the decoder timing and are excluded from the table. They caution against replacing bounded-string parsing with repeated character access on the whole response.

Run a01 recorded only the one-second case (40 ms decode), then stopped because the legacy extraction fixture lacked a safe landing footprint for the new rescue fleet. It is retained as a failed attempt. Between a01 and a02, the only changed launched addon file was Configs/ERR/Tests/LegacySites.conf; the speech source and benchmark fixture were identical. The second run used the already-surveyed airport extraction location.

The a02 runner reports smoke-completed, three passing assertions and no script/runtime failures. Strict native analysis still reports 21 engine/resource errors, including missing external localization references, obsolete prefab fields, the world GUID and a Steamworks warning. This is not an engine-clean certification. Raw errors and the exact launch inputs remain in each run directory.

Queued preparation

Run a04 passed the complete production-queue fixture after the ordering correction described below. It injected the same four local payloads: 1, 7 and 30 seconds, plus a second 30-second clip cancelled after a partial write. Only one clip received preparation work per update, with round-robin progress. Cancellation removed the partial file and queued work; completed audio waited for playback readiness and played exactly once. All three preserved native WAV outputs exactly match their decoded source inputs by SHA-256.

Synthetic clip Preparation calls Largest preparation call Largest output chunk PlaySound Queue-start to playback
1 second 24 3 ms 2,736 bytes 0 ms 0.691 s
7 seconds 200 2 ms 2,400 bytes 0 ms 4.799 s
30 seconds 1,256 3 ms 1,584 bytes 1 ms 18.376 s

Preparation now yields between small pieces; audio can still wait before it starts. The waits above are for several simultaneous clips sharing the queue, including the deliberately gated seven-second clip. They exclude input loading, HTTP and cloud synthesis and are not single-clip latency measurements. The batch finished in 18.385 seconds over 1,481 updates, with exactly one preparation job per update; the cancelled clip accounts for one job.

The seven-second WAV also passed the production soldier-speech spatial graph path: graph write 1 ms, initialization 2 ms, and PlayEvent 1 ms with a valid handle. This checks native graph compatibility, not actual soldier delivery, remote clients or audible quality.

The full game-script update still reached 110 ms, with a 120 ms gap between updates, because those measurements include other active mission work. The isolated preparation maximum of 3 ms therefore must not be presented as a whole-game maximum frame time. All 36 a04 assertions passed with one completion and no script/runtime failures. Its 21 strict engine/resource errors have the same normalized signatures and occurrence counts as baseline a02; they remain a separate limitation.

Run a03 injected four local payloads into the production preparation queue: 1, 7 and 30 seconds, plus a second 30-second clip cancelled after its first partial write. The three retained clips reached playback with a maximum measured preparation slice of 3 ms, and the seven-second clip waited behind the readiness gate. These are promising component measurements, but a03 failed a real queue-fairness assertion and is not a passing validation run.

The production queue used Enforce Remove(0), which fills the removed slot with the last element instead of preserving order. This violated the intended round-robin schedule; the earlier host test double had modeled ordered removal. Production removal was corrected to explicit ordered operations before a04 repeated the same native fixture. Only ERR_WelcomeVoice.c differs between the a03 and a04 launch manifests. The a03 failure happened before final output copying and spatial playback checks, so a03 provides neither completed output hashes nor a qualified spatial result. Raw partial timing and failure evidence are retained.

Existing client logs and separate admin-menu fix

Filtered client events preserve real playback timestamps without request payloads, authentication tokens or endpoints. On October 13, download-to-play gaps were 2.061 s, 1.465 s and 1.134 s. One other request failed with HTTP 0. The client performance logs contain only cleanup memory records, with no frame-time samples. These network-plus-preparation intervals do not establish the duration of a render hitch.

The newest session also produced 684 admin-menu warnings between 17:12:19.127 and 17:13:33.183: eighteen absent Random_ buttons requested on each of thirty-eight two-second snapshot polls. This affects every focused admin page; the TTS page merely appears in the inherited callback stack. ReceiveSnapshot now uses the same ERR_Settings.Randomizable(key) guard as menu initialization. The exact one-line patch and existing menu-test result are retained. No post-fix native UI frame-time improvement is claimed.

The final check rerun, after correcting ordered queue removal, retains successful focused checks followed by the existing ERRM_AtlasLeaf.et missing saved root ID failure. The earlier check is also preserved. The repository-wide check is therefore incomplete; the unrelated asset failure is not a TTS regression. All three changed production sources exactly match the passing a04 launch bytes.

Evidence and reproduction

The aggregate report contains structured timings and run status. Each run preserves the console, script and error logs, runner result, launch metadata, external addon inputs, relevant frozen source files, benchmark fixture and exact compressed synthetic inputs/outputs. input-output-sha256.json records sample sizes and hashes; artifact-sha256.json binds archived files.

archive.py validates native_provenance.read_launch and archive_source, including the original source capture, stopped raw-console hash, unchanged complete addon sources, confirmed process cleanup, idle prefix and unchanged external addon inputs. It refuses to overwrite an archived run. To append another stopped benchmark from the corresponding /tmp/erri-tts-bench-source-* and run-* directories:

python3 docs/evidence/tts-lag-2026-10-13/archive.py a05

Running the archiver with no arguments revalidates preserved artifact hashes and regenerates the aggregate report. It does not launch the engine, contact a service or modify the server. The synthetic benchmark is not a multiplayer latency test, a live server deployment or a rendered-frame percentile measurement.