iris-hostgl: run command buffers on a GL thread of their own

[atomchild411]

Host GL answered every call on the emulated CPU's thread, so while it replayed a command
buffer the whole machine stood still: no instruction ran and no interrupt was taken. Big
buffers take 2-5 ms. IRIX then took its 1 kHz clock late ~15 times a second, its audio
driver refilled HAL2's DMA ring late, and sound crackled under any busy GL program, on
every loud low note of a demo, in Quake, and so on. It is the host-GL side of
rules/mgras/guest-gl-benchmarks-miss-board-time.md: whatever keeps the guest from taking
its ticks costs it time and sound.

What changes

The GL service runs on a thread of its own (iris-hostgl/src/threaded.rs), which owns
every context. The CPU thread only hands it work, through a bounded queue:

• OP_BATCH_ASYNC (new, 0x1_0010): the buffer is read on the CPU thread (a missing
  page is still a need-page answer), queued, and answered at once. Only buffers that read
  nothing outside themselves may be sent this way; one that tries is cut short and logged.
  A host without the thread treats it as OP_BATCH; an older host answers EINVAL, which
  is how the library finds out. The protocol number does not change.
• Every other call goes through the same queue, so calls stay in order, and the CPU
  thread waits for its answer. It lends the GL thread its view of guest memory for the
  call, so a call costs two hand-offs however much it reads.
• ENOTYET (new, 0x3001, iris-hostcall): a call from a client with buffers still
  queued is answered at once instead of waiting behind them; the library yields and asks
  again, so IRIX keeps running. Only a client that has sent OP_BATCH_ASYNC can have
  buffers queued, so older libraries never see it.

IRIS_HOSTGL_INLINE=1 keeps the service on the CPU thread, for comparison.

The guest half (sending self-contained buffers with OP_BATCH_ASYNC, retrying on
ENOTYET, IRIS_GL_ASYNC=0 to turn it off) is in iris-guest-tools,
gl/glshim_rt.c.

Measurements

Same features (hostgl,jitv2,tcache,lightning,gpujit), same disk, this tree with and
without the change. IP28, Max IMPACT, IRIX 6.5.22, macOS; the guest's libGL is the
host-GL library with async buffers.

	without	with
GL demo with synthesized sound: dropouts	19.7/s (31 ms lost/s)	3.7/s (4 ms lost/s)
Quake 3 demo four timedemo, 1024x768	30.2-30.5 fps	33.9 fps
glcheck (host GL)	PASSED, 227-238 fps	PASSED, 247 fps
gltest n32 / o32	see below	167/167, 167/167
irisgltest	all ok	all ok
boot to ssh, idle iris CPU	27 s, 208%	27 s, 208%
the demo's score with no graphics	0 dropouts	0 dropouts
SGI's libGL on the emulated board (glcheck)	40-45 fps	42 fps

The library on a host without the thread falls back to OP_BATCH: gltest 6 of 6 runs
clean. Libraries without OP_BATCH_ASYNC work unchanged. gltest's "the window kept its
pixels" fails now and then on either build (once in 12 runs without the change, with or
without the new library); it is not this change.

Tests: 4 new in iris-hostgl/src/tests.rs (buffers sent without waiting run in order
before a waited call; one that reaches for memory is cut short; on the CPU thread an async
buffer is a plain batch; a call behind queued buffers is told to come back). 31/31 on
macOS and on Linux (Debian 12 arm64, with a Linux backend that is not in this PR).
iris-hostgl is outside the workspace, so CI does not build it.

Notes

• The thread wants a host core beside the CPU thread. On a 4-vCPU VM it cost frame rate
  (Quake 3 22.8 -> 14 fps); IRIS_HOSTGL_INLINE=1 is there for such hosts.
• Emulated graphics boards do not go through any of this.
• See rules/hostgl/host-calls-hold-the-cpu-thread.md.

🤖 Generated with Claude Code