Native titles' thresholds came from Eden's device memory figure, which
is capped near 7 GB: on a tester's RTX 4070 SUPER (11.2 GB budget) title
1 kept evicting 800-1,600 textures per 600 frames with 3 GB unused. They
now follow the budget (critical at budget - 768 MiB, expected at budget
- 1536 MiB, re-read every 60 frames); the EDEN_NX2_TC_* overrides still
win, and the thresholds are logged only when they move by 256 MiB.
Title 1 calls both ~75 times a second with the display name "default".
The auto stub's reply (success, no data) works; it also logged two lines
per call, which made a tester's 18 minute log 50 MB. Same reply, no log.
It runs before most draws (constant buffer updates go through the
invalidation accumulator) and took the texture and buffer cache mutexes
one after the other.
- Compute engine macros replay repeated calls like 3D ones (title 1's
compute macro path fell from ~27% to ~14% of the GPU thread).
- AccelerateInlineToMemory takes the buffer and texture cache mutexes
together instead of one after the other.
- ShaderCache::InvalidatePagesInRegion looks at the rejected programs
only when the write can reach one (bounds kept as they are added);
it ran a locked scan for every inline upload and pre-draw
invalidation.
Inline uploads went from ~10% to ~4% of the GPU thread.
The flat map re-hashed the already mixed u64 keys with MSVC's byte-wise
FNV-1a (~1.4% of the GPU thread), and calls stored again after a data
RAM guard failed appended new records, so the arena filled with stale
copies and was cleared every few seconds. EDEN_NX2_MME_MEMO_STATS logs
the counters: in title 1, 97.3% of calls replay, 2.1% fail a guard.
- A channel waiting on a semaphore the CPU releases slept 1 ms per poll
(~25 waits a second in title 1); poll for 0.5 ms first.
- The GPU profile zones read the wall clock (the TSC) instead of
QueryPerformanceCounter, which cost ~5% of the GPU thread when on.
Title 1 makes ~1,000,000 native macro calls a second in 3D scenes, 7
instructions each on average, and 96% repeat an earlier call exactly.
A call whose effects depend only on its start, its parameters and the
data RAM words it read (no MME_READ, no memory reads, no re-read of
its own data RAM writes, no code upload while it ran) is remembered
with those reads as guards; a later call with the same start and
parameters whose guards still hold replays its data RAM and method
writes. Uploading macro code clears the memo. EDEN_NX2_MME_MEMO=0
interprets every call.
Checked offline against the previous interpreter on 200,000 recorded
3D-scene calls (method writes, registers and data RAM all identical,
94% replayed, 1.9x faster); in the game the macro zone fell from ~0.26
to ~0.20 us per call.
The per-600-present log only had the average frame time, which hides
shader-compile hitches; it now also gives the longest frame and how
many took over 100 ms.
Notes: today's profile of title 1 and what came of it (TSC on AMD,
rejected-shader rethrows, debug hooks, texture churn from VMA slack,
reactive flushing, macro interpreter).
Writes the first 200,000 native macro calls (start, parameters) and the
macro code, so an interpreter change can be replayed against the
previous interpreter (all 200,000 matched for the operand-slot rewrite,
which was then dropped as no faster).
Title 1's texture cache deleted 1,400-2,600 textures per 600 frames
and created them again: device memory sat at the collector's critical
line, but ~38% of it was empty space in 256 MiB VMA blocks. Freeing an
image returns that space only once its whole block is empty, so the
caches kept collecting without the usage dropping (one mass purge
deleted 4,300 textures).
- 64 MiB blocks on discrete GPUs too (as on integrated ones).
- GetDeviceMemoryUsage subtracts the unused bytes in VMA blocks, so
each eviction counts at once.
Now 0-224 deletions per 600 frames, ~4,400 textures resident, and
driver usage 6.24 GB of a 6.97 GB budget.
The scheduler lock is taken by every emulated core thousands of times a
frame (title 1's job system: WaitForAddress / SignalToAddress / yield)
and held for microseconds. As a std::mutex, a contended Lock put the
host thread to sleep at once. Now a three-state futex lock: one atomic
op uncontended, ~128 pause spins when contended, then
std::atomic::wait. Time blocked in Lock fell ~20-25% on title 1's cores.
- AMD CPUs have no CPUID leaf 0x15, so the TSC frequency stayed 0 and
every guest CNTPCT read went through QueryPerformanceCounter (~10% of
the game's main core in title 1). Measure it like Intel CPUs without
a crystal frequency.
- A rejected native shader was rethrown (two C++ throws) every time it
was bound; return false instead (~1-2% of the GPU thread).
- The semaphore watch, pushbuffer entry hashing, first-write trace and
per-macro debug logs ran on every command word or macro call; they
now cost a flag check unless their environment variable is set
(~4% of the GPU thread). Also starts the macro engine's profile zone.
EDEN_NX2_GPU_PROFILE=1 times the GPU thread's stages (command parsing,
macros, draw setup by step, compute, texture uploads) and logs each
one's share of every second.
EDEN_NX2_SAMPLE=<thread names> (Windows) pauses the named threads about
600 times a second for EDEN_NX2_SAMPLE_SECONDS, walks their stacks and
writes per-function counts plus folded stacks (EDEN_NX2_SAMPLE_OUT).
Sub 215 dropped the region's GPU mapping but kept the kernel pages 214
had allocated for it, and their SMMU mapping (cached per CPU range).
Title 1 creates and destroys these regions as scenes load, so the pool
ran dry within minutes and later regions got no memory at all (37
allocation failures in 85 s, 165 in a longer run): GPU writes to them
were lost.
Destroying a region now drops its GPU mapping, then (through a new
registry release step, in GPU-thread order) each address space's SMMU
mapping and cache entry, and frees the pages once the GPU thread is
past both. A 4-minute run destroyed 116 regions with no later failures
and renders cleanly; only the 5.8 GB virtual texture regions and three
170-340 MB regions at startup still do not fit the pool.
Read from the SDK's nn::codec VP9 decoder: the decoder service session
(vdec TIPC 1) takes 200 initialize (work buffer transfer memory), 101
configure (X: codec configuration, width, height), 3 its submission
event (copy handle at word 3), 6/207/8 start/stop/flush, 115 finalize,
311/314/212 register/release/unregister an output image (reply a
16-byte NativeSyncHandle; all zero reads as signalled), 109 submit a
frame (A: bitstream, u64 pts, u8 end of stream) and 213 the next
finished frame (image id, size, colour space, pts, end of stream).
Output images come from gsv 911, the same request as 429.
The service completes each submitted frame at once into the next free
image (no picture yet) and keeps its event signalled. Title 1 now sets
the decoder up without aborting; in the runs so far it has not submitted
frames yet. Auto-stubbed requests on nx2:vdec* sessions log 48 words.
- vi2 229 now waits for the previous frame's fence by default, so the
game prepares frame N+1 while the GPU thread runs frame N. Waiting for
the current frame serialized them: 8-56 ms of each 34-105 ms frame was
spent blocked. Title 1 now blocks under 1 ms per frame on average and
its GPU thread runs ~90% busy, which is the next limit.
EDEN_NX2_FRAMES_AHEAD=0 restores the old wait.
- syncpt 202 (cancel a slot request) signals the slot's event when the
request was still pending. NvOsEventPool waits on it again after its
~100 ms wait times out; title 1 hung there forever whenever a mid-frame
wait outlasted 100 ms with the GPU thread a frame behind (2 of 4
runs). Two 185 s runs with 2-3 cancels each completed.
- EDEN_NX2_VSYNC_HZ=<rate> sets the vi2 vsync event rate (experiment;
240 Hz did not shorten title 1's frames).
- gsv 901/1001 succeed and 902/1002 return sessions; their TIPC 1
(CreateVideoDecoderService) returns another, auto-stubbed. Title 1's
movie player no longer aborts with 2001-0114 (no frames are decoded).
- The present's wait for its fence is woken by the GPU thread through a
syncpoint action instead of 200 us sleeps (~1 ms each on Windows).
- Every 600 presents, log the average frame time and how much of it the
present thread spent blocked on the GPU fence, compositing and pacing.
- accelerate_astc=Gpu with astc_recompression=Bc1/Bc3 now decodes ASTC
with the GPU decoder into a temporary RGBA8 image (dedicated memory,
freed once the GPU is past it) and compresses it with a new compute
shader (bc_encoder.comp: bounding-box endpoints along the colours'
covariance, inset 1/16, nearest palette entry; BC3 alpha ramp; BC1
punch-through), then copies the blocks into the image. No CPU decode,
so streamed textures are ready the frame they are first used instead
of showing placeholders for seconds; the decode pass no longer waits
for the GPU on this path.
- The CPU path (CpuAsynchronous) fills a new image with zero blocks
before its first decode instead of showing stale VRAM, and the
transcode pool is a quarter of the hardware threads.
- Switch 2 titles: collector thresholds near the device budget, re-read
every 60 frames from VK_EXT_memory_budget (another game shrank it from
7.0 to 5.4 GB).
- EDEN_NX2_HIDE_WINDOW=0 now leaves the window visible.
The Turing macro interpreter re-extracted every field of each 96-bit
instruction on every execution and copied the register file per
instruction; title 1's GPU thread spent about half its busy time there.
Instructions are now decoded once per PC (re-decoded when their words
change) and both lanes' operands are read up front instead of copying
the registers. The parameter FIFO is a member instead of a vector
allocated per macro call. Rendering checked unchanged on the attract
loop; the GPU thread went from ~68% to ~45-50% busy.
- Unmapped GPU addresses get a stand-in CPU address fixed by the GPU
address. A fresh one per lookup meant FindImage never found an image
there again: title 1's bloom chain at 0xf0000-0x15b600 was re-created
on every use (~5,400 images per 600 frames), filling VRAM.
- ASTC whose blocks carry no alpha (exact endpoint-mode scan reusing the
decoder's block parsing) is re-compressed to BC1 instead of BC3, half
the size; about 70% of title 1's ASTC qualifies. Copies between BC1
and BC3 hosts are skipped (their guest data is uploaded instead).
EDEN_NX2_ASTC_BC1=0 turns it off.
- Deleting an image cancels every decode still queued for it. An image
re-decoded mid-decode has two; the first to finish cleared IsDecoding,
the second was not cancelled and was uploaded into the next image to
take the slot: garbage textures (the title logo, road patches) and a
staging overrun when that image was smaller.
- Diagnostics: images created and deleted per 600 frames by shape,
collector deletions, async decodes, BC1/BC3 split; collector
thresholds overridable (EDEN_NX2_TC_EXPECTED_MIB/CRITICAL_MIB); GPU
ASTC decode wait timing (EDEN_NX2_ASTC_FLUSH=1 submits without it).
Title 1 attract loop: device memory 4.8-5.1 GB instead of 5.9-6.1 GB, no
decoded textures evicted, ~17 -> 22-23 fps.
MiniDumpWithIndirectlyReferencedMemory left an empty file on Eden's
address space, so print the exception, address and stack as
module+offset before writing a smaller dump, and report a failed write.
A window close or quit request is now logged, to tell it from a crash.
nn::codec::CalculateDecoderImagePoolBuffer sends 991(preference, count)
and reads a size and an alignment. Auto-stubbed, the reply kept the
request words, the frame size read as 0x167027f'00000001 and title 1's
movie player aborted on its budget check when its first movie started.
Answer a block-linear NV12 size (4 bytes per pixel for other layouts).
TickAsyncDecode locked a finished decode's mutex, erased the context (freeing
the mutex) and only then let the lock go out of scope, unlocking freed
memory. With ASTC re-compressed on the CPU (accelerate_astc=CpuAsynchronous)
title 1 hit this a few minutes in: cdb reported "Free Heap block modified
after it was freed" at the mutex offset, and a run without the debugger
crashed. The lock is now released before the erase.
Also, deleting an image while it is being decoded (deletions other than the
collector, which already skips them) cancels the decode instead of
uploading into a freed slot.
Verified under cdb past 330 s (failed at 243 s twice before).
Under cdb title 1 reported "Free Heap block modified after it was freed"
when it recreated its display session (~4 min in); a run without the
debugger crashed at 113 s. The vi2 vsync timer callback (HostTiming thread)
captured the session and signalled its events; the destructor unscheduled
the timer (which does not wait for a running callback) and closed the
events. The syncpt slot actions (GPU thread) had the same race with
CloseEvent, as did the frame-capture and dump callbacks.
vsync events now live in state shared with the callback and are emptied
under its lock before closing; syncpt actions check a shared "alive" flag
under a mutex the destructor takes; capture/dump callbacks hold no session
pointers.
With half (8 on a Ryzen 7 8845HS), re-compressing streamed ASTC textures to
BC3 took ~3.5 cores away from the six emulated cores and the GPU thread.
Title 1 attract loop, 65-150 s: 8 threads 16.0 fps, 4 threads 12.6, 2
threads 18.9 (scene noise is large). EDEN_NX2_TRANSCODE_THREADS overrides.
Title 1 has ~700 ASTC 8x8 textures (~280 MB as stored); decoded to RGBA8
they took ~4.5 GB, pushing Eden past the 8 GB of an RTX 4070 Laptop
(8.4-9.8 GB in use) and the texture cache into permanent aggressive
collection. With astc_recompression=Bc3 (CPU, asynchronous) they fit
(4.6-5.6 GB). The two 8192x8192x68 terrain arrays would still not fit as
BC3, so ASTC 2D arrays over 1 GiB as RGBA8 always use the GPU decoder with
trimmed levels (RGBA8 host image, UNORM/sRGB views), and the decoder pass
now exists whenever the GPU lacks native ASTC.
Also logs every 600 frames: device memory against the collector thresholds,
the largest images, totals by format, and buffer cache totals.
Title screen 65-150 s: 10.1 -> 16.0 fps.
- eden-cli: SDL called SDL_AppIterate back to back, keeping a host core at
100% polling input; run the main callbacks at 240 Hz.
- EDEN_NX2_IDLE_SPIN_US: optional spin before an idle core blocks (measured:
no gain on title 1, so off by default).
- EDEN_NX2_SVC_TRACE=<first>:<last>: trace SVCs (with thread ids) for a range
of presents only.
- MME DMA reads flush GPU-written memory first (never triggered on title 1,
kept for correctness); render-enable override histogram instead of the
first-400 log.
Measured on title 1's title screen (RTX 4070 Laptop): GPU ~20% busy, the six
emulated cores ~0.9 host cores in total, Eden's GPU thread 45-70% busy and
largely waiting: on host fences for semaphore acquires (GPU accuracy High
delays every release to host completion), on ASTC upload Finish(), and in a
1 ms poll sleep. gpu_accuracy=Low: ~12.5 -> ~16 fps.
SM70+ removed RRO: compilers multiply the angle by 1/2pi (FMUL.RZ by
0.15915494) and MUFU.SIN/COS compute sin/cos(2pi x). Eden's Maxwell MUFU
computes sin/cos of radians (its RRO is a plain move), so every lowered sine
and cosine used the wrong angle (wind animation, rotations, the ocean FFT
twiddles). Scale the operand by 2pi first.
Title 1's terrain virtual textures are 8192x8192x68 ASTC 8x8 sRGB arrays with
12 levels: 1.5 GB in guest memory (fully backed) but 23 GB each as the RGBA8
Eden decodes ASTC to, so they were created unbacked and the near terrain
sampled black. A GPU-decoded ASTC 2D array over 1 GiB decoded now keeps only
the levels from the first one that fits in 512 MiB (here 3: 1024x1024, 380
MiB); host mip 0 is that guest level. Uploads read each layer's tail of the
mip chain into a compact buffer and decode only those levels, views shift
their level range, and these images are never downloaded. Normalized
coordinates and derivative LOD address the smaller image correctly; the
nearest terrain is lower resolution than on the console.
Also logs how unbacked images are used (views, render targets, blits, DMA).
The colour cast on title 1 (magenta/cyan, then red/green, changing per
course) came from the sky/atmosphere volume shader (a pixel shader drawing
a 32^3 RGBA16F table slice by slice), which starts with an unguarded R2UR.
R2UR, MATCH and HFMA2.RELU use P6 as scratch and saved/restored it with the
uniform-guard mask, 1 << (6 - 7) for an unguarded instruction: undefined, in
practice bit 31, which shifted into the opcode bits and turned the save into
another instruction. They now always save and restore P6.
Debugging aids used to find it:
- EDEN_NX2_DUMP=<present>:<addr>:<size>,...: flush GPU memory ranges at a
present (on the GPU thread) and save them next to the frame captures;
nx2work/vol3d.py untiles Switch 2 3D textures from such dumps.
- GPU::RunOnGpuThread.
- Logs of Switch 2 3D images and descriptors.
- With a BRX the jump targets live in a constant buffer, so capture and
lowering keep going past EXITs to the compiler's closing branch-to-self
(a title 1 compute program jumped to 0xae0, just past the first EXIT).
- 128-bit stores of RZ use R248-R251 (Eden wants 4-aligned data registers).
- The rejected-program cache covers environment construction, where native
programs are lowered (it never applied before), and logs the stage slot.
Title 1 reaches its title screen with the ground drawn.
- IMAD by a power of two lowers to SHL imm, the form Eden's jump-table
tracking expects (a title 1 compute program failed "track indirect branch").
- EDEN_NX2_REJECT=<hex addresses>: treat those native programs as rejected,
to bisect a behaviour change without rebuilding.
- A rejected graphics program logs its stage slot, header and slot registers
once (for the task/mesh pipeline setup).
- Notes: rows 87-92, title 1 status.
- LDS [Ra.Xn + URb]; ATOMG.ADD at a register address; RED at [URa+offset];
8/16-bit STG; REDUX.MAX; R2UR and REDUX into any uniform register.
- UIMAD immediate; F2I from a uniform register; FSET.BF combined with a
predicate; TXQ single component to an odd register; SHF.L.W.HI of an
immediate by a register.
Title 1's dumped programs: 1654 of 1667 lower (the rest: 12 task/mesh
programs using ISBERD/ISBEWR, and one SURED).
A rejected program was lowered again (and logged) on every draw or
dispatch: 1.7M log lines in 13 minutes of title 1. The shader cache keeps
the first exception per address until guest memory there is written, and
the error logs are deduplicated.
- RET: accept any copy of the call link into the return register in the
function (MOV or IMAD.MOV); library functions may EXIT; @Pg EXIT Pc.
- PLOP3 with any table (SEL to 0/~0, LOP3 sets the predicate), including a
uniform third source; UPLOP3 (with a uniform guard); UP2UR; ULOP3 with a
uniform predicate output and guard.
- UIMAD.WIDE.U32, uniform USHF, SGXT.U32, FMNMX with a uniform source,
I2F.U8 from a uniform register, I2F.U16 (.H1), F2I.FTZ floor/nearest,
TXQ width/height masks, FSWZADD any swizzle, PRMT any byte selection.
- LEA.HI with an immediate addend; 0x923 is ISBERD, not PIXLD (removed).
Title 1's dumped programs: 1379 -> 1446 of 1473 lower.
- IADD3 with a third source negated (bit 75).
- LEA.HI Rd, Ra, Rb, RZ: Rb plus the high word; bit 73 sign-extends.
- SHF with a uniform shift amount (0xc19) and SHF.R.U32 of the low word.
- Uniform registers UR0-UR21 in R216-R237 and UR22-UR27 in R240-R245 (pairs
stay adjacent); ULDC and uniform operands use the mapping.
Title 1: 1359 of 1434 dumped programs lower.
Title 1 samples an 8192x8192, 12-level, 68-layer array (a sparse virtual
texture, ~23 GB): vkCreateImage failed and Eden aborted, or VMA put it in
system memory and staged a full upload (34 GB private). Images over 2^28
texels are now created sparse-resident with no memory bound; the texture
cache neither uploads nor downloads them. Images merely over the VRAM
budget are allocated without the budget limit instead of failing.
Title 1 (768 dumped programs): 386 lowered cleanly, now 701.
- IPA: bits 76-77 are the sample mode (most varyings are centroid), mapped to
Maxwell bits 52-53.
- LDG: U8/S8/U16/S16 sizes as Maxwell's.
- Uniform predicates UP0-UP6 (UISETP outputs, USEL/VOTEU, uniform execution
guards) in R239 bit 6-n, swapped into P(6-n).
- PIXLD.COVMASK reads all-ones (no sample mask input; a mask write ANDs it).
- USHF.R.U32 of the low or high register.
- SULD/SUST 2D_ARRAY (dimension 4, as Maxwell's).
nn::gpu::sdk names 214 CreateSparseRegion and 215 DestroySparseRegion.
215 (range in words 3-6, job slot in word 8) drops the GPU mapping and
completes its job behind the GPU; it was auto-stubbed and never signalled.
Switch 2 titles map most of their memory twice plus GPU-only regions;
title 1 ran out of the 16 GiB SMMU space. The buffer cache page table and
memory tracker cover the wider space.
Macro code stays shared by the channels of a process, but each channel
now has its own MME data RAM. Title 1's channels keep their own QMD ring
base in data word 0x27; with one shared RAM every channel used the last
channel's ring, QMDs collided, releases came early or never and the GPU
waited on semaphores (4 s timeouts per frame).
The QMD scheduler state is one table for all channels: a QMD's
dependency counter is state of the QMD, scheduled and decremented from
different channels.
Diagnostics: EDEN_NX2_QMD_TRACE (scheduler loads, counts, launches) and
EDEN_NX2_MME_DUMP (compute macro code, calls, data word 0x27 changes).
