Wanting to treat this posts a bit like a blog an explain the process/developments. I may transition to using a GitHub page to do longer posts with pictures. So here's some recent development.
---
I've been doing a lot of debugging on the FIFO, which is essentially a range of memory where the CPU writes GPU commands that are then pushed to the GPU. Dolphin has a FIFO log, so that I can compare what Melee does to the Title vs. what FRAY is doing. When the single frame render was occurring, I was seeing bad lighting updates, a missing fog update, and completely different TEV changes. Texture objects were being put in place, but not all of them were living and were out of order.
To begin debugging that, I used CPPCheck and some other static analyzers, as well as Clang's own warnings, to debug issues with the code. In doing so, I discovered actual bugs in the HSD Lib, which were present all the way to Killer7's version (Some of which is in the available leaked code, such as TObj). With those fixed, I also caught some issues with for loops not updating properly, unnecessary null checks within for loops, and some other minor issues.
From there, I fixed the lighting updates (turns out I needed to map indexes to Light IDs, as HAL had done), determined the missing fog update was the result of a GX Proc that being in debug mode sets, and have yet to determine the cause of the TEV issues. In the course of these fixes, I got Clang to compile the project completely over devkitPro's GCC, which reduces the file size by about 30KB and showed there was undefined behavior and stack corruption issues occurring that GCC was avoiding by virtue of being a piece of **** compiler.
I began debugging the stack corruption using a stack guard via the flag "-fstack-protector-all," which is available on GCC and Clang. What a stack guard does is at the end of every function, if the stack is improper it'll throw an error message about a stack smash and abort the running program. Using those guards, I could set a breakpoint on the stack_chk_fail function and look at the call stack to see what function failed the check. Once I caught the breakpoint, the stack corruption appeared to be coming from JObjLoad. I didn't see any noticeable side effects from Melee's version, but I changed to a disassembled version from K7 instead of Melee's as it was slightly more succinct. I think it was fixed as a result of the stack corruption I was seeing.
After that, I was looping and eventually crashing from HSD_VIXFBCopyAsync, as the GXRModeObj (which is essentially a one of a set of hardcoded structs in the SDK containing NTSC/PAL screen data), was sometimes returning NULL. I did not determine the cause and began double-checking all of my VI functions against Killer7's version, which lead to me adding some inlined functions back. Killer7's version *is* significantly different in a few ways, which I suspect is again the result of bug fixes.
I also looked into Melee's custom exception handler and determined what was causing Libogc's crash handler to not show. I've addressed it and prior to the following issue, I was actually seeing my stack traces again on the rendered console. Getting my custom exception handler to fully take over would require writing some assembly, which at present I have no interest in as Libogc's does the job just fine. Once FRAY is at a "public release" stage, I will look into doing that as I can print out build date information and know if someone isn't running the latest version when an issue is reported.
---
After all the above fixes and changes, the game is looping and rendering nothing at 60 FPS. According to the FIFO log, it is pushing proper frame data at the beginning of the first object each frame, but it does not go through the rest of the render pipeline. As a result, no fog is set, etc.. This is obviously bad, but given that each frame is now containing non-garbage data in that first object, fixing this should hopefully result in the image persisting and updating every frame. Plus, it hopefully will resolve the case of the missing textures.
At this point, I believe it's likely the GX Procs not running, which may be a manifestation of a separate bug. I'm also experiencing a crash from a function that should never run after the main loop has run for several frames, which I'll also be looking into.
Edit: So, procs are running. Rendering is not. Also the crash is a direct result of the loop that handles when the game should transition to the demo.
Savestate
has previously documented.
