Thanks for the info!
Punkline
(who uses CE)
So the RAM start randomization (is it truly random, or game dependent?) was introduced in build 5.0-3981, and then later removed in 5.0-5977? What was the point of it?
Also, what build was MemCheck first added to main? (I see that your commit was merged on Sept. 10, 2016, but I don't know where that falls on the build timeline.)
no the builds version number are the lattest stable release number followed by the number of commits since that stable release so 5.0-3977 is 4 commits before 5.0-5981 so the randomnisation was INTRODUCED in 5.0-3981, the lattest dev build that DID NOT had the randomnisation was 5.0-3977.
....I jsut relaised, I typoed the version number again, 5.0-5977 doesn;t exist sorry.
so let me be very clear
Lattest dev version BEFORE the randomnisation: 5.0-3977
First dev version to have the randomnisation: 5.0-3981.
I typoed this the second time lol sorry about that
As for the nature of the randomnisation, let me esplain.
Newer OS got a new security feature called ASLR (no, it has nothign to do with the closer to mics video
. Essentially, this feature will randomnise the location of executables int he RAM, this helps to protect against a buffer overflow. However, programs that wants to support this feature needs to be PIE compliant (position independant executable). The program has to be made in such a way that it can be compiled with PIE support and so, it will support ASLR if the OS supports it.
Now, you probably get the idea: Dolphin began to be PIE compliant since Dolphin 5.0-3981. In fact, here's the exact PR:
https://github.com/dolphin-emu/dolphin/pull/5271
As for why, well Dolphin is an ever evolving project so it's nice to ahve it support it and as devs, more security is better
Now, before I explain HOW it randomnises and how it impacts CE, I need to explain something very important: the CE setup you have to use (lke one of the methods I describe in the TASVideos thread I linked above), it is and has been VERY VERY fragile and only worked because of luck (unless you use the pointer method).
Basically, since you are ONLY searchign through the emulated RAM, you want to know where it starts in the emu process, you could then limit the range of the ram search and only search the gc memory. The thing is, that start address, due to a lot of circumstances put together, got very stable. On Linux, it was litterally hardcoded to 0x2580000000 because they couldn't exactly ask to have a mapping (something about error handling being complciated) and it ends with 0x80000000 because it just happens to be the first virtual address of the gc ram. On Windows, they asked the OS to give them the first available ram mapping that woudl be huge enough so it's not guaranted to be the same, but in practice and for unknown reaosns, it always seemed to be 0x7fff0000. I really do not know why it did this, but I do know some users did not had it stable, it would change a lot.
So, for complete luck, it was possible to use CE WITHOUT using any pointers. The pointer method btw, is you check the log of Dolphin (I made one available in every build that tells you what the start address is). With this start address, you use CE's pointer scan feature and you find a pointer within Dolphin that points to the start. The problem is one: it;s super fragile, it will not be the same pointer for each version of Dolphin so if you ever want to change version, you have to recalculate your ENTIRE table.....yikes. Two: it;s painful, you have to bind every addresses you find with that pointer meaning you have to compute using a hex calculator the difference between your address and the current start. I used this method for a year, it's terrible and I was so glad when I foudn out the new method.
So, about PIE, it is NOT game dependant, it is completely up to your OS, Dolphin and you cannot possibly predict the address. I tested on my arch linux AND on a Windows 10 vm, yeah, it is random, the only thing I noticed in common was it ends with 0000.
But you can know it by checking the log Mi and Memmap on notice level (pass Dolphin -l or -d). The other way is you can check the patterns in the mapping info of Dolphin's process like on Linux, you cat the file /proc/[pid]/maps where pid is the current pid of Dolphin.
The point is, CE CANNOT figure out on its own the start on its own without something like a fancy script or the pointer method. This was the last thing I needed to hear before I start my own RAM search since if I do this, I can make assumptiosn on how Dolphin work and you wouldn't have to deal with the start address EVER.
If you want more info, I wrote a blog post some time before I started the project about my frustration with using CE and Dolphin, this isn't the msot aggravating problem, the most annoying one is actually you can't track dynamic memory.
https://aldelaro5.wordpress.com/2017/06/19/the-annoyance-when-no-ram-search-works-with-an-emulator/
As for the first build where the MemCheck was first supported on every build, it is 5.0-583
https://dolphin-emu.org/download/dev/a15b3fda6e5c56388989704dfa82f3b029101843/
HOWEVER I DO NOT RECOMMEND TO USE THIS BUILD FOR REVERSE ENGINEERING!!!
I did A LOT after this build, I would recommend minimum 5.0-3023, but as I just said, you can go up to 5.0-3977 if you don't want to deal with the pointer method. There is small improvements actually that happened after 3023 that were msotly not mine, but after thsi build, I considered the debugger to be "stable". So you can go between these 2 version and you shoudl be fine.
btw., in case you notice new stuff that mgith interest you on using the debugger, might as well link the TASVideos thread I made on using the debugger (I don't teach PowerPC, but I link some ressources to do so, honestly, I only learned it in 3 days, it's such a simple ASM tbh.
http://tasvideos.org/forum/viewtopic.php?t=18555
I am sorry if the whole thign with versions sounds confusing, but believe me, it's nothing like it was before when I found where the debugger got broken (it's a mess, it got broken somewhere, got semi fixed, got broken again compeltely and it maintained this way from since somewhere in 4.0).
And as for my RAM search, I am aiming to solve most of the problem CE has. IMO, a ram search shoudl be convenient and painless to use, I can say that so far, I am programming the watcher and it's going very well, if you are interested, I can post my github repos once I release the first beta version
(it is going to be open source for sure, MIT license to be exact).