Programmatically working with ISOs

[DRGN]

This thread is for Python IPC (inter-process communication) with command line, specifically in this case (though you could use it for other things) to work with ISOs. This will give our programs some very nice power and convenience. This is only going to be useful to a few individuals, but it's a hot topic atm so I thought I'd create a thread to discuss it. In

Example functionality: GALE01.iso file offsets and lengths:

Spoiler: partial example only!

3ba48000+ 283b6 164790 P-GALE/files/TyTeresa.dat
3ba78000+ 7ed7 32471 P-GALE/files/TyThnder.dat
3ba80000+ 12cbe 76990 P-GALE/files/TyTogepy.dat
3ba98000+ b4d3 46291 P-GALE/files/TyToppi.dat
3baa8000+ b341 45889 P-GALE/files/TyTosaki.dat
3bab8000+ 2d1c 11548 P-GALE/files/TyTosanz.dat
3bac0000+ 3874 14452 P-GALE/files/TyTotake.dat
3bac8000+ 6977 26999 P-GALE/files/TyTwinkl.dat
3bad0000+ 357b 13691 P-GALE/files/TyUfo.dat
3bad8000+ 35e57 220759 P-GALE/files/TyUKnown.dat
3bb10000+ 9960 39264 P-GALE/files/TyUsahat.dat
3bb20000+ 1d7d7 120791 P-GALE/files/TyUsokie.dat
3bb40000+ 383f1 230385 P-GALE/files/TyVirus.dat
3bb80000+ 95b57 613207 P-GALE/files/TyWalugi.dat
3bc18000+ 1d1b4 119220 P-GALE/files/TyWanino.dat
3bc38000+ 5a611 370193 P-GALE/files/TyWario.dat
3bc98000+ 106eb4 1076916 P-GALE/files/TyWaveRc.dat
3bda0000+ 29b82 170882 P-GALE/files/TyWdldee.dat
3bdd0000+ 22b3e 142142 P-GALE/files/TyWolfen.dat
3bdf8000+ 297c 10620 P-GALE/files/TyWpStar.dat
3be00000+ e4c5 58565 P-GALE/files/TyWtBear.dat
3be10000+ 410a4 266404 P-GALE/files/TyWtCat.dat
3be58000+ 106eb4 1076916 P-GALE/files/TyWvRcUs.dat
3bf60000+ 29f46 171846 P-GALE/files/TyWwoods.dat
3bf90000+ 61609 398857 P-GALE/files/TyYoshi.dat
3bff8000+ 44c9e 281758 P-GALE/files/TyYoshiR.dat
3c040000+ 43235 274997 P-GALE/files/TyYoshR2.dat
3c088000+ 60f1d 397085 P-GALE/files/TyZelda.dat
3c0f0000+ 6aed6 437974 P-GALE/files/TyZeldaR.dat
3c160000+ 3efe7 258023 P-GALE/files/TyZeldR2.dat
3c1a0000+ 12c4c 76876 P-GALE/files/TyZeniga.dat
3c1b8000+ ff4f 65359 P-GALE/files/TyZkMen.dat
3c1c8000+ 1f598 128408 P-GALE/files/TyZkPair.dat
3c1e8000+ fda1 64929 P-GALE/files/TyZkWmen.dat
460000+ 0 0 P-GALE/files/usa.ini
49d88000+ 80a0 32928 P-GALE/files/Vi0102.dat
49d98000+ a564 42340 P-GALE/files/Vi0401.dat
49da8000+ 1b304 111364 P-GALE/files/Vi0402.dat
49dc8000+ 13bcc 80844 P-GALE/files/Vi0501.dat
49de0000+ ebc 3772 P-GALE/files/Vi0502.dat
49de8000+ 30030 196656 P-GALE/files/Vi0601.dat
49e20000+ 1774 6004 P-GALE/files/Vi0801.dat
49e28000+ 8444 33860 P-GALE/files/Vi1101.dat
49e38000+ 6ca 1738 P-GALE/files/Vi1201v1.dat
49e40000+ 56a39 354873 P-GALE/files/Vi1201v2.dat
49e98000+ 610 1552 P-GALE/files/Vi1202.dat
0 74 116 P-GALE/setup.bat
0 65 101 P-GALE/setup.sh
0 df 223 P-GALE/setup.txt
- - N=5 P-GALE/sys/
2440+ 1c2f4 115444 P-GALE/sys/apploader.img
440+ 2000 8192 P-GALE/sys/bi2.bin
0+ 440 1088 P-GALE/sys/boot.bin
456e00+ 1d4a4 119972 P-GALE/sys/fst.bin
1e800+ 4385e0 4425184 P-GALE/sys/main.dol

That's just a partial snippit because the whole list is huge. But you get the idea.

First value is the file's offset in the ISO, second is the file size in hex, third is file size in decimal, fourth part is the directory structure of the file in the ISO. You might notice some oddities, but I'll get to those in a minute. Important to note that the offsets and some file sizes are different for different game versions, so you'd need to plan for that if you want to work with more than one game version.


How did I generate this?

BAM: http://wit.wiimm.de/features.html

The exact command I used for the list above was:
wit.exe -t files "H:\Games\GameCube\SSB Melee\v1.02 (original)\Super Smash Bros. Melee (v1.02).iso" --no-header --show=offset,size,files

Know that you could hardcode certain operations, like extraction/insertion, into a program if you want though. Like for example if you want to create an auto-installer for a pack of files (auto-20XX installer, anyone?); you actually wouldn't even need to incorporate this into your program. If you know the offset and length of a file, you can get that file into your program with Python (2 or 3) via:

with open( filepath, 'rb' ) as isoBinary:
   isoBinary.seek( offset )
   fileBinary = isoBinary.read( fileLength ).encode("hex")

You could then do whatever you like with it, and then put it back and/or create a file out of it.

However, if you want to do some really powerful stuff, you might want to create that list of files dynamically within your program, or run other cmd commands. To do stuff like that (and a whole lot more), I wrote this function to communicate with command line in Python 2.7:

def cmdChannel( command ):
    try:
        outputStream = subprocess.check_output( command, shell=True, stderr=subprocess.STDOUT, creationflags=0x08000000 )
        status = 'success'
    except subprocess.CalledProcessError as error:
        outputStream = str( error.output )
        status = str( error.returncode )
    return ( status, outputStream )

It's slightly modified from a function I had in DTW2. In Python 3.4, it's almost the same; you just don't need the "shell=True" part. It returns a tuple, with either "success" or the exit code if the process failed, along with the process output. Took a while to settle on this simple function when I was first trying to get everything working.

So if you want to use this to get an ISO file listing like above, you'd use something like this:

( status, outputStream ) = cmdChannel('bin\\wit.exe -t files "' + filepath + '" --no-header --show=offset,size,files')

and then you could do "lines = outputStream.splitlines()" get an iterable list of each line/file. And separate individual lines with, say, line.split(), which in this case would give you a list in the form of [fileOffset, hexFilesize, decimalFilesize, isoFilepath]


Now, as for those oddities I mentioned from the file list:

1) dat files, audio files, and basically all others I looked at appear to have the same file sizes as GCRebuilder. But some files, namely those in the sys folder, have different file sizes. I don't think this is a problem though because it appears to just be padding. Importing/exporting with vs. without the padding should be irrelevant.
2) some different files in the system folder than what GCRebuilder shows? da fuq? idk why that is, but we typically only care about the dol so far anyway, so I don't see it affecting us.
3) setup.bat, setup.sh, setup.txt, all with an offset of 0? strange and interesting. Idk what these are and haven't tried extracting them yet.​

I've only just started testing wit, so if you guys find out some interesting things with it or whether specific commands do or don't work, feel free to share here.

 

[Doq]

Compiling a GC ISO using wit is impossible. All combinations of commands produce a Wii image.

That is all I know.

 

[CeLL]

Is there a way to get wit working without installing it, like wimgt in DTW?

 

[DRGN]

Compiling a GC ISO using wit is impossible. All combinations of commands produce a Wii image.

That is all I know.

Have you tested it? There are options to change the output formatting. WBFS is default, but can be changed to ISO, which I feel like should work.

Do you know what the differences are between GC images and Wii images? It sounds like a silly question, but I don't actually know. In any case, it looks like there are still a lot of useful things we can do with wit. And like I said, some stuff like export/import can be done programmatically "manually", if that makes sense.

Is there a way to get wit working without installing it, like wimgt in DTW?

Yeah, just use a relative (from your current directory) or absolute filepath to wit.exe. You only need the 'bin' folder. And probably other files in there aren't needed either, but I haven't gone through to figure out which ones yet.

In my example above when I called the command I just used the relative filepath of 'bin\\wit.exe (+ the rest of the command)', and had the bin folder for wit in the same directory as my program.

When I want absolute filepaths, I use this which I typically have at the start of my scripts.

from os import path
scriptHomeFolder = path.abspath(path.dirname(args[0]))

...and then usually I'll just combine it with other strings so I have a set of variables for all my paths, e.g.:

imagesFolder = scriptHomeFolder + '\\imgs'
wimgtPath = scriptHomeFolder + '\\bin\\wimgt\\wimgt.exe'
witPath = scriptHomeFolder + '\\bin\\wit.exe'
texDumpsFolder = scriptHomeFolder + "\\Texture dumps"

 

[Doq]

Tried that, creates a Wii image.

The core difference between a Wii and GC image is encryption.

That's all I know.