Stage Hacking: NEW Research + Documentation

[Milun]

Ooh ooh ooh. Maybe it's because I didn't change the Display-List size?
00 00 00 00 00 00 00 00 00 01 60 E8 80 00 00 AE <- It's this part right?
00 01 BE C0 00 00 00 00 00 00 00 00 00 02 AC D8 <- This points to the next mesh struct offset
00 01 5E D8 00 01 D4 80 00 00 00 00 00 00 00 00

EDIT: Confirmed, that is the problem. Now to do some math.

 

[zankyou]

Ooh ooh ooh. Maybe it's because I didn't change the Display-List size?
00 00 00 00 00 00 00 00 00 01 60 E8 80 00 00 AE <- It's this part right?
00 01 BE C0 00 00 00 00 00 00 00 00 00 02 AC D8 <- This points to the next mesh struct offset
00 01 5E D8 00 01 D4 80 00 00 00 00 00 00 00 00

EDIT: Confirmed, that is the problem. Now to do some math.

youve found whats called the mesh structure. You should just be able to copy over the value needed. Also that number is just how many items are in the display list. You need to go to that offset and copy the data over.

 

[Milun]

Ok, so I changed the display list number for some... odd results. Changing it to the number that was in the actual TyDosin file DID give faces to the whole model, but also resulted in a large amount of junk faces (like the ones shown in the picture, but a LOT more). I got a better result by lowering the number (but as you can see there's still a few junk ones).
I know it's TyDosin's faces causing it too, since I blanked all the Goomba faces completely. Very strange.

And what makes it stranger is that there are no vertices at the locations that the bad faces are connected to.

This is what happens when I use the right size (according to TyDosin). And I know it's not reading the UV's as vertices because I even changed the pointer to the end of the vertices to match the lesser amount that Dosin offered.

 

[Tcll]

lookin good so far milun
you can figure out those junk faces by looking at the attributes of the old model

EDIT:

Ooh ooh ooh. Maybe it's because I didn't change the Display-List size?
00 00 00 00 00 00 00 00 00 01 60 E8 80 00 00 AE <- It's this part right?
00 01 BE C0 00 00 00 00 00 00 00 00 00 02 AC D8 <- This points to the next mesh struct offset
00 01 5E D8 00 01 D4 80 00 00 00 00 00 00 00 00

EDIT: Confirmed, that is the problem. Now to do some math.

btw, if anything's inaccurate on the HAL_DAT page, please correct it.

for example the display list size should be multiplied with 32 (as it's in a 32-byte alignment)
I'm not looking at the page as I type this, so irdk

and to co-note, I'm using my terms such as 'vert' and 'facepoint' rather than the pro terms 'vertex-position' and 'vertex'.
you can instantly see my terms make more sense to everyone (except for ImaginaryZ).

though the pro terms should be documented, and I will work on that when I'm not blocked.
(it's a mental block to manage my autism and prioritize my work so as to not overwhelm myself any further)

 

[zankyou]

Ok, so I changed the display list number for some... odd results. Changing it to the number that was in the actual TyDosin file DID give faces to the whole model, but also resulted in a large amount of junk faces (like the ones shown in the picture, but a LOT more). I got a better result by lowering the number (but as you can see there's still a few junk ones).
I know it's TyDosin's faces causing it too, since I blanked all the Goomba faces completely. Very strange.

And what makes it stranger is that there are no vertices at the locations that the bad faces are connected to.

This is what happens when I use the right size (according to TyDosin). And I know it's not reading the UV's as vertices because I even changed the pointer to the end of the vertices to match the lesser amount that Dosin offered.

You could try also fixing the texture flags.

 

[Tcll]

hey zankyou, if you know more about the flags than what I've documented, the texture structs could really use some work there

 

[zankyou]

hey zankyou, if you know more about the flags than what I've documented, the texture structs could really use some work there

Well what I know I got from the dat thread. I could try to help you there if you want.

 

[Achilles1515]

*face palm*

Thanks to @Anutim , I just now realized that the Melee Toolkit automatically calculates the root offsets for the strings at the end of a DAT file....

@Dan Salvato is awesome.
(Pic is Battlefield)

 

[zankyou]

*face palm*

Thanks to @Anutim , I just now realized that the Melee Toolkit automatically calculates the root offsets for the strings at the end of a DAT file....

(Pic is Battlefield)

View attachment 34490

Theres actually a template floating around the internet for the entire dat. But the hex editor its for isnt free. I used the trial like a year ago before I knew what I was doing. Ill see if I can find it again. Maybe someone can convert it.

 

[Milun]

lookin good so far milun
you can figure out those junk faces by looking at the attributes of the old model

I think I'm........... lost again.

Is it this part? (which appears above the first instance of face data and has a pointer to where the Vertices and UVs end)?
If so the one in TyDosin is formatted differently to Goomba, and aside from those two pointers I don't know what any of the flags mean. (I keep reading that HAL Dat thing but I can't find what this part is).

 

[zankyou]

I think I'm........... lost again.

Is it this part? (which appears above the first instance of face data and has a pointer to where the Vertices and UVs end)?
If so the one in TyDosin is formatted differently to Goomba, and aside from those two pointers I don't know what any of the flags mean. (I keep reading that HAL Dat thing but I can't find what this part is).

I could be wrong but I think the 800008 is the display list and the 3f800000 is the texture hearder.

 

[Milun]

It is. I'm just wondering which part of this I'm meant to copy over to fix the busted vertices.

 

[zankyou]

It is. I'm just wondering which part of this I'm meant to copy over to fix the busted vertices.

I copy all of the displaylist over. Im surprised you got those results without doing it.

 

[Milun]

Hm... If I understand what a display list is correctly, I already have that part (it's what actually defines the faces right)?

I was wondering about the part directly above the first Display List. Would it have an impact on the dodgy faces? Tcll mentions "looking at the attributes of the old model", and I thought that was it.

 

[Tcll]

hey milun, it might help you better if you were to understand exactly how the file works.

start at the root nodes, and look for a bone/joint structure.
follow the structures on my HAL DAT format page until you get to the mesh data.

think of the data as a sort-of state-enine setup (since GX is GL, which is a state-engine)


that data you're showing me looks to be the attributes at the top half and possibly the end of a texture structure
while the noticable change in data pattern at the bottom half looks to be a display list.

already looking at it, I can make out the first primitive has 8 facepoints
0x80 - quads
0x0008 - 8 facepoints

the actual facepoint structure is determined by the attributes, which the data, when divided by 8, looks like:
13 9C 13 81 13 4B
13 99 13 7E 13 48
13 9A 13 7F 13 49
13 9B 13 80 13 4A
13 77 13 5C 13 2D
13 74 13 59 13 2A
13 75 13 5A 13 2B
13 76 13 5B 13 2C

from the looks of it, the data appears to be non-weighted (with knowledge this is a trophy on a stage) and contain 1 of each vector:
vert
normal
UV

139C 1381 134B
1399 137E 1348
139A 137F 1349
139B 1380 134A

1377 135C 132D
1374 1359 132A
1375 135A 132B
1376 135B 132C

I split the data so you can see the quads

I did not look at the attributes to figure this out, so if I'm wrong, that's on me

 

[Milun]

Thanks Tcll, but I already knew that part. I'm wondering about the part directly above it (14440 -> 144B0) and what influence it has on the display list to cause the bad faces. I copied Dosin's display list exactly, so I don't think the problem is in the Display List as a result.

 

[zankyou]

hey milun, it might help you better if you were to understand exactly how the file works.

start at the root nodes, and look for a bone/joint structure.
follow the structures on my HAL DAT format page until you get to the mesh data.

think of the data as a sort-of state-enine setup (since GX is GL, which is a state-engine)


that data you're showing me looks to be the attributes at the top half and possibly the end of a texture structure
while the noticable change in data pattern at the bottom half looks to be a display list.

already looking at it, I can make out the first primitive has 8 facepoints
0x80 - quads
0x0008 - 8 facepoints

the actual facepoint structure is determined by the attributes, which the data, when divided by 8, looks like:
13 9C 13 81 13 4B
13 99 13 7E 13 48
13 9A 13 7F 13 49
13 9B 13 80 13 4A
13 77 13 5C 13 2D
13 74 13 59 13 2A
13 75 13 5A 13 2B
13 76 13 5B 13 2C

from the looks of it, the data appears to be non-weighted (with knowledge this is a trophy on a stage) and contain 1 of each vector:
vert
normal
UV

139C 1381 134B
1399 137E 1348
139A 137F 1349
139B 1380 134A

1377 135C 132D
1374 1359 132A
1375 135A 132B
1376 135B 132C

I split the data so you can see the quads

I did not look at the attributes to figure this out, so if I'm wrong, that's on me

Weight structures... I should probably look into what these do instead of just swapping them.

 

[Milun]

In essence, my theory is that there's some parameter which applies to all display lists that's different in TyDosin to Goomba, and because of this parameter, the display list that works perfectly for TyDosin is having issues in Goomba.

 

[Milun]

Oh shoot. Could they really be weighted? I mean, the Goomba doesn't move so it shouldn't even have bones right?

 

[zankyou]

In essence, my theory is that there's some parameter which applies to all display lists that's different in TyDosin to Goomba, and because of this parameter, the display list that works perfectly for TyDosin is having issues in Goomba.

Did you copy the unknown mesh flags over as well. Gotta be meticulous.

 

[Milun]

I uh... think I need to take a break.

 

[Tcll]

sorry, I do that here and there XD

but it's for that exact reason on your questions I want you to follow those structures.

the mesh structure should give you a pointer to the attributes and display list size/32

as for the attributes, here's a single attribute structure as noted in UMC's session-info.log:

0x0000006980: read 0x00000009 as 9 -- CP-index
0x0000006984: read 0x00000003 as 3 -- index-type
0x0000006988: read 0x00000001 as 1 -- component count
0x000000698C: read 0x00000003 as 3 -- data format
0x0000006990: read 0x0B as 11 -- exponent
0x0000006991: read 0x00 as 0 -- unk
0x0000006992: read 0x0006 as 6 -- entry stride / color format
0x0000006994: read 0x00000000 as 0 -- data offset

 

[Tcll]

Weight structures... I should probably look into what these do instead of just swapping them.

just skimmed across this...

the weight structure is used to calculate an influence matrix to apply to the current vert-set.

basically, it's what transforms this:

into this:


normals included (rotation matrix only)

EDIT:
just got in a discussion with a friend I'm helping get into Melee vertex hacking...
he was talking about how the old method I used was really hard to work with...
so I figured what I PM'd him might help you out Milun

the way the verts look is what I call "UnTransformed"
I literally just got finished referencing a similar issue onthis thread:
*link to this post given*


what I'm referring to on that thread is the weight struct documented here:
http://wiki.tockdom.com/wiki/HAL_DAT_(File_Format)
as for how to use it, the best resource I have is my UMC script:
https://copy.com/GhS6PAbuBH7MedUI
at line 84:
transform() applies a supplied influence matrix to a vert
Ntransform() applies a supplied influence matrix to a normal (rotation only)
getWeights() calculates the influence matrix from a list of bone offsets with associate weight values.

EDIT2:
made a visual edit on the HAL_DAT format page in your area of work Zankyou.

added reference data to the page, and better described the weight-structure arrays

keep note though, the weights offset in the mesh data points to an array of pointers.
these pointers then point to the weight structure arrays.

I went through alot of work in figuring them out, but each weight structure array is used in calculating an influence matrix which is stored in the XF register and referenced by the weight index in the facepoint.

here's the code that generates the influence matrix from the weight structure array:

    def getWeights(WOL):
        Jump(WOL, label=' -- [ Weight_Offset ]:' )
        ML=[]
        for WO in array([bu32])(): #Matrix/Weight Offset
            Jump(WO[0]+32, label=' -- [ Bone_Offset , Weight ]:')
            inflmtx = [
                [0.0,0.0,0.0,0.0],
                [0.0,0.0,0.0,0.0],
                [0.0,0.0,0.0,0.0],
                [0.0,0.0,0.0,0.0]]

            #---restructured using the src for BrawlBox:
            _weights = array([bu32,bf32])()
            if len(_weights)>1:
                for MO,W in _weights:
                    bind,invbind = matrices[bones.index(MO+32)]
                    tempmtx = MtxMultiply(bind,invbind)
                    for r in range(4):
                        for c in range(4):
                            inflmtx[r][c]+=tempmtx[r][c]*W
            
            elif len(_weights)==1:
                MO,W = _weights[0]
                bind,invbind = matrices[bones.index(MO+32)]
                for r in range(4):
                    for c in range(4):
                        inflmtx[r][c]=bind[r][c]*W

sorry for the obfusticated code... I really didn't know how to term these when I wrote it
vars:
WOL - Weight Offset List (this is the supplied pointer to that)
WO - Weight Offset (pointer to the weight structure array)
MO - Matrix Offset

this was back when I hardly understood anything about weighting the models... heh

 

[Tcll]

just wanted to mention, I'm taking a short break from my GUI...
every time I try to work on my scrollbox widget (holds other widgets), I get a headache >_<

so I wanted to note,I'm updating my DAT template for HexEdit with a current method quite similar to my pathfinder
(the only difference is my template uses <define_struct> which is more like a function definition in python)
^ in my script, I'm using a dictionary

so yea, instead of defining a structure like:

        'bone': [ 64, _Bone_Set, [
            [0,'pass'],
            [8,'bone'],
            [12,'bone'],
            [16,'object'],
            [56,'pass'] # (marix) no pointers, no need to test.
        ]],

instead, what I'm doing is:

    <define_struct type_name="pfBone" comment="" expr="">
        <eval expr="expected=-1, Struct=pfPass" display_error="false" display_result="false" comment=""/>
        <use_struct name="pfPtr" expr="" type_name="pfPtr" comment=""/>
        
        <data type="none" name="skipped"/>
        
        <eval expr="expected=64, Struct=pfBone" display_error="false" display_result="false" comment=""/>
        <use_struct name="pfPtr" expr="" type_name="pfPtr" comment=""/>
        <use_struct name="pfPtr" expr="" type_name="pfPtr" comment=""/>
    </define_struct>

it's a little more straight-forward, and slightly more taxing,
but it's still easy going and works about the same

 

[Milun]

So I tried this again. This time, I pasted his face data in a different offset and then re-linked the mesh structure. Note that despite the face data not changing, pasting it in another location caused significantly less junk faces. I then tried to move the location of these values:

00 00 00 00 00 00 00 00 00 01 60 E8 80 00 00 AE
00 01 BE C0 00 00 00 00 00 00 00 00 00 02 AC D8
00 01 5E D8 00 01 D4 80 00 00 00 00 00 00 00 00

To be directly after the new end of the new faces I posted. And apparently I've done something wrong there (I've redirected all relevant pointers to the new location, but it still crashes).

 

[Tcll]

I'm having issues with my new template...

apparently this new pathfinder is a sword to HexEdit's heart...

the template only works with very small sized DAT files >3<
anything larger freezes HexEdit >3<

this would greatly help you out if I could manage to get it working >3<

EDIT:
my current progress:
https://copy.com/g9iNkU42kqH1tOJq

the template should just highlight the root structures with a red hue

 

[Milun]

Sorry man. I really can't understand, let alone help with this sort of stuff...

 

[Tcll]

hey milun, I've got a friend helping me out with UMC, though he can't really do much in his current standpoint as he's another noob... heh

the good part though, he shares my ability to visualize logic in a 3D space as nodes with wires, so thus, I'm able to get him up to my position from nothing in about a week or so

the plus side, in being a noob, he's able to find what I've missed that made it difficult for people to understand what I was saying, so I've asked him to log anything he finds as he learns.
and that just happend last night on my DAT page

if you're having trouble seeing the data structures, there's now a beginners section to help explain the pointers and such, and offers an alternative hex editor called HexWorkshop

I'm not sure if it can do what HexEdit or 010 can do, but it looks promising

I can run windows programs on linux, so I might give it a try

if it runs faster and highlights more than HexEdit with just as much customization, I might switch to it. :D

also, if Andrew doesn't respond w/in the next 3 years (as it's been 3 years already since his last activity), I might make my patch a little more open.

 

[Ripple]

what are the odds of one of you guys making the homereun contest level bigger so it doesn't just drop off into nothingness?

 

[zankyou]

what are the odds of one of you guys making the homereun contest level bigger so it doesn't just drop off into nothingness?

What do you mean. Its probably easy enough to edit so whatever youre envisioning is probably possible.

 

[Ripple]

after a certain point in the HRC stage, the ground drops off and the sandbag will fall under the stage. and then crash the game or something

 

[Tcll]

I agree, it could more than likely be expanded, but I think recently Nintendo's been doing stage repeats, which I'd like to figure out how to code in

infinite HRC ;3

 

[zankyou]

after a certain point in the HRC stage, the ground drops off and the sandbag will fall under the stage. and then crash the game or something

Ah that sounds more challenging. THe only thing I could think of off the top of my head is having the grounds collision points move with the camera. There would probably still be a set limit to how far apart characters could be though.

 

[Milun]

I remember a few years back, I was contacted for a similar reason. A TASer had made a video involving the Ice-Climbers, and hit the bag so far it never even fell before the game crashed. I had little luck fixing that.

However, what I DID have luck with was...

The problem was that at the end of the .dat file, there were references to the pointers that the face data overwrote. Blanking them completely fixed all the broken faces.

 

[Tcll]

OK if anyone wants to play with a stage file using my template, GrTSk.dat is the smallest stage and takes roughly 1.5 minutes to load.
(I've gotten the thing as simple as I could possibly get it)

but all the template currently does is outline the root nodes and display the root structures.
you can download it here: https://www.copy.com/browse/a:d4adb28;z:copy;b:myfiles/HexEditTemplates/_dat2.xml/revision:84132
(this is a specific revision btw, so updates won't change this link, other than renaming the file, I think)
^ contact me if this link ever doesn't work

EDIT:
oh hey there ninja, you caught me typing a post... heh
looks awesome there dude =3

EDIT2:
finally made an image:

^ that red structure is raw CMPR image data

so yea, the pathfinder should work properly, the problem is it'll just take FOREVER to load

also, I'm running HexEdit in VBox on WinXP for safety measures... it seems to freeze easier on Wine...
(I will NEVER use anything above WinXP because now google cracked MS's RAT and can control anyone's compy if they so want to)
^ anyone on anything above XP64 that is, because MS was stupid enough to build a RAT into the Windows kernel to control everyone, and I'm not sticking around for when they actually press their big red button.

 

[zankyou]

I remember a few years back, I was contacted for a similar reason. A TASer had made a video involving the Ice-Climbers, and hit the bag so far it never even fell before the game crashed. I had little luck fixing that.

However, what I DID have luck with was...

The problem was that at the end of the .dat file, there were references to the pointers that the face data overwrote. Blanking them completely fixed all the broken faces.

Wait you didnt redirect every pointer to the new area and it still works?

 

[Milun]

No no. Basically, I overwrote a few existing pointers with the face data, but didn't blank the removed pointers from the end of the file. Blanking them fixed that.

 

[Tcll]

ah, so resizing to under the allocated size works.
(including the complete removal of data)

interesting

 

[zankyou]

No no. Basically, I overwrote a few existing pointers with the face data, but didn't blank the removed pointers from the end of the file. Blanking them fixed that.

You mean the pointers in the root node table?

 

[Tcll]

You mean the pointers in the root node table?

I think he means a few pointers to like... material data or something of the sort...

hey @ Milun , progress is coming along on my template, it might help you out =3
unfortunately you can't load files with alot of pointers in the relocation table or it freezes

and when I say alot, PlPkNr.dat freezes HexEdit

what stage are you working on??
it might load in my template