Universal Model Converter - Side Development Thread

[Magicalbarf]

Has anyone figured out how to get it to work yet?

 

[Doq]

^ Should be relatively easy. If you're missing external things grab them and run UMC.bat/exe
---
I'm only wondering if this is still in development as I'm planning on including it (finally) in The Melee Toolbox 4.5, and I don't want to bother if it doesn't work and is abandoned.

Tcll

 

[Magicalbarf]

What external things?

 

[Magicalbarf]

If you know the external things can tell you what they are?

 

[Tcll]

It does the exact same thing for me and I don't know how to fix it.

you can't...

that's what I'm working on fixing...
I've gotten somewhere good enough to get a window and importing,
but the import is not what one would expect...

feel free to play around with stuff as I've just shared my resources, but don't expect anything working for a while
https://mega.nz/#!5ggTlTJA!J9cU30BrkvnCeehUajyUtgv5NPCcqktu7jPQ-5mAHco

note that the API structure is finalized, so future versions should have no frontend changes.
(it shouldn't be a pain to manage your scripts through UMC/UGE updates anymore)

as for the broken links:
- copy's service (the best service) has discontinued leaving me with a TON of them to clean up...
- I don't like Gov't-infested DropBox, but use it where I have to, removing files occasionally
- MediaFire can delete my files all they want, I'm done with them...
- 000webhost (my website's domain) has garbage cleanup, deleting unused files like MediaFire
- nobody likes 4Shared for some reason >_>
- I like Mega, but don't like using it (I try to only use it for the big stuff)
- Google Drive is reserved for my Picasa images (soon to become G+ >_< )
- I'm working on setting up an organization on GitHub so you guys can download a working repo, but I need to actively sync my revisions for that to work properly (like copy did)

so yea, I got alot of crap I'm going through, and can't even be online to maintain it (no net at home) >_<

once I get net, I intend to host a Banana-Pi RAIDed cluster server to take care of all this unreliable network garbage.

 

[Tcll]

^ Should be relatively easy. If you're missing external things grab them and run UMC.bat/exe
---
I'm only wondering if this is still in development as I'm planning on including it (finally) in The Melee Toolbox 4.5, and I don't want to bother if it doesn't work and is abandoned.

Tcll

^ sorry if your color is incorrect...
normally I would hack the webpage to copy your user-color for the BBC tag, but this hospital net is so slow I need to view smashboards through a web-proxy.

I promise you, it's still majorly in dev,
I ain't quittin until UGE (the future project from UMC) is capable of converting Minecraft into a Wii(U) executable

as far as externals, I really hate 3rd-party packages and am working on getting rid of as much as possible...
heck, I'm bundling DLLs and SOs into zipped plugins to be loaded directly with python's ctypes library.
(yes, I hate 3rd-party so much I'm writing my own DLL wrappers for UMC's plugins)


so I've got alot of stuff I'm re-working...

I do promise Iwon't stop working on it
I'm just back to having no internet again >_>

 

[Magicalbarf]

Thank you so much

 

[Tcll]

btw, if anyone would like to talk to a more-active me,
I'm over on discord for about a week on my dev-chat:
invite: https://discord.gg/0Y8h5pLQK5uS7xMH

@MLP H8Rs: trust me, this has no reference
I hate MLP as much as skype, though I do like Discord alone in that show...
in any case, I get far more functionality than skype (linux) will EVER have using discord...
F Microsoft and their evil ploy to force everyone to use their cruddy joke of an OS.

I'll never understand why they support mac better than linux, when mac has a linux implementation, though is slower than linux with even worse support for various things.

 

[JohnTravolski]

Even the new link you just shared still crashes for me. Is there any way to get around this for now? I really want to see if I can import a Kirby Air Ride model or something now that I know I can extract all of the game's file right out of the Dolphin Emulator.

 

[Tcll]

tbh, I'm not sure it's even worth importing...
the download was only to show what I was dealing with.

I finally got a screenshot though, so this is how things import currently:

this is not a script issue, in fact I've disabled weights in the script, and we all know the raw verts don't look like that, let alone the color...
I'm not sure what's going on with the API, but I know it deals with the matrix decomposition function...
not sure why it's affecting the verts though and why everything's rotated 90 degrees...

I'll fix it, but this is a major headache...
it's one of the nightmares with a completely dynamic interface.

just looking at that image gives me a headache...
hope you guys don't mind me taking a minecraft break for a while to gather my thoughts and ideas.
I'm actually starting on a mod for MC 1.10

if anyone's interested, you can read up on it here:
https://oc.cil.li/index.php?/topic/953-1102wipoc-hard-light/

no this is not an excuse to quit, I want KAR models as bad as anyone else here, but my interface is just being ********, and I need a moment.

 

[Tcll]

Good news =D
finally got SIDE working with the new interface.

Bad news
I had to remove some old support to get it working.

What was removed:
- multi-value data types: bu32( ['']*size )
- list-based structures for array()()

So things work as expected now, and since SIDE works, I think I've found the problem from above (there's absolutely no change).

I believe the polygon problem originates from a misinterpretation of the attributes (in the HAL script).

I'm still looking for the bone problem, I have a test script it affects too.

 

[Tcll]

btw, despite the removal of those things, it actually makes scripts a bit simpler

The workarounds which are exactly as the interface is designed around:
- array( bu32, count=size )()
- array( struct ( ... ) )()

Hey as complex as things get, I promised to keep the design as simple and easy to follow as possible

Just keep in mind, the design has to be as versatile and manipulative as possible.

With as much as what goes on between various formats, it's hard to keep things noobishly simple, but I seem to be the only developer who strives to do just that... heh

Sorry if you find something to be a little complex with the interface, I'm trying to clear it up as much as possible
I'm even adding in depth documentation with usage examples.

If you still can't understand something, ask me, and I'll even clear that up in an update.

 

[Tcll]

Btw, new feature which just might be operational by the release.

Data = bu32()
Data.__color__ = 0x008080

What does it do??
well for one, it's an optimization for SIDE, so colors won't be recalculated in the tracer, but is also a crossover implementation for the SIDE plugin for UMC, and also for other plugins which might want to make use of the colors interface.

So yes, it means custom user-defined colors for data types in SIDE.

And it's not hard to integrate into the new interface, just tedious.

EDIT: btw, it's a private variable for a reason.
struct() can do it too, and 'color' is commonly used.
you can't use private variables in structs, so that makes it safe.

 

[Tcll]

FINALLY!!!!!


It ain't perfect, but it's a first step after a long time!
I'm finally gaining some faith for the new interface!

 

[Tcll]

New features
struct names are now displayed (retrieved from the variable name using ast).



Still fixing various issues though, like struct colors not matching between viewers, and labeling.

EDIT: well that color implementation looks like it's happening now, getting minor issues translating everything in real time, so screw it.

Aside from that, pichu's diffuse color is finally correct, but the fix broke the emissive color, so he now looks like he has no shading.

Either way progress is being made

Once I finish the interface and rebuild the viewer, I'll be able to work on animations

 

[Magicalbarf]

Oh goodie
Now to wait for the download

 

[Tcll]

Woah there, I said I'm making progress, and mentioned a few things for whenever I release.

I never said anything about releasing soon, because this thing currently isn't anywhere near a release just yet.

EDIT: just adding, I had to write this in short time with the keypad software giving me issues...

I don't want to sound like a douche or anything, but yeh, really going through some tough times with this interface... heh

I'd built this interface to make everything easier, yet am getting blind-sided by the port itself... lol
once all the old stuff is out of the way, things should be really easy.

Currently though, I'm having an issue with structuring automation in SIDE with the HAL script.
specifically the vector reads from the facepoints, which are currently read after the facepoint array.

I want the facepoint struct in that array to list each vector read for each attribute.

The new data interface is supposed to make this easy, but I'm missing things to make those things happen.

 

[Tcll]

Just to show off some of the issues I'm going through, as well as a few updates, here's a screenshot of SIDE running in PyCharm:


Issues:
- undocumented front-end functions
- improper struct attributes (display only)
- colors aren't broad and too dark (new interface)
- .__color__ doesn't currently do anything visible

Features:
- children of arrays now display their index in the tree view
- the inaccessible stop struct is displayed with an index of '[-]'

I'm currently working on fixing struct attributes, but am running into an inaccessible function global used in processing those attributes.

Makes no sense right?
it's acting like the global is deleted right after it's used before SIDE's tracer can do it's work.

 

[Tcll]

Just an update, I got custom colors working and fixed dynamic structs.

look at line 32 in the code editor, and then look at the RGBA8 struct highlighted in the tree viewer. ;D

It's still full of issues, but things are finally starting to function properly.

EDIT: btw, struct now has a 2nd argument for offset automation:

struct( 4, bu32, ... )

EDIT2:
Got struct labeling working, but still having issues with dynamic structs...
array labeling still doesn't work atm.

I'll post a screenshot when it looks better.

 

[Tcll]

Well I think I may have finally hit a major rut with the new data interface and may need some help to fix this... :/

So arrays keep track of their associated data structs via the very id used in coloring them in SIDE.

So I've just tried a new structure formatting with them, which works as expected, however since these arrays are dynamically calculated when building the facepoint struct, the same id is used for each attribute of the struct, meaning when the struct is called, each attribute is initialized with the settings of the last attribute.

To put that simply, in the image above, the facepoint consists of a vert, normal, and uv...
these attributes are all read as the uv attribute, offset and all.

To fix, I need to use a different array id for each attribute.
But doing that will throw off everything tracked by both the array and SIDE...

So yea, I'm currently at a loss... :/

EDIT:
Well I removed the array's tracker, and did python's job with a local variable hacked into the initializer's namespace.

That fixed everything except the offset which the data is read.

So it still reads at the uv location... -.-

EDIT2: fixed, the offset.
it was the augment's global being set to the last CP-attribute before the fp_struct gets called.

So now the only remaining issue is improper labeling of the stop struct in the tree viewer.

 

[Tcll]

Progress:


I'm not sure what's up with the normals, but I had to downscale the vertices to get it to look like that...
Not sure what's up with that either...

The transformations is an old issue I have yet to resolve...
It's with the matrix composition and decomposition functions...

As for SIDE, I'm still dealing with the array assignment issue...
Basically indices are not what they're supposed to be...

Basically a case of fixing one thing breaks an earlier fix...
gotta love programming. #awesomeface

EDIT:
btw, the script now supports multiple meshes per object
So there's less objects in the display panel, and the vertices associated with the objects are no longer split up, so it's a little more "normal".

 

[Tcll]

Uhh guya buh...


Read the comments in IDLE's python shell...
python can't do math -.-

 

[Tcll]

Ok, after losing a paragraph because this phone sucks...
(And this forum can't remember your post once the page gets suspended)
forget the important stuff... I'll just post the images:

Python can do math btw...


After flipping the __r###__ operations:



Also, the brightness of the shading is due to the color values.
they're supposed to be 0.0 to 1.0, but instead are 0 to 255.
I'm gonna work on making the viewer auto-detect this, since the value range is allowed in the API for lossless data reasons.

 

[Tcll]

After applying a basic test to the material colors:


I'm gonna be lazy and not work on anything advanced with this viewer...
it's scrap anyways...

 

[Tcll]

Just wanted to update, though I don't have anything different to show...

I've fixed the matrix composition and decomposition functions, but the problem above still isn't solved...

All I did really was fix the LRS export results, which the viewer no longer uses for display...

There was nothing wrong with the composition function, although it referenced from math, which was needless and bamfed performance a tad, but it's a little lossy (not noticeably though) for some reason...
I'll have to figure that one out later...

It's a difference of about ~0.00000001, so yea, nothing to worry about until UMC gets more advanced.
just remember not to convert a converted model.

EDIT: btw, the HAL script only uses the composition function, but that takes place during transformation, after importing...
So the problem is likely with either the matrix class, or the Bone class, both of which are new.

 

[Tcll]

Btw, don't rely on this too much because anything can happen...

But I think I might have a release time somewhere before late September.

I don't have too much more to fix, and string()() is the last API feature to fix currently.

I've held off on deref() for now, but ref() works perfectly

But at least now, UMC (or UGE) scripting has become much easier to use and is much more solid.

EDIT: let's not forget the amount of scripts I still have yet to port and fix... lol

EDIT2: btw, a note left out regarding automation...
currently, with augments and all, automation is good, but not exactly desirable...

There's still currently unknown things left to implement before I can really work on cleaning things up...
Think of it this way, if it requires an augment to work, it's a feature left to be implemented.

Hey, all good things start small.

 

[Tcll]

Also, compression and archives won't be supported until the next update.
along with an update system...
yes, updates are still fake... lol

I'm setting something up on github with various branches to be supported by the update system.

But it's still unfinished because I have no internet, and need to be able to sync branches with my computers.

EDIT:
Btw, a little note about the above issue...
it's another thing I can't wrap my mind around...

So basically, it happens during verification...
when I run my test script, it sets the Location of the 2nd bone to ( 2., 2., 0 ), which is the bone a multi-colored square is fully weighted to.

So what happens, since you can't set bone matrices through the scripting interface, is the LRS is composed into the bind and inverse matrices during verification.

What the current viewer does from here is invert the inverse matrix to obtain the bone's world-related coords.

Now that we know that, we know that the bind and LRS values are useless to the viewer, which partially explains the problem, the other part that explains it is during transformation, when an influence matrix is calculated from the bind and inverse matrices of each weighted bone.

So, what's the problem then?
Well, I'm not sure if every inverse matrix is like this, but with my test script, it's always a basic identity matrix, instead of an inverse of the world matrix.

The LRS and bind values are correct, it's just the inverse that seems to have a problem...

I'm looking into it, but it's not exactly simple to find...

 

[Tcll]

So this thing is proving to be quite the conundrum...

In effort to fix the inverse matrix, I tried removing the transform class (which needed to be done anyways), but that ended up creating a broken bind matrix which in turn broke the LRS values...

So yea, there's some major ghosting going on somewhere and it's screwing with the compose function...
all external tests are still correct.

 

[Tcll]

Btw, I'm working on some termage and need some help with a term for the classes which define a data struct...

Currently focusing on calling them data designers, but am not sure if there might be a better term...

So here's how the new data standard works:

d_struct = d_designer( *settings )
d_type = d_struct()
addr = ref( d_type )

To put some perspective into this:

example = struct( 8, offset, value=bu32 )
result = example()

struct is a designer
bu32 and example are data structs
result is a data type.

An addition to this standard is called data augments:

def d_aug( value, label='', *args, **kw ):
return bu32()+32

You can use these in place of data structs, but the result should not be expected to be a data type.

 

[Tcll]

Well I got my test script working properly, but now Pichu imports with even more mesh distortion, and the bones still at the same locations. >_<

 

[Tcll]

Btw, new feature in the works, though sadly, not by this release...


Here's a small treat the new interface allows me to do.
on the right, I've turned UMC's console into a terminal.
this allows manual data interaction.
bone names update in real time, but bone lines need a viewer refresh...
Don't even get me started on what the mesh needs... D:

In any case, you can hack this release to get some support working, but currently it's super buggy and unstable...

I need a custom console before I'll be able to do any real magic with that, which should be available by the next release.

As for real-time stuff, I'll need a new viewer, which I'm working on right after I release.

So trust me, I'm working on a lot of treats.

 

[Cheetaboy]

..........

 

[Tcll]

I've isolated the bone problem

After fixing the transformer and getting results like the last image again,
I'd gotten the idea to print the given and decomposed LRS values...
Sure enough, the results were different for some bones.

Pichu's model data follows common standards, so the data is already formatted the way UMC expects it's own data.
Meaning the given data should match the resulting data.

Now that I've found it doesn't, I tested the same thing on the composition function and the manual test gave me the same issue...

So I need a better matrix composition function...

Once that's done, the API should be nearly ready for a release.

Still gotta do string()() and all...

 

[Tcll]

Ugh, this is proving to be quite the hassle...

Tried an algorithm by crmccreary, and found out he doesn't know how to math...

So then I tried a src from brawlbox, and asin() throws a "math domain error"

What I had already was the same as transformations.py only many times faster and not using NumPy...

There were other sources I found on websites, one of which looked very convincing, though the comments suggesting improvements are what turned me away from it...

Getting sick of looking for broken sources...
does anyone know how to do math??

 

[Tcll]

Btw, let's not forget about SIDE

I'm not gonna focus so much on SIDE's script editor... just use IDLE or PyCharm for now...

If anything, I probably won't be working on SIDE so much anymore, now that I have UMC's plugin interface.

So yea, do whatever you want to the Qt source, cause I'll be working on the plugin after the release.
just please for the love of god don't commercialize any of my codes. D:
and give me credit where due please

Anyways, I'll be releasing SIDE with UMC in it's current state (after fixing the array assignment values) since it's a far better debugging tool than an IDE.

 

[Magicalbarf]

Just saying it's good to see you active again

 

[PrinceJohn]

Tcll threads are perhaps the best thing on Smashboards.

 

[Tcll]

Tcll threads are perhaps the best thing on Smashboards.

why thank you, your threads are cool too.
if I mistook your uplifting comment about my works as sarcasm at my multi-posting then please accept my apologies.

Just saying it's good to see you active again

it's good to be active again :D
it may not be 100% just yet, but still, I missed you guys

EDIT: btw, I think I may have called early September for a release a bit too soon...

I got string()() working, and am currently porting my mdl0 script, but due to lacking automation support, the script is nightmarishly ugly.
and it currently freezes with pikachu, though not Remly.

I think it might have to do with the dynamic structs and some of the lazy automation workarounds in the script...

In any case, I don't think I'll have that array assignment issue fixed in SIDE by the release...
reason being is that it stems from augments and map functions, which are not traceable by SIDE...
so the tracers sees more nodes than expected, and doesn't know how to deal with them...

And neither do I tbqh, since python doesn't know half its own compiled code, meaning finding it during runtime is next to impossible...

 

[Tcll]

Btw, SIDE now displays some of what's read when an exception is encountered.


Now if an exception is encountered in a map or augment, the whole root (struct or array) is discarded...

I'm trying to work on fixing that, but I can't make promises...

 

[Tcll]

Idea for defining widgets in scripts with the new UI.

def ugeModelImportUI(UI):
    UI.property = CheckBox(X,Y,'Label')
    UI.property.text = 'New Label'

def ugeImportModel(filetype,UI):
    if UI.property: pass

I hope that's not too complex

UI is a class-instance created if any uge**UI functions are defined.

Normally you can't define properties (aka class-attributes), but the UI instance has a special __setattr__ method which creates undefined attributes as long as they're not __private__

So anyways, that instance will be passed to the creation and I/O functions for property retrieval and updates.

Note that a new instance will be created on any I/O event, so there's no property confusion between events.