Universal Model Converter - Side Development Thread

[Tcll]

if you're wondering why I'm at a halt, it's because I'm not getting the help I need:
https://www.daniweb.com/software-de...tions-from-a-restricted-namespace#post2183373
and google is hopless at searching:

google search: python get dict from secluded exec code

result: AIDS AIDS's African African's Africans Allah Allah's American
another result: Dictionary Txt - Oracle Documentation
and another: homely I a - Department of Computer Science and ...
I'm done: Title in "Monty Python and the Holy Grail" - Mordo Crosswords

I wish google searches were as good as they were 5 years ago... =3=

and I can't ask on StackOverflow

 

[JohnTravolski]

Well, be sure to let us know when you continue. I wish I could help, but I cannot. If you're really struggling with this particular issue, or possibly future ones, I recommend trying to contact other programmers to see if they'd be interested in this; maybe they can help you in ways you wouldn't expect.

 

[Tcll]

oh you'll know, I'm the last person who'd keep my development a secret.
after all, having an IDE with intellisense as good Visual Studio has been something I've wanted since VS2012

in any case, I should mention I've contacted (emailed) Zooba since the night before.
Zooba is the developer of Python Tools

 

[JohnTravolski]

 

[StarLand]

Tcll, is it possible that you can create an directory in UMC named "plugins" so, we can makde plugins for that?


My idea is (ReadOnly):

1 Directory for Plugin - *PluginName*

1 XML File - plugin.xml

<?xml version="1.0" encoding="UTF-8"?>
<plugin>
    <name>MyFirst Plugin</name>
    <text>This is a description... ... ... more and more... Example ...</text>
    <version>1.0.0</version>
</plugin>

1 DLL File - *PluginName*.dll

I am very excited if you can realize that.
I want to make/programming plugins for that.
Sure, you only need say "yes, I can" or "No, I can't".

 

[Tcll]

plugins are already planned for UMC 3.0 and are .py files (most of which are currently in the data/ directory).
a plugin system for 3.0a will not be available (I'm scrapping 3.0a because it sucks).
I'm only releasing 3.0a to give me time to work on 3.0 while everyone develops scripts for it.
(unfortunately though this only applies to scripts, not libs, the 3.0a backend will not be capable of fully supporting them)
^ this is a design flaw, and one of the reasons why it's scrapped

in 3.0, COMMON and FORMAT are moved to and split up in the API/ directory with a higher-leveled backend.

just give me time, great things aren't developed in a day, especially if the developer is a noob like me.

 

[Tcll]

just thought I'd post a little something of interest, and have it show you guys the progress... lol



this is typical... fixing one thing breaks another... 9_9

 

[JohnTravolski]

What game is that model from? XD, PBR, Colosseum, etc?

 

[Tcll]

it's a trophy from Melee... lol
(I guess TyNyanthR.dat wasn't obvious enough in the image)

it's supposed to represent the demo "Meowth's Party"

 

[Tcll]

just fixed that annoying little '...' at the top-left of the screen

 

[Tcll]

good news that may change how DAT files can be hacked

it is believed the matanim section may assign IDs to bones and that the actual location of the bone doesn't matter.

if this is the case, we can throw custom model imports into the mix
but like the old days of using FortWaffles bone adding program for MDL0 hacking, the bones at the initial IDs must be the same as before.
if they're not, it invalidates the animations and logic.

 

[Tcll]

hope you guys don't mind if I take a tiny break from UMC...

basically, I'm writing a stand-alone data module that provides a UMC 3.0a dev5-ish file data manager.
it's basically something meant to replace struct on so many levels.

I'm just a little overwhelmed from reworking SIDE's syntax highlighter...
I'm using the re module the way IDLE does things now rather than the slow and incapable QRegExp() class from PyQt.
so yea, I'm reworking a good 3rd of CodeEditor.py with stuff I know little to nothing about.

I'll get back to work shortly

 

[Tcll]

well, if anyone wants to play around with it, here it is
https://copy.com/670IF7htQIKplXTJ
^rw.py read/write binary files using a UMC 3.0a dev5-ish data system

actually it's a bit improved...
found a few flaws with string()() in UMC, so rw.py currently does better and is actually slightly faster.

when importing:

from rw import *

see the bottom of the script for example usage.

if you have any traceback issues, let me know

 

[JohnTravolski]

I would try to use it if I understood any of it, but I really don't.

 

[Tcll]

I would try to use it if I understood any of it, but I really don't.

lol it's not really meant for noobs anyways
it's meant to make a programmer's job easier

but hey, doesn't stop anyone from trying to learn. ;D
all you need is python and an understanding of binary data.

also, I'll add support for array()() and struct()() later.
things are easier to work with since it's an extension, not an environment like with UMC.

 

[Tcll]

you know...
because this wait is taking so long, and I can't do anything script-wize until I finish SIDE...
I think I'm gonna work on NBT support just to make up for it...

I mean, the only lack is in VIEWER.py and a few formatting functions (can just copy/paste and modify the normals code)

so yea, that'll be a plus...
might work on a few GUI fixes as well, and if it comes to it, script-UI support (depends on SIDE's progress)
one thing I'll definitely work on is the laggy scrolling.
basically I just have to call remove()/add() instead of disable()/enable()

 

[Tcll]

just thought I'd mention, the reason I've been away is I've been working on porting a modern GLSL viewer from pyassimp:
(I had to remove numpy from the viewer and pyassimp and port from pygame to pyglfw as well as completely restructure and optimize the code)

^the imported model is an obj export I had lying around, but it's running at 79 FPS tops on a VBO
and yes, again, this is a completely modern GLSL implementation, so say goodbye to the slow FFP
feel free to download, all you need is python27
https://copy.com/uRZUHfk6i9gXMxt4
^note: the included pyassimp is outdated (I've submitted a PR and will update when it's accepted)
(currently, the official pyassimp will not work without numpy, (mine will))
numpy is a cancer and anything that uses it should be patched to work without it.
numpy is NOT faster, numpy arrays are actually slower than python lists, all they do is provide extended operations.

anyways... the viewer isn't exactly perfect and needs a little work (it was ported from an example, and is still an example),
but it's muuuuuch better than what UMC currently has.
the only problem is matrix transformation is still FFP-based, so that's a big slowdown on it.
next update to UMC, I intend to use a modern GLSL vertex shader which will manage the transformations.
so yea, that means really fast animation support

oh and btw, that will put me ahead of BrawlBox, since everyone's still stuck on the FFP there

in any case, what I'm currently doing is downloading regex, which is a replacement library for python's re module.
what I'll be doing with this is using it on SIDE's syntax highlighter to replace the disgusting QRegExp() class.

the new implementation will only have 1 expression, which will be recompiled when a new data type is defined.
the expression will put all matches (including recursive matches) into groups which will determine the syntax highlighting to be done on them.
this will be a MAJOR speed improvment from what I currently have and should be about as fast as IDLE, if not faster.

 

[JohnTravolski]

Sounds like you've made significant progress. It's good to hear that.

 

[Tcll]

sorry I disappeared on everyone

my neighbor who provided my internet is moving
my mother's working on getting some service, so I should be back soon

you guys will like what I've done to UMC ;D

 

[JohnTravolski]

Sorry I haven't checked in in a while, how's the progress going?

 

[Tcll]

Excellent! ;D

Posting off a phone atm, you guys will absolutely love what I've done with everything. ;D

So much progress has been made and things are alot more usable and informative.

 

[JohnTravolski]

Nice, I can't wait!

 

[Tcll]

Just wanted to make a note...

UMC/UGE is currently in the process of being redesigned...
Currently, nothing works at the moment, but functionality will be greatly improved by the release.

What's going on is an API overhaul that should've been done a few years back.
What this means is there will no longer be 2 versions in the works, but 1 version I can continue working on.

That aside, new features such as plugins are being implemented as planned back then.

The only bad news is the extended time...

I'm currently trying to figure out zipped executable integration, basically executing a dll inside a zip file wwithout extraction.

 

[Tcll]

Good news
UMC is back up and running ;D

Although it's very glitchy and still has alot left to be ported, but I'm able to import MDL0 w/o textures.

Plugins work perfectly, but binary modules and packed python extensions still need to be copied to a temporary directory before they can be loaded...

I'm currently looking into loading binary modules via injecting their data streams into kernel memory.
Since ctypes can't find a zipped dll, and copy is stupid.

Can anyone experienced assist??

 

[Tcll]

well, I've got some mixed news here...
basically, I've taken my laptop-desktop to a friend's for a few days...
so I was able to transfer a few reserved dev-images along with my src, however, the src won't be available for reasons described below.

so here's some of what you guys can expect when I'm able to release:
MDL0 updates: (somewhat fixed bones, and updated transformer, credit to Remly)

IQM imports:

other features include:
- weight display
- object display
- struct() support (yes, it's actually here)
- (currently WIP) ref()/deref() support

but UMC itself isn't the only thing with a few updates...
struct support has been extended to SIDE:

this includes libraries:


however a small problem sprung up recently which made me stop what I was doing and completely redo the data system.
(this is why ref and deref will be available soon)

if you'll look at the values here, nothing makes sense, heck even data structures aren't what they're defined to be:


so UMC is currently not operational due to this rebuild.

what's to be changed is what's returned by the data functions will be their own instances with their own support.

bu32() will return a BU instance rather than an int value.
this will allow the API to track what's being done with the data unless the data is modified.
(if the data is modified, then it's a raw python type)
you cannot write to the datatype instances other than with these methods

data = bu32( value )

dptr = ref(data)
deref( dptr, bu32 ).set( value ) # this updates the data instance

meaning doing something like this:
data = bu32()

data += 1

or this:

data.value += 1

will raise an error.

if you want to modify the data for script-related attributes, a simple workaround you can do is:

data = bu32().value # copy the raw value of the BU instance
data +=1 # modify for whatever's needed


in the future, I do intend to restore modification, but there's a conflict:

data = bu32()

data += 1 # should this modify the file, or should it change itself to an int??

personally, I'm leaning toward file modification, since it only modifies a virtual copy of the actual file (similar to how RAM works).

EDIT:
btw, this is an older image, but it shows 73% less code is not a far-off estimate from a blender script

^ this is the IQM import script, almost fully-automated.

if this were blender, you'd have to define the data handlers and manually update each blender reference (including switching between them).

not to mention blender has far more dot operators to do anything with...
and then the fact you need to now register a class, which is far more complex than with Blender24.

so that's why I did this

 

[Tiberious]

So, what about Melee importing and weight viewing? Will we be able to see joint positions and weighting per vertex? So long as I'm able to somehow get the skeleton joint position data and the weighting information for the characters I'm interested in, I'll be happy, because it's straight from the game's data and not some kind of wild-ass guess.

 

[Tcll]

oh yea, that reminds me


I'm looking for a good export format that's simple to write to and supports as much as Melee's data.

EDIT:
btw, I should probably mention the colors for those who don't understand...
it's quite simple really, think of hot to cold relating to a bone's influence on a vert,
where hot is 1.0 and cold is 0.0, and where warm is represented as green instead of purple.

the concept is bettered from Blender's weight paint where you can see the exact point of affection on the texture.
this will really help out when UMC becomes UGE, where you'll actually be able to modify the influences

EDIT2:
may as well post this

 

[JohnTravolski]

Okay, sorry for being completely illiterate in regards to all of your above posts, but it appears that you're making significant progress, which is really good. Now, I'm curious, because I don't really understand how most of this works:

Would it be possible to use this program to modify models in a game like Kirby Air Ride?

 

[Tcll]

eh, not directly, but yes
and when UMC becomes UGE, you'll be able to quite literally import KAR and edit it.

but currently UMC is inoperable due to a revision of the data system... heh

 

[JohnTravolski]

Was that a Donald Trump joke?
Now, when you say "import KAR and edit it," are you referring to going as far as changing all types of aspects of the game (like the actual probability of events occurring and things like that) or just the models in the game?

 

[Tcll]

eww never again
no matter how many times I send that douche to the darkness, he keeps coming back!

but yea, I mean the full game

it ain't called Universal Game Editor for nothing

 

[JohnTravolski]

That sounds pretty incredible. So it would be possible to go into the game's code to change all sorts of things? For example, this tool would have the capabilities to add stadium events to City Trial (when it's finished, of course)?

 

[Tcll]

yea, but it might not be as simple as one might think...

to do something like that, you'd need to learn a language I'm working on called UGESL which is a general language for game scripts and logic,
but it's usage is to provide a common interface to decompile ASM into code, and also compile code into ASM.

the language is node-based, so if you've had any experience playing with redstone, you should be able to use this

that being said, the language is closer to ASM, meaning you'll need alot of nodes to perform a simple task,
despite the fact that some functionality is dumbed down, such as adding 2 values with the equation node.

also, the language is debug-friendly, a rewind feature is provided, meaning you'll quite literally be able to run the program's execution in reverse from the current position.
(kind of like playing Prince of Persia, going back in time)

 

[JohnTravolski]

Well, I doubt that I'm savvy enough to understand any of that, but hopefully other people will be able to make some great Kirby Air Ride hacks at some point! Just curiously, about how far along in progress is this form? I'm sure it'll be a while, but I'm a pretty impatient person. Any possible dates you'd say that something like that would be possible?

 

[Tcll]

until the release, all I really have to work on is the stupid data system...
make sure everything's working and cooperating there... heh
(that's currently the problem I'm having and why I had to destroy the data system)

this update though is special in that I will now be a step closer to exporting HSD archives (Melee, KAR, WCTV, etc.)
the special thing I need for writing the HSD's relocation table is my new ref() function:

data = bu32() # BU instance

rtable = []
rtable.append( ref( data ) )

in the future I might actually do some updating to this for automated address updates
basically when data above an edit point is shifted left or right due to the value resize.

if you ref() the address of a value before updating the value before it,
the values that get shifted will have their addresses updated


anyways, after the data system, I've stil gotta make a few tweaks to the plugin system for injecting execution streams into process memory,
or for loading eggs or wheels from inside a plugin.


after that I need to work on the registration interface for libraries, so we're not simply injecting the lib's __dict__ into the script, but injecting a new dictionary composed of what's registered for the current process into the script.


I also would like to attempt to get script-widgets working in the GUI, as well as menus, hovering, and the keyboard...


so yea, don't be too impressed with what's working right now, as it's not really all that big... heh
but I do have some very big things planned, such as my Banana-Pi cluster server which will host the UGE-share service.

the idea of the share-service is to let people know who gets credit for what where it reflects on the rep of those who helped.

when I build the service, I will also be including a forum...
basically you login to the share service to be able to share the scripts you develop for UMC/UGE.
whenever you update that script, everyone using it will be notified of your update if of course they're connected.

the UPDATE plugin will allow you to browse and install scripts to your local copy.
(I do plan on making things more like chromium where what you've installed and developed is backed up on your account)


not only that, but jumping back to the plugins system, I've also mentioned I will be incorporating mods which will be loaded after the plugins.
basically, think of minecraft, except for the fact mods are actually mods, not plugins.
for any mods you develop, you're also given the option to share those as well.
the policy for mods though is that if it's considered beneficial or is popular enough, it may be integrated directly into the API or plugins.

when this happens, you will still be allowed to mod, but users will no longer be able to use your mod.
(not a big deal though as it'll be integrated into an API or plugin update by the time it's cut)
any further maintenance on the mod shifts responsibility over to me (or any other UGE dev) to integrate
(so don't expect integration until your mod has been finalized, as I don't want to have to deal with a list of updates)


in any case, I'm not listing all of my plans, but as you can guess, I have alot more just for UMC alone.

don't worry about the share-service though, as I have to actually build my server before that can happen... heh
I'm pretty much gonna be working on support and improvements for now.

 

[legoj15]

Wait, you're making you own 'language' for this? Now that's something I can most definitely get behind. I want to know more about it so I can fry my brain on trying to learn it(even if it's easy). I can't wait to see all of these concepts put into place in the release(or any public) version of UGE!

 

[Tcll]

lol trust me, the only monetary stuff I'd ever put on UMC/UGE is PayPal-donations
EDIT: oops, autism... that had nothing to do with money

but yea, don't expect much in terms of performance with UGESL, heck it's ran in python of all languages
it's meant specifically as a bottle-neck between stuff like PPC and x86_64 ASM
but then there's the fact I can do whatever I want with it because the backend is Python (completely dynamic)

it shouldn't be too hard to understand though, heck I have the node references here:
http://tcll5850.proboards.com/thread/271
and some older screenshots of the pre-developed language here:
https://picasaweb.google.com/110263988688421174390/UGESL

although I plan on making the language a little simpler by getting rid of the various array types and just using a plain mapping,
similar to what LUA does to define arrays.

EDIT: legoj15
would you like to join in on development??

 

[JohnTravolski]

This sounds great, but, once again, I don't think that I'd have the skills necessary to use a great program like this even when it's fully complete. Do you know of anybody who would actually make a Kirby Air Ride hack with this program?

 

[Tcll]

don't underestemate yourself, while I do state using UGESL won't be as easy as you might think, don't underestemate yourself as I'm designing this thing for someone like you to use
heck SIDE is meant to help you develop the already extremely simple scripting format.

so yea, don't sell yourself short now, if you can use redstone in minecraft, you should be able to do this. ;D

 

[JohnTravolski]

Alright, I hope I'm able to figure it out, then. By the way, I have no idea what redstone is (I've never played minecraft). Now, about how long do you think it'll be before this program is ready to do something like make a Kirby Air Ride hack?