Nope. Transparency is always defined in the palette value itself. It's the alpha-value.
Couldn't you just post the TEX0 and PLT0 files here, or the TPL for that matter? I might be able to fix it.
To make it simple, a byte consists of eight bits. A byte always goes from 0 to 255. To keep it simple, I'll number each bit like this:
7654 3210
Basically, the most right bit, bit 0, can have a value of 0 or 1, while bit 1 can have a value of 0 or 2. So, with each subsequent bit, the values multiply by two. It's logical, because:
0 - 0
1 - 1
10 - 2
11 - 3
100 - 4
101 - 5
110 - 6
111 - 7
To make it a bit easier for us to read, we use a hexadecimal numbering system. To explain it, we humans normally calculate in the decimal system, which counts from 1 to 10. We use the numbers 0 to 9. In binary, we count to 2, by only using the numbers 0 and 1.
Also, we form larger numbers by multiplying it by a certain value.
To explain what I mean, let's take the number 1337. It consists of 1 times 1000, 3 times 100, 3 times 10 and 7 times 1. To put it in powers of:
1 * 10^3 = 1 * 1000 = 1000
3 * 10^2 = 3 * 100 = 300
3 * 10 ^1 = 3 * 10 = 30
7 * 10^0 = 7 * 1 = 7
So, when we add these numbers to each other, we get 1337.
Same goes for binary. we take the binary number 1011 for example:
1 * 2^3 = 1 * 8 = 8
0 * 2^2 = 0 * 4 = 0
1 * 2^1 = 1 * 2 = 2
1 * 2^0 = 1 * 1 = 1
Added to each other, it becomes 11.
However, most of the times we end up using hexadecimal when hex editing. Hexadecimal basically is counting in numbers of 16, so counting from 1 to 16, using the numbers 0 to 9 and A to F, which means 16 characters. A represents a 10, while F represents 15.
So, how does it work? Well, let's find out with the hexadecimal number BF40.
B * 16^3 = 11 * 16^3 = 11 * 4096 = 45056
F * 16^2 = 15 * 16^2 = 15 * 256 = 3840
4 * 16^1 = 4 * 16 = 64
0 * 16^1 = 0 * 1 = 0
Added together, they make 48960.
But how can I convert from hexadecimal to binary? That's actually easy.
Like I said, hexadecimal counts from 1 to 16, and binary counts from 1 to 2. We also know that the highest value of a hexadecimal number is F (or 15). And, we know that 1111 also is 15, so, we can conclude that:
F = 1111.
So, to fill a whole row:
0 = 0000 | 4 = 0100 | 8 = 1000 | C = 1100
1 = 0001 | 5 = 0101 | 9 = 1001 | D = 1101
2 = 0010 | 6 = 0110 | A = 1010 | E = 1110
3 = 0011 | 7 = 0111 | B = 1011 | F = 1111
So, knowing that, you can convert each hexadecimal value to binary.
Now, what use does it have? Let's say we have the color values of BF40 and 1337. Let's analyse them. First BF40:
10111111 01000000
Now, this value starts with a 1. We know that in RGB5A3, when the first bit is a 1, the color is RGB5, which is 1rrrrrgg gggbbbbb. We are going to calculate the final value by deviding it by the maximum value and multiplying it by 255. So:
10111111 01000000
1rrrrrgg gggbbbbb
Red = 01111 -> 123
Green = 11010 -> 213
Blue = 00000 -> 0
So, in this case, the RGB value is (123, 123, 0).
Now let's take a look at 1337:
00010011 00110111
When the first bit is a 0, we use RGB4A3, which is 0aaarrrr ggggbbbb. Again, we're deviding it by the maximum and multiply it by 255, except for the alpha value, which we multiply by 100.
00010011 00110111
0aaarrrr ggggbbbb
Alpha = 001 -> 14
Red = 0011 -> 51
Green = 0011 -> 51
Blue = 0111 -> 119
So, here the RGB value is (51, 51, 119) with alpha value of 14%.