Hm. I think the point about the Gerudo is rather valid. How does a group of female-only-save-for-one-guy 's suddenly create a culture with (we're assuming; we don't know how many guys or girls there are, but considering Midna probably had a father and that Zant is a guy...) multiple guys? Or how does it even keep going? Though, we forgot to mention that the Twili share the Gerudo's orange hair.
The Gerudo are dependent on other races for their survival. But eventually their blood will get diluted enough that they aren't a separate tribe, barring magic or divine intervention. I think the reason why the Gerudo aren't in TP is because they simply had to move in with the Hylians because there was little to distinguish the Gerudos from the Hylians (and humans). So it looks like the Gerudo were on a timer of sorts, only destined to go so long before fading away. Why would the goddesses banish a group that was going to fail anyway? I'll come back to this in a minute.
I think the Sheikah explanation is very likely. However, why would a group of people so disciplined and devoted to the Royal Family suddenly have a desire to rise up against Hyrule? Yes, there are individuals who have their own ideas, but how could the others who are so devoted to the Royal Family allow the individuals to get under the radar? Maybe it was just a mistake, maybe it wasn't. This is point 2.
Thirdly, the Twili show some characteristics of other magic-heavy tribes. The magic-heavy tribes share characteristics between themselves, such as symbols, magic, hidden ( yet broken and tormented) pasts. I think they (the magic-heavy tribes) all came from a mother tribe, the tribe the Dark Interlopers came from. I'm going to steal the "Zelda's Babel" idea for a second....
Okay, perhaps there was, originally, one tribe of magic. More powerful than the Sheikah, Gerudo, and Twili (and whatever other tribes we've seen that may be candidates). They know they're powerful. They're not as unified as the smaller tribes, however. They all have a darker nature about them (in mentality, or skin, or both). Some delve into darker magic, others use their magic intuition to hurt others and steal from them. Some use their powers to help people, even though they can't fully cover up their dark nature. They are left unwatched by the goddesses, spirits, and neighboring tribes. They learn the secrets of the land and become extremely powerful.
Now, a few individuals who had been looking into the darker magic decide that they want even more power, so they look for the ultimate power, the Triforce. These individuals gather some people from their smaller tribe and seek it out. They build themselves a weapon to strengthen their magic and help them look for the Triforce. Story of the DI.
The goddesses see this and realize that this one tribe is too powerful together. So they split the tribe up, roughly according to the smaller factions the tribe had already developed. The goddesses also tweaked the tribe some in terms of physical, mental, and magical capabilities so that even the new tribes couldn't rebuild themselves completely. The darker tribe was sent to the Twilight Realm, and the Twili were formed. The Thieves were sent to the desert to become physically darker and isolated, so that they couldn't steal from others so easily. They were also given a self-termination mechanism: Only one guy every 100 years. They became the Gerudo. The nicer group was probably altered the least, but were given an ugly temple filled with evil to remind them of their scarred past and banished cousins. Behold the Sheikah.
Each group took the customs of the mother tribe, and each focused a bit more on different customs. We see that each of the tribes we know have similar qualities, but nothing very concrete.
Also, it may be that the Ikana were also a tribe from the Mother tribe, who were banished to Termina. However, it may be that the goddesses did not tamper with the Ikana too much because the goddesses did not like Termina (Termina had forsaken their goddesses). The Ikana prospered for a time and did their own deeds in Termina, even constructing their own weapons for a possible re-attempt at the Triforce (giant's mask, Majora's Mask possibly, the Stone Tower). After a time they were destroyed, possibly because the goddesses realized that their dislike for Termina was not worth having a bunch of magical beings running around destroying everything.
The Zuna also built great magical weapons. However, I haven't played FSA, so I can't comment too much on them. They may have been another faction.
This can also tie into Ganon's jab at the Twili. As sister tribes, the Gerudo and Twili (at least, those who remember the split) probably feel hostile towards each other. Namely, the Gerudo are probably more hateful and the Twili more submissive. So Ganon was simply talking down in the hateful-competitive way his tribe raised him to be.
This still doesn't explain where the Sheikah went (maybe they too had a self-termination mechanism), nor why the sages would send Ganon into the Twilight Realm (maybe they had hoped the Twili would recognize a Gerudo and destroy him themselves due to a want of revenge). There's probably a ton of holes in this, but it's an idea that popped into my head that I wanted to share.
