I found this in Zeldapedia. Pretty interesting stuff.
"It is said in the game that the Twili descended from the Interlopers, though the race of these interlopers is never revealed. All of the following are reasonable candidates since none of them are ever seen in game (thus supporting the story that they were banished) and that there are some physical or cultural similarities.
Sheikah
The main similarities are with the Sheikah, as both are described as wielders of Dark magic, have red irises and dwell in the shadows (even being referred to as 'the shadow folk'). Also the symbol of the Sheikah, a stylized eye with a teardrop, is present on both Zant's throne and on the back of the Fused Shadows. The version on Zant's throne appears several times throughout the Twilight and on the black walls separating the parts of Hyrule covered in Twilight from the rest of it. Notably, in the Twili version of the eye is smaller and rounder, and missing the three-triangled eyebrow. The Sheikah becoming the Interlopers may be the event that caused them to either pledge their loyalty to the Royal Family or be banished.
Ancient Ones
Another race called the Ancient Ones, though not featured in the series and is only briefly mentioned in The Legend of Zelda: Majora's Mask, are those that used Majora's Mask in their 'hexing rituals'. There are many similarities between the Twili and Majora's Mask itself, which implies that the Twili could be the same race. The similarities involve some shapes featured on Majora's Mask being present on Midna's helmet, the Fused Shadow (the eye especially). Zant's behavior is very similar to creatures specific to Majora's Mask and the Mask itself, such as Zant screeching in a similar fashion to Majora's Mask, being a sympathetic villain much like Skull Kid who wore the Mask. During the mention of those who used the Mask, the Happy Mask Salesman mentions that the mask was hidden in shadows, and the Twilight Realm is sometimes called "The Realm of Shadows". Also, the entrance to the Stone Tower Temple in Majora's Mask appears to be a large face with it's tongue sticking out. Zant's helmet also resembles a face with it's tongue out, which would lead to the belief that the ones who built the tower (sometimes believed to be the same race as the ones who made the mask) could be the Twili (possibly also connecting to the Garo, based on one of their undead spirits being found in the tower). However, as it is implied that Termina, the land in which the events of Majora's Mask take place, is in another realm entirely, it is unlikely that they would be the same specific people, though it is still possible that they are of the same species.
Gerudo
Another possibility is that at least a portion of the Gerudo tribe were the ones who gained the mysterious power of the Fused Shadows and were cast into the Twilight Realm. Midna's story that the Twili race were originally a tribe of thieves, her mention that the Twili's ancestors "lost their king to greed", and why the entrance to the Twilight Realm is located in the Gerudo Desert, though it is extremely difficult to reconcile with Midna's complete unfamiliarity with Ganondorf. As for appearance, Midna does have the classic red-orange hair of the Gerudo and her true form is tall and slender, just as the Gerudo women once were. Also, there is a Gerudo pattern and converted Gerudo symbol on the front of Zant's robe, which he wore even before his encounter with Ganondorf, whom he worshiped as a god. Gerudo also worship Ganondorf as a god, which a Gossip Stone tells Link.
Other
Finally, there remains the possibility that the Interlopers were not of any one race or tribe, but were a cult of people made up of several different races and cultures. Evidence to support this is Lanayru's vision where Link kills his closest friend to seek after the Triforce, only to become an Interloper himself. It can be interpreted that several people throughout Hyrule betrayed their fellow countrymen to obtain the Triforce and eventually united as the Interlopers. Once sealed in the Twilight Realm, the long-term effect of the Twilight on them could have transformed them into one single race, and may explain the large differences in appearance that some Twili have from others."
I think the ancient ones are another possibility, seems reasonable.
