Dude, you're missing the biggest part.
Ganondorf refers to the Twili as "your people". If he was banished along with the Gerudo during MM, then he would have sympathy for them because they would have been "his people". We know this: Ganondorf was banished to the Twilight Realm either during, or post-MM. Were the Twili already there? If so, then their ancestors had to have been banished before OoT, and since the Gerudo were present during OoT, that rules them out. If the Twili were not in the Twilight Realm when Ganondorf was banished there, then that's suggesting that the Gerudo were either banished there at the same time or after he was banished there, but given that scenario, he would have known that his people were those banished and would hold sympathy for them.
And again, for a race of women who birth one male every hundred years, how can they uphold a royal family? The Gerudo have not shown a sense of royalty in any of the games they've been present in.
Ganondorf refers to the Twili as "your people". If he was banished along with the Gerudo during MM, then he would have sympathy for them because they would have been "his people". We know this: Ganondorf was banished to the Twilight Realm either during, or post-MM. Were the Twili already there? If so, then their ancestors had to have been banished before OoT, and since the Gerudo were present during OoT, that rules them out. If the Twili were not in the Twilight Realm when Ganondorf was banished there, then that's suggesting that the Gerudo were either banished there at the same time or after he was banished there, but given that scenario, he would have known that his people were those banished and would hold sympathy for them.
And again, for a race of women who birth one male every hundred years, how can they uphold a royal family? The Gerudo have not shown a sense of royalty in any of the games they've been present in.