I'm not positive about this, this is just what I think from my observations though...
The first time you whistle (or any time when your pikmin are shuffled) it sorts them according by color (ex: all the reds one line up together). I'm pretty sure when you do this, the color at the front stays at the front, but im not sure about that part. So if you pick say the following colors: Y R B Y R R, when you whistle itll become Y Y R R R B.
So once they are arranged in color order like that, whenever you whistle it switches to the next color. So whistling for the above arrangement (Y Y R R R B) makes it: R R R B Y Y (the yellows are cycled to the back).
I'm pretty sure thats it, but im not positive, so if I got it wrong feel free to correct me.