I don't think there's a time when you can't fast fall. What I meant is when you try to fast fall but your move's animation is not completed (like ZSS's UpB, which lasts pretty long), so you can't move in another direction yet.
When you UpB with ZSS, after the blue color goes off from he legs, she still does a 360 before you can influence the direction. If you try to fast fall when she does it, it won't work, even if you keep the c-stick down until she lands. That's probably why it doesn't work for you; you press down during that animation, it doesn't work, and then you press down again after that, and THEN she fast falls. Try just waiting a bit longer.
Do you also have this problem when doing other moves? Try to jump and Uair before reaching the peak of your jump, then fast fall. Since Uair is really fast, you should be able to fast fall a lot more easily than after your UpB. Jumping without attacking, and fast falling after that, is also easier, but a little harder than after an Uair, at least for me. It just depends on the move you use. Try to get the timing of the animation right, it varies for every move.