This is only my particular distinction, but I'd say its the difference between a brand, a form, and a formula.
Star Fox has in many ways become the former, going from rail shooter, to an RTS, to Zelda clone, and so many elements in between to the point where occasionally shooting things in space is about the only element can find in all the titles (and not much of it in Adventures).
3D Mario typically has form, always being a platformer with Mario collecting certain macguffians via power ups. However its manifested in different ways; contrasting aesthetics, linear vs. semi-linear vs. nonlinear, amount of moves Mario has, and so on. There is a common thread gong through them and you can sense an evolution, but barring direct sequels like SMG1 to SMG2, each one is taking the core platformer tools and doing something distinct with them.
The NSMB series eventually became formulaic to me because the titles eventually felt so close to each other in terms of gameplay and especially aesthetics that it became tricky to discern a stage in one game vs that of another. The 8 worlds with a bonus 9th world, constant repeating visual tropes (grass, desert, water, cliff, sky, lava, jungle, etc) and a fundamental lack of any real experimentation beyond a few new enemy types and select power-ups? It ends up creating the video game equivalent to TV dinners, you know exactly what you're getting and you're very likely not to be surprised.
I know these may seem like arbitrary distinctions and there's a whole discussion about who spin-offs fit into this or what makes up the core of a franchise, but formulaic as a concept is simply one where I know it when I see it.
