If Shantae were first party, she would already be in the game.
I would make Shantae's jab a simple jab combo, and make her forward tilt her hair whip. Making hair whip her ftilt allows her to use the move while standing still or while walking. I think the ftilt input would feel right for a hair whip, but that is subjective. With its range and disjoint though, there is no way hair whip can have the startup or end lag of a typical jab. Its frame data would have to be more in the ballpark of a typical ftilt. Depriving Shantae of a more normal jab would really hurt her close quarters options.
Shinuto
using Risky's pistol as her ftilt instead is an interesting idea, but my concern is that - especially with hair whip for a jab, and fireballs or storm puffs in her moveset - it turns Shantae into a dedicated projectile zoner, which I'm not too crazy about. So if we are sticking with your idea to have her hair whip be her jab, what other options are there for her ftilt? An obvious idea would be her kick attack that she has in Pirate's Curse and Half Genie Hero. From a design perspective, I don't like this choice. To save myself (and you) the trouble of an extended digression, I'll just say that it wouldn't "feel" as much like an ftilt as her jab, and depending on how it was balanced, would overlap too much with jab or fsmash in purpose. Another option would be to simply make her ftilt also hair whip, similar to Mega Man. While I like that choice for Mega Man though, I don't really like it for Shantae. It detracts from her versatility, and shrinks the effective moveset of a character that has way too many good options for moves.
Okay, back to considering ftilt as hair whip, and jab as a standard jab combo. To give Shantae's jab combo a little flavor from her games, she would punch, punch, and then perform the kick attack from Pirate's Curse and HGH. It would have unusually good knockback for a jab finisher, and at high percents would be a kill move near the ledge. It could also be her fsmash, but I think there is a much better idea for Shantae's fsmash...
...specifically, her fsmash should be the elbow dash from the original Shantae, as
BigShad
suggested. It would have very long startup, but exceptional power and travel distance, and low end lag. I disagree about using her twirl attack from her first game as her dash attack though, I think it should be part of her fsmash. If you press A during fsmash, the move cancels immediately into the twirl attack, which flies through the air and also has exceptional power and travel distance, but unlike the elbow dash has very high end lag.
That's probably enough explaining for one post, but since we seem to just be throwing all our ideas out there, I'll briefly run through what I have in mind for the rest of her moves.
Jab - Punch punch kick, the kick being taken from Pirate's Curse / HGH. Described in more detail above.
Ftilt - Hair whip.
Utilt - Hair sweep upwards, kind of like Greninja's or Shulk's utilt. I can't think of anything from her games that would work well for this one.
Dtilt - Hair whip.
Fsmash - Shoulder bash, cancelable by pressing A into twirl attack. Described in more detail above.
Usmash - Spin kick from original Shantae, cancelable by pressing A into drill kick from original Shantae. Exactly what
BigShad
suggested, in other words. That way all her fighter's gear moves are represented perfectly, and she has some pretty unique smash attacks.
Dsmash - Either a breakdance style leg sweep, maybe kind of like Sonic's dsmash in PM, or she swings a pike ball in a circle on the ground around her. I'd prefer the pike ball become its own item though, rather than a part of Shantae's moveset.
Dash attack - Scimitar charge, as
Shinuto
suggested. This means that she has to be dashing to perform scimitar charge, just like in her games, which I like. Also she has too many other good choices for B moves to make scimitar charage a B move.
Nair - She extends her limbs and spins in a circle, animation similar to her death animation in the original Shantae.
Fair - Hair whip. Just like dtilt and either jab or ftilt, this absolutely has to be hair whip.
Uair - Risky's hat. Pretty good-sized hitbox, slightly disjointed, low damage and knockback. Hold A to float in one direction or the other, allowing for it to be used as a secondary recovery move.
Dair - Scimitar dive, taken from Pirate's Curse. Similar to Link's and Cloud's dairs, but fairly safe move with pretty low power.
Bair - Spins around, swinging her hair behind her horizontally, like Ike swings his sword in his bair. Good kill move.
neutral B - Storm puff. Only one at a time, it takes a second or two before it starts droppiong its lightning bolt, which can also hit Shantae. Any character can push it back and forth horizontally with a hitbox.
I considered giving her Risky's pistol as a neutral B, but it is a more generic move, and making both neutral B and side B projectile really set Shantae up to be a zoner/keepaway kind of character, where I prefer her to be more of a versatile, bag-o-tricks spacing kind of character.
side B - Triple fireball. Startup and especially end lag make it so that this is not at all an oppressive zoning tool. Kind of like Ryu's Shakunetsu Hadouken, but smaller, a little faster and further traveling, no multi-hit, and there are three of them obviously.
up B - Cannon from Pirate's Curse. Shantae can use it up to three times. Each cannon blast individually doesn't give her much air, but the three combined give her very good vertical recovery. It does not put her into freefall animation. Each cannon shot has high end lag before you can air dodge or use a different move. Pressing A or B before the end lag expires causes Shantae to use her next cannon shot. If you try to do an aerial between each cannon shot, you aren't going to get much if any vertical height, because the of the end lag after each cannon shot.
down B - Transformation dance. Shantae cycles through three dance moves. Release B during a dance move to transform into the corresponding animal form - spider, tinkerbat, or elephant (tinkerbat is negotiable). Like Peach's or Pokémon Trainer's down Bs, Shantae must be grounded to use this move. Details about these animal forms are too much to describe here, plus I did so in a previous post anyway.
grab / pummel - Just a normal grab, and she knees her opponents for her pummel. I do not think she should grab opponents with her hair, I think that would be kind of weird. It has never been prehensile in any of her games.
I don't have any special ideas for her throws. Like most characters, I think she would probably just have some regular *** throws, made up for Smash. Whether she has a combo throw just depends on game balance. I don't really see her as having a kill throw, but it's not impossible.
forward throw
up throw
down throw
back throw
final smash - Hmmm I'm not sure. I can't think of anything better than the gem mini-game, which is actually a pretty cool idea. I'll go with that I guess.
Oh, and one more absolutely critical part of her repertoire I completely forgot about!
Up taunt - She strikes her victory pose from HGH and shouts, "I'm RET-2-GO!"
Side taunt - She flips off the camera and says, "eat your heart out, Shovel Knight!" jk obviously, I don't have any ideas for this one.
EDIT:
Shinuto
suggested the lamp holding dance from Pirate's Curse, that would be perfect I think.
Down taunt - She enters dance mode from the first game. In dance mode, she can do any of her dance moves from the original Shantae. Shield, grab, or jump inputs cancel dance mode. If she does any of the complete dances from the first game, she does some corresponding taunt. For example, do the monkey dance and she will transform into the monkey and do a taunt as the monkey. Do the dance for Water Town, and Bolo will briefly appear and do a taunt with Shantae. For simplicity's sake, there is no special rhythm you have to hit for the dances to work.
Alternatively, you could leave out the up taunt and side taunt, and just pressing any taunt input puts you into dance mode.
EDIT:
@ProtoTwi yeah I remember seeing a trailer for her Smash Bros. Crusade moveset and being impressed. I haven't messed around with Crusade myself though, so I don't know her moveset in it very well.