I can think of few that instantly came to mine but some of these could be a bit of a stretch so fill free to disagree with me or share opinions.
Ganondorf was the first character I thought of when you asked high risk and high reward. I think his gameplay essential revolves around applying mix-ups, intimidation, and if you guess right a massive reward. He is slow, combo food, can easily be pressured, and has a decent recovery, but not enough for him to go to the blast zone or recovery well vertically. What he does have however, high survivability, massive damage that scares anyone, and moves that conditions the opponents to make mistakes he wants to capitalize for big damage. But to do this, you need to establish control of the battle and opponent. this is where that high risk comes in. If the opponent guess right, Ganondorf can be punish or have to continue in the neutral, which is a bit better for him in this game, but still a bit of a challenge. If your looking for someone with that type, defiantly try him.
Jigglypuff can be consider a high risk character IMO. I like to think of her as glass canon in the sense that her offstage present and aerial game makes her a strong candidate, but her weight makes her at death percent's around 40 plus ( different factors ). Not to mention that breaking her shield result in instant death. Now, the main thing that ultimately makes a high risk is Rest. Rest can be hard to land but being on the receiving end is just as nerve racking. one slip up and you basically gave up a stock to the opponent. And if it doesn't kill you, you are left receive DOT damage. But, if she misses, your free to do whatever want to, which is scary because she can die by just a gust of wind alone. definitely a gamble when it come down to it. Feels like you have to play a near perfect game of control, spacing, and patience.
Lucario thrives off taking risk. At low percents, he does little to no damage at all. But in a way, his combo and stringing potential is better ( Not always, I don't know a lot about Lucario ). His range is left to be desired at lower percents as well. But when he receives damage, you better watch your back. Everything about him becomes almost 2x stronger ( Aura does have a max limit from what I am aware of ). His range, his damage, his normal, his specials. Everything becomes so powerful. * most noticeably his aura sphere *. This makes him a killing machine at the cost of your own stock being in danger. You are still in tough spot even with this power. Yes, you can space better and killer harder at time but one mistake and you can be sent flying, he isn't as heavy as others. But a worthy trade if you ask me.
I feel like you guys know where I am going with this. Little mac has the ground speed and damage to secure the stock, but he lacks good aerials and though his recovery options are better, its unfortunately not good enough to contend with others. This means he is easily one of the easiest character to gimp basically being able to kill him at almost any percent there is. If you can hang in the middle of the stage or avoid not going out of the board, I still feel like he is a decent pick. He has a great kill option in the K.O Punch and his attacks can armor through others with the right timing. His counter is also powerful so to me, it all comes down to playing a well balance of aggressiveness and defensiveness. If you really want, you can actually pick one or other but be careful not to let yourself stray too far off the stage. This could be more of a design choice for the character rather a means of making it a playstyle but I think Little Mac deserves a mention.
* ( Same thing can apply to Rosalina and Luma ) * I feel like this can be a bit of a stretch and maybe a bit biased from the point of view that this my main for years. But Ice Climbers rely on having both the duo with you in order to have full access to damage, desyncs ( not needed tbh ), and a good recovery. Having 2 characters can be difficult on the opponent but on the user as well. Keeping Nana or Popo alive especially against someone who knows the matchup well isn't as easy as it looks. Once your partner dies, you are left fighting alone with only half a character. Your damage is worse, your moves aren't as powerful, and your recovery is Little Mac tier. So your main objective really is to keep your teammate alive to have all the full benefits of being a dual character. Once you get it down thought, they can be very challenging to beat, especially one on one battles. It makes them challenging but soo fun to play as by many.
originally I didn't have these guys down but I thought I should add them because not only are they high risk characters, but they are simply also hard to use even. They are extremely short ranged and even in some situation out pace but some characters making it a bit harder on them. The new turn around animation in one on matches is brilliant and true to the characters, also having its perks, it can harm them too. trapping on ledge is easier but using Bair as a killing option or even as a way of conditioning is a bit harder. Their combos are tight and precise. They are hard to pull off but when you get them down and their movement options, these are really threatening characters. Again, a character I don't know much about, but from what I seen, they have the kit to be decent, it takes time. Mastery of their focus attack is key as it gives them landing options, a counter for series damage, and ability to absorb attacks because they can be easy to pressure, combo, and overall wall them out.
Overall those are the main examples that come to mind but I am sure there are more character in this game that can be considered high risk high reward characters.