This is going to be a long post. I've thought a lot about this kind of stuff, and if I were on the dev team (which I'm not, sadly) this is what I'd suggest.
Brawl vs Melee is a foolish debate to be fixated on from a development standpoint. Your main goal here is to make a smash game that follows the general ideas of the series while going in a new direction that makes for the best game yet. Melee and Brawl are both instructive games for your design ideas, but it's not like they are opposite ends of a spectrum and you're trying to find some middle ground.
Ledge behavior is a problem in both games. It's easier to abuse ledges in Brawl, but planking is possible in Melee too it's just harder and Melee doesn't have Meta Knight or G&W with his good nair from Brawl. You also have really hated interactions with ceilings near ledges (like Melee Battlefield or Brawl Final Destination) that deserve some serious thought as well, and ledge detection coming from the "wrong" side (very easy to do on pass-through floors like Melee DK64 or Brawl Delfino Plaza) kinda sucks across the board. To get to it, you need to think of the core gameplay of ledges. Grabbing ledges is supposed to be mostly easy if the opponent didn't actually follow you off-stage which is a huge risk on their part, but being on a ledge is supposed to be a disadvantaged position. From the ledge, the endangered player needs to be trying to find a way safely back onto solid ground and neutral position. The ledge offers five options: attack, climb, roll, jump, and drop. The abuses come from the fact that the fastest and most flexible option, drop, can be easily followed up by a simple regrab. In fact, drop carries extremely little risk compared to the other options so it mostly dominates the ledge game anyway.
The solution to planking and to establishing the ledge as a workable but disadvantaged position is simple. If you ledge drop, you can't grab another ledge until you reach solid ground or get hit. Ledge drop is still pretty good like this, but there's only really one viable thing to do if you can't grab any ledges at all if you don't get back on stage and that's immediately move to get back on stage. That's still your fastest and most flexible option with most characters (double jump and aerial of choice tends to be a safer thing to go for than any of the more restrictive ledge options), but of course ledge traps are a lot better since as the on-stage defender you can move forward and throw out hitboxes in response to seeing the opponent drop knowing where they have to go next. Of course, if you move too far forward before they commit to dropping, they'll just roll right past you, and a lot of the seemingly clever ways to cover both roll and drop would lose to at least one of the generally inferior three options (climb, attack, or jump). This whole suggestion would be both revolutionary and intuitive. It would be revolutionary because it would completely change the nature of the ledge game as it pertains to the currently strongest option, but it would also be intuitive given that it would always have a really clear causality why you couldn't grab the ledge.
Controlling how intentional ledge grabs are and how hard it is sometimes to grab ledges is best addressed by examining another good mechanic all smash games have. If you hold down, you can't grab ledges. This is great mechanic that I'm sure every decent player here has made great use of already. I propose that the converse should be true. If you are holding up, ledge detection should grow to be very generous in all directions. It should be so generous as to scoop you out from under the ceilings on stages with them and very good at sucking you to the ledge from inside the ground as well. This will mostly not interfere with legitimate tactics since holding up is not generally useful though there should be some delay between expanded detection and hitting up (a small handful of frames) to allow for up specials to be performed in the vicinity of ledges without forcing a ledge grab and so generally it will be looking for "hold up" and not "tap up". Then set your baseline ledge detection when holding nothing/left/right to something a bit less generous than Brawl but much closer to Brawl's detection than Melee's with the caveat that if you are facing backward ledge detection should be very strict (it should still be possible to grab ledges while facing away and not holding up, but it should require you to be virtually right on top of the ledge). That makes neutral ledge grabbing still mostly easy but less likely to happen at unintended times which should be your biggest goal (making sure as often as possible players are getting the interactions they want when they try for stuff).
Hitstun is another sticking point. Generally the issue is this. Landing hits should be rewarding, but loops and long combos are bad for the game. Part of this is simple parametrization work that needs to be done carefully; moves like Sheik's ftilt in Brawl should just not exist like they do. A hardcoded lock against chainthrows could help as well ("thrown" is a different state than "hit"; just make "thrown" ungrabbable which incidentally also does a lot to make Ice Climbers a non-stupid character if they're going to return at all and of course "grab release" states should also be unthrowable). However, once you get past better parametrization practices and chainthrow fixes, you get to the core issue which is that in Brawl landing hits too often doesn't seem to be particularly great for the attacker whereas in Melee getting hit often screws you just too badly. Of course, situationally you find counter-examples both ways like getting hit with certain moves near walls in Brawl or getting hit with several of the generally bad moves in Melee. I think the best solution is something like hitstun scaling but designed for smash. Make hitstun somewhat more generous than Brawl (the interruptible semi-hitstun is a neat gimmick but should probably be put at 2/3 through hitstun not 1/2), but as moves are comboed actually scale base knockback up on each additional hit. Most moves that create exploitable situations rely on keeping the enemy close so an in-built system that causes comboing to push enemies away would be pretty great. As another practical matter, it would also quickly make wall locks create enough knockback to make the hit techable. The overall effect would probably be that upon landing one of those "combo starter" moves, instead of trying to link as many fast moves as possible to do stupid damage you'd do a minimum of those and try to transition into either the most powerful move you can combo into or just something that will put the enemy off-stage. It should also generally be implied to go along with hitstun, but shieldstun should probably also be slightly higher with repeated moves into a shield with continuous blockstun increasing pushback. Either that or just let the player powershield the first hit of any new move even if you're in blockstun before it hits (this would also include being able to powershield the start of each loop of a rapid jab).
Mashing also needs fixed though it's not as serious as some other issues. Grab breaks are a pretty dumb skill to reward. In principle, the choice of whether to mash to break throws is interesting since if you mash to to try to break out you most likely give up DI. However, a handful of superhuman mashers have a real advantage over everyone else because of a pretty ridiculous ability they have to hit buttons really, really fast. The simple solution for this mashing situation is just to cap mash inputs within a certain window so the game rewards mashing vigorously but being a super-human masher no longer helps you. Also, breaks should be a bit harder at low percentages and a bit easier at higher percentages. Characters with slow pummels often can't go for any at all until high damage, and that's pretty dumb. At said high damage, characters with very fast pummels can often do a whole lot, and that's also pretty dumb. Flattening out the break timing a bit (and also trying to avoid really awful pummels existing at all) would make the whole "go for pummel?" game play out better across the cast.
The other mash issue is how the game handles SDI and the deal with multi-hit moves in general, and it's a bit more complex how you address that. Again in principle, multi-hit moves having the inherent risk of being escapable versus their generally superior damage is good design. Problems come when you start introducing particularly good players who can very consistently escape a majority of multi-hit moves in virtually all circumstances mostly making those moves just plain terrible. Especially bad is a move like Pikachu's down smash which is completely devastating against bad players but nearly worthless against good players because bad players always eat every hit and good players almost never eat more than two hits. I think the solution is to minimize or even remove SDI but increase the role of aSDI. The idea would be this. You should generally be able to wiggle out of multi-hit moves if you get hit near the edge of the area (like if G&W's beloved turtle hits you with his nose or such) but if you get hit deep it shouldn't be escapable. This will make multi-hit moves retain their basic design of more reward for potential lack of reliability but make the reliability based on situation of use rather than on the mash ability and game knowledge of the opponent.
Airdodges are another core issue, and it goes back to the bigger issue of using excessive aerial mobility to run away. This is a tough one since stages that are defective by design (i.e. Hanenbow) will always enable such tactics in any smash styled engine. Airdodges are the thing though since they let the fleeing characters remain safe. In general I think airdodges play out better in Brawl than in Melee since Melee airdodges just get you killed if not used very near the ground or a ledge, but both cases miss the mark of what should be an "aerial spotdodge" that is generally safe to use but punishable if baited. Brawl airdodges have two issues. One is that they generally recover just too fast, especially the airdodges of most of the new to Brawl characters which are most probably for animation design reasons that never should have been important to the game like this 10 frames faster than normal airdodges which is a whole lot since we're talking about less than a second animations. Airdodges should be a little slower than a standard Brawl airdodge (like Jigglypuff's) but generally quite close to that (maybe 55 total frames?) and nothing should be in the realm of the faster ones (no more Wario airdodge!). If they all don't have the exact same frame data, it should be a case of deliberate design (like maybe DK and Bowser have a few frames faster airdodges to make up for their general weakness in juggle situations otherwise). Further, airdodges should be used as a "dodge" not as a part of your mobility. Landing during the invincible frames of an airdodge should have a brief but real landing animation (after the invincibility ends, they should of course auto-cancel like most aerial attacks can). This animation should be very brief, just long enough to be hittable if the opponent is specifically going for it knowing what you're trying to do but not punishable on reaction under any circumstance. Furthermore, while moving when airdodging is inevitable, airdodging while bobbing around shouldn't be possible. Air control inputs, including beginning to fastfall, should simply be ignored during an airdodge. The short version of what this means is that airdodging would not affect your character's velocity but it would completely prevent input based acceleration for its duration, which would further greatly limit potential use by someone like Wario for the purpose of running away. Combined with ledge changes and assuming generally reasonable stage design (i.e. not Temple), run-away would probably not be particularly good at all like this.
Certain new move properties need to exist:
-"No powershield": This should exist on very slow moves like Falcon Punch. These are the easiest moves to powershield but also the ones that as a matter of design we should want powershielded least often since they're the easiest to avoid otherwise and powershielding tends to be a reaction response no prediction required. Similarly, fully charged smashes should not be powershieldable (but any partial charge, even one frame short of a full charge, should still be powershieldable if the move is in general). I wouldn't make this a common hitbox flag for sure, but I think it would be nice to exist.
-"Weak clash (percentage)": Currently hitbox clashing is very simple. Hitboxes that do more than 10% greater damage than the other destroy the opposing hitbox. If they're within 10%, they clash. Certain moves, or should I just say Mach Tornado, use this to have absurdly high priority. A special move property by which any given hitbox can be destroyed by variably weak hitboxes (maybe for Mach Tornado the opposing hitbox only needs to be 5% stronger instead of 10%) would be great. This should for the most part be seldom used but would be a bit of a reversal on transcendent priority. It's a rare type of priority that results in reduced, not increased, effectiveness overall.
-"Unscaled" and "Minimum scaling": These come into play for the same reason. The overall damage and knockback scaling system Brawl uses is actually very good, but some of the outlier cases can get kinda dumb when small hits designed to set up for other hits don't actually reliably link. Off the top of my head I'm mostly thinking of jabs which should have minimum scaling such that jab combos are still actually combos (of course, several jab combos in Brawl never combo which just shouldn't be!), but I could swear there are other cases as well.
In general special incapacitated states should not be loopable. An opponent flinching from a footstool shouldn't be capable of being victimized by another footstool. A tripped opponent shouldn't be trippable by a banana peel. This is how infinites are born. Brawl has all sorts of corner abuse cases like this, and it could all be fixed by just making this the general rule for all special incapacitation states.
Sudden death should respect percentage differences between players as a tie-breaker. Actual sudden death itself should, instead of using raining bombs to force games to end, instead simply begin to shrink the blast zones forcing the players closer and closer together, creating a king of the hill situation sure to quickly resolve. Serious players should not have to ignore the game's in-built tie-breaking system!
The only real speed issues that separate Melee and Brawl beyond match pacing issues (which are the implication of the whole system, not a matter of actual speed) are some Melee ATs which to be very blunt I don't think should come back (they make the game a whole lot less accessible and design wise are mostly side-grades to the game, making it different but not necessarily better or worse) and fall speeds. Fall speeds I would handle like this. The difference for a typical character between fastfalling and normal falling should be increased with normal falls generally being Brawlish whereas fastfalls would just be, well, faster. Likewise though, I think characters should have greater variety in fall speeds. I don't think it's a problem for someone like Falco to fall like Falco did in Melee (as long as you don't introduce the horrible hit physics that made vertical KOs mostly depend on victim fall speed, much to the disadvantage of characters like Bowser), but I don't think someone like Jigglypuff needs to fall an iota faster than she did in Brawl and I don't think a character should have to be like Jigglypuff to be floaty for that matter. Just increasing the spread I think could make everyone happy so people who like floaties won't end up in the "Jigglypuff or nothing" sitaution they were in in Melee but likewise people who like falling fast can have their rock characters as well. We expect a roster of around 50 so I'm sure there's plenty of room here to make everyone happy in this regard.
The amazing thing is that I could carry on for at least twenty times this long, but I think I've hit most of the high points. The summary is that they should look at their engine as a whole, think of the things that have given them problems in the past, and work to re-enforce the underlying design while delivering the best possible gameplay.
