A quickly curated list of games (pulling out odd titles like Swap Note, or Picross), starting the year after Smash 4.
2015:
Animal Crossing (Happy Home Designer and Amiibo Festival)
BoxBoy
Chibi Robo Zip Lash
Code Name S.T.E.A.M.
Fatal Frame: Maiden of Blackwater
Fossil Fighters
LBX
Mario Maker
Mario Tennis + Mario Party
Splatoon
Majora's Mask 3D + Triforce Heroes
Yo-Kai Watch
Yoshi's Wooly World
So, for "relevant" new IP, that gives us Splatoon, Fatal Frame, Yo-Kai Watch, BoxBoy, Code Name STEAM, LBX, and Fossil Fighters.
2016:
BoxBoxBoy
Bravely Second
DQVII
FE Fates
Kirby Robobot
HW Legends
Lost Reavers
M&S at Rio
Federation Force
Miitomo
Pokemon Go and SuMo (and Pokken)
Rhythm Heaven Megamix
Style Savvy: Fashion Forward
Tokyo Mirage Sessions
Yo-Kai Watch 2
Again, going by new IP, that's BoxBoy, Lost Reavers, Rhythm Heaven, Style Savvy, YKW, Miitomo, and Tokyo Mirage Sessions.
2017:
Animal Crossing Mobile
ARMS
Bye Bye BoxBoy
Ever Oasis
FE Heroes, Warriors, and Echoes
Hey, Pikmin!
Samus Returns
A bunch of Mario stuff
Pokemon Ultra SuMo
Splatoon 2
Kirby Blowout and Kirby Clash
Breath of the Wild
Tank Troopers
Yo-Kai Watch 2 Psychic Specters
For new IP, that's BoxBoy, ARMS, Ever Oasis, and Tank Troopers.
Total list of IP not yet represented in Smash that had games released in between the two Smash games:
Splatoon
Fatal Frame
Yo-Kai Watch
BoxBoy
Code Name STEAM
LBX
Fossil Fighters
Lost Reavers
Rhythm Heaven
Style Savvy
Miitomo
Tokyo Mirage Sessions
ARMS
Ever Oasis
Tank Troopers
I think even 2017 is too new. But that's not a lot of new franchises, and several of those are titles that slipped under a lot of people's radar. Yo-Kai Watch is a third party and unlikely to get in. What does that leave?