I saw a comment elsewhere that I think sums up the weird place Donkey Kong is within the core Nintendo model and it's that they always prioritize DK the character... but not really as much the franchise itself. It's such a staggeringly simple idea, but it makes a lot of sense and arguably explains so much of the franchise's history in the last 25 plus years.
DK himself was always around in some fashion and appearances in so many of the Mario spin-offs indicated just how critical he was seen as including in things like Mario Kart and Smash. The Mario vs DK titles had consistent entries, the original arcade game is often highlighted/referenced, even in things like advertising and marketing he's pretty well represented.
It's just that so much of DK's world outside the context of relating to Mario has so often fallen by the wayside because its seemingly not viewed as critical/important to keep as relevant. Diddy managed to pop up in various titles because every Mario/Sonic needs their Luigi/Tails, though even hardly guaranteed consistent appearances> He had 15 years between mainline Mario Kart appearances (and literally got in the last possible DLC of the 8 Deluxe to boot). Dixie Kong's never managed to get any non DKC playable appearances barring Tour (which basically included everyone Mario adjacent under the sun).
The 2000's period might be most illustrative of this. How could Konga and Jungle Beat be the big output of the series with both having few elements from the actual games that were most successful for the brand? Because they had DK in them and that seemed to be enough. The series was basically a springboard to try things at that point and to be fair the reticence by certain development teams to not include elements from the Rare games because they personally didn't create them is understandable. It's just that them being so far removed from even the spirit of the 90s titles in many respects suggests that sort of tonal consistency was simply not really a high priority when it came to the series.
Fascinating to note also is that Western teams seemed to be relied upon for the mainline titles typically expected (and often do) sell best: Rare for the GBA remakes, Retro for DKC Returns & Tropical Freeze, even little studios like Monster Games for re-releases like the Returns 3DS port. Part of that is likely that was simply easier utilizing them for it then various internal teams or an Eastern third party, though the Nintendo EAD release (Jungle Beat) being one that feels almost the most removed from the other mainline games could be suggestive of where NoJ's specific interest in the franchise was. Also, notable that it was a NoA executive specifically in Reggie Fils-Aimé
who objected to Donkey Konga & believed it would hurt the brand. It's an objection that could be just indicative of some differing regional views on what the franchise should be.
Finally, I think the last 10 years bear this out to a degree too. No new Donkey Kong games of any type (with even the recent Mario vs DK being a well-done remake) and the biggest highlights being a Switch port of Tropical Freeze, the DK extended cast in the Mario movie, and K Rool in Smash. It all makes a certain sense with the aforementioned focus though. DK himself has not gone anywhere, and the Kongs simply stand out as easier for potential movie spin-offs to Universal than the casts of the Yoshi or Wario subseries do. K Rool getting into Smash thanks to a ballot because fans viewed him as more worthy of Smash than many higher up in Nintendo seemed to really tells the whole story about where certain priorities with the IP really were/are.
Could this change? Possibly. The big push at Super Nintendo World has put characters beyond DK himself in the marketing and there is something prudent in actually making the franchise more of a living and breathing one beyond just featuring Donkey Kong specifically in spin-offs and calling it a day. Time will tell and it remains to be seen if the approach to the franchise will really change.