I actually completely agree with this. That's part of the reason I've slowed down so majorly with my game playing over the past year -- I've grown very attached to listening to my personal library of music, but I feel wrong to just override the standard music of the game, so I end up doing something else entirely so I can listen to my music without doing so. I know the music of TWEWY had a huge influence on my experience with the game, and the Ace Attorney series has great music that perfectly complements the gameplay -- those are just two examples.
However, I tend to compromise -- once I feel I've gotten enough of a feel for a game with its soundtrack, especially if I'm replaying it, I tend to allow myself to listen to other music.
The most significant example of music fitting a game well that I've found is with Mirror's Edge. Pretty much
any song by Imogen Heap/Frou Frou works great with that game. "
The Moment I Said It" is by far the best song I've encountered for the game, however. First night I heard it, I had it on repeat while playing the boat level and it was the most amazing combination.