Fatal Frame is niche? You realise that prior to awakening and their huge ad campaign the same was said about Fire Emblem right?
So let's look at what sets Fatal Frame apart from that game:
Obviously the genre, horror games are going through a sort of revival with stuff like Amnesia kicking off and the first triple A 'pure survival horror' coming soon in the form of The Evil Within. This game will follow the genre's re-emergence, and most importantly is made by a team who have still been working on horror games since 2001... there's a lot to sell on there!
Secondly, unlike Fire Emblem Awakening it's getting a huge amount of marketting in Japan! There's a Manga coming out, a theatrical movie, and this game all coming out with the Zero title; add in an upcoming Hollywood movie and you've got a massive marketing movement by the time the game comes our way.
Third: The controls, everyone's asking for a Camera game for Wii U, and there's one coming! That alone is enough to put in a Nintendo Direct and generate some more hype... and there's already quite a lot of hype for this game!
The marketting in Japan is insane, the timing and Gimmick for the Wii U is something everyone's been asking for, and the genre is re-emerging... the stars are aligned, this is likely going to be the break out hit for the series.
As for the selling to the limited western install-base, you have to consider how this will effect the install base. You'd have practically the entire Fatal Frame fandom buying the console, and with all the promotion going on that fandom is growing, and a large number of Survival Horror fans will tempted into the system, and that's not counting everyone interested in a camera game for Wii U; this is probably going to be quite the system seller with proper marketing.