Well, this is unfortunate to hear.. The developers should have at least provided an alternative feasible "in-game" method of unlocking for this particular Challenge for individuals who don't have many people nearby to "StreetPass" with.. I mean, Sonic had like four different unlocking methods in Super Smash Bros. Brawl, so why couldn't the same be done here for a Challenge?
I'm one of the individuals who refrains from using Golden Hammers on any of the Challenges (even though I haven't completed all of the Challenges in Brawl - the "Collect all Stickers", "Boss Battles", and "Cruel Brawl" ones come to mind - I don't plan to use a Golden Hammer as an "easy win"), since I make it a point to actually complete the Challenges as best as I can. For this "StreetSmash with 5 people" challenge, if I end up being unsuccessful with it, I'll probably resort to a Golden Hammer only because this one is largely out of my control due to being reliant on other people's devices, no matter how much I play the game or how skilled I am. Ugh...this instance reminds me of that Spiritomb thing from the Sinnoh-based Pokemon games where you have to interact with a lot of people in the Underground..
While Nintendo 3DS StreetPassing may be a trend now, it's going to suck in ~10-20 years when the 3DS is largely rendered obsolete (with StreetPass potentially becoming relatively uncommon) and someone starts playing Super Smash Bros. for Nintendo 3DS has no choice but to use the Golden Hammer. I feel like it is similar to when you get a 'new' old boxed Pokemon game, and you see that flyer for that cool event Pokemon distribution that has already passed which you can never obtain.. :/ (let alone old Pokemon games generally being "uncompleteable" the more they age - at least in regards to completing the Pokedex)
But, in any case, I'm hoping StreetPassing becomes more common for Super Smash Bros. for Nintendo 3DS to alleviate the Original Poster's issue; it's bad enough having things like Mii outfits and custom moves being subject to randomness - I don't think it needs to be extended to Challenges as well...