As a Tink main, I tend to use the returning boomerang a lot when playing keep-away as a means to win Neutral, start combos, and escape combo strings if my opponent manages to break through my zoning, and it's certainly helped me a lot in the past. I do however feel as if I'm not optimising this tool very well. Usually I'll attempt to Nair my opponent just before the Rang hits, and it tends to start combos or doesn't get punished very often on-shield, or go for a grab. At later percents though, the knockback causes the grab to whiff and I get punished, so I'll just Nair.
Are there better options though? I haven't been able to practice what I should use with this tool as much as I would like, since I only get to play on my friend's Wii U. Most of it's just improvised when I attend my local monthly. If anyone more experienced than I could give some insight on this, it'd be much appreciated.
(Yes I know I'm late in responding, but I've been off Smashboards for a while so I'm just seeing this)
Boomerang is my favourite projectile and I've been improving the technology on it for a long time. Here's some of the basic and advanced things I do with it.
Early on in a set, using it to help land attacks or grabs is the best use obviously. As your opponent adapts and recognizes the returning boomerang, they shield more often, and mixing up your timing of the grab (Way before it returns to cover a whiff, and while it collides with your opponent to lead in).
As the boomerang returns, it's not a bad choice to force your opponent into it either. Some common choices I like include bomb -> Returning Rang -> Up-Smash, or as you mentioned, Nair -> Returning Rang -> Up-Tilt or something else. Something I'm working on is Zair -> RR, since the long range catches people off guard as they get pushed into RR (Yes I just made RR a term XD).
One of my favourite things to do is, as the boomerang returns, I run to my opponent, and as the boomerang collides with my opponent, the boomerang comes back to me and cancels my dash, which makes Toon Link walk. I then use F-tilt, and if spaced right, it's easy to get the sweetspot, an excellent attack. If your opponents shields though, although F-tilt on shield is usually not punished, it's still kinda unsafe, so be aware. I suppose other options include Smash's and Jab, so experiment
Of course, one of the hardest things I find is reliably setting up into these things. Just like you, I kinda improv what I do with it, though I find that I usually throw boomerang up down at a slight angle while grounded, since it covers more surface area in front of your opponent, and returns at a nice low angle too. I also like SH -> Straight Boomerang Throw to stay low enough to the ground to make grounded setups while still having it aerial in a position to return.
With both of these setups, if people end up behind me, something that I've been working on is grabbing opponents and D-throwing as the projectile returns, which locks them as the projectile returns, dealing more total damage (Usually 9% from RR). On that note, at low %'s, it's sometimes possible to D-throw and knock them into the RR, changing the KB to lead into another attack like Up-Tilt. Me and some friends will be writing a guide on it at some point in the future so keep an eye out.
Anyway, hope that helps. I tend to write a lot but I hope you found this essay helpful XD