It isn't harmful but it's so far less than necessary unless you've got more than just a few bolt-ons in your car. If you're talking about competitive driving.....well doing it with an auto is probably a bad idea to begin with, but just accelerating the computer can do it better than you ever could by hand, you also run the risk of redlining.
Personally if I'm not launching the car there's a bit of a tip in problem with the TPS if the car's in drive. So if I want to leave a stop light with any sort of grace and civility I have to shift myself. And if you've GOT to go from a roll then you can shift down manually to get a better start, but from a stop "D" is the way to go.