As per my Technical Specialist, it’s possible to damage the clutch or synchronizer if the transmission is downshifted at too high of a speed. He recommends not doing such a drastic downshift. Instead try 6th-4th or 6th-5th. The larger the change in RPMs, the greater the chance of damage to the driveline. I hope this helps.
Maybe if you're a klutz. Proper rev-matching, in theory, eliminates any undue shock to the driveline.
I know a guy who used to race RX-8's...he was getting on his co-driver's case for skipping shifts while downshifting for a corner (his footwork wasn't fast enough, I guess). Something about how the synchros in the glass tranny in that car didn't take well to this technique. But that is under racing conditions, a few times a lap over more than a hundred laps. I can't see how the occasional 6th to 3rd downshift to blast past some trucks would lead to lasting damage. Unless you're a klutz.