If it sounds like plugging a guitar in, it could be a loose wire.
Also when using the harness, it may still be using the amps, but may not be routing the signal the correct way. If you do a fade back to front and balance r to l, does it all line up the way it is supposed to?
If you didn't bypass the amps yourself with new wiring, then what that harness is doing, is sending a speaker level signal to amps that require line level signal. That will screw an amp up. Unless your harness accepts RCA lines from your head unit and routes them to the amps...
I dunno, like I said, mine was such a pita that I just rewired the whole thing.