To me it's personal preference. I would rather hear the music "surround sound" than just from up front and that's why I changed out all four of mine and what a huge difference it made. Actually, it's not that big of a pain to swap out the rear speakers, it's just time consuming.
And I might change out the rears soon. I've done a couple of trucks and after watching a youtube video it appears its about the same type of job. Removing top seat belt bolts, rear panels, etc. Like you said just time consuming. I've also swapped out an entire interior on a fox body before so its gotta be easier than that!