Next time you need to replace your belt and have it off, take your hand and spin the AC pulley. Watch it keep on spinning. This is all the engine is driving when the AC is turned off. You will certainly gain some underhood room and shed some weight by removing the AC system, short of that I wouldn't worry about it.