I believe the only reason to replace them is if they are visually damaged (poked holes), discolored (smoke or UV light), or no longer sound 'good'.
It's the third one that is tricky - with no reference, you don't know when they stop sounding 'good'.
If you are unhappy with their sound, and they 'used to' sound a lot better, and they are old, try vaccuuming them, try listening to another pair to make sure you aren't going crazy, and then try replacing the panels.
-Allen