Yes some seal conditioners do have ester content. But all liquids (fuel, oil, mixtures etc) can impact seal materials - it all depends on what the seal is made of and how the chemicals in the oil (not just esters) affect those seals.
In the API 1509 Chart G-6 it lists the required bench tests that all current engine oils (that meet API SN) must undergo in order pass the seal compatibility requirements. All registered oils will have met this standard which is agreed on by the OEM's so I think that as long as you use an oil as recommended by your OEM (likely ILSAC GF-5 or API SN) then you should be good. If you are going to use something that doesn't meet those specifications you may want to contact the oil company and see if they will supply you with the data on elastomer compatibility and compare it to limits on the G-6 Chart. (the chart provides a list of the ASTM tests that you would be asking for and the elastomer materials that the oil should be tested against).
Short answer - no you shouldn't avoid things just because they have esters, but be careful with oils that don't meet API specifications.