Asbestos is not, per se, a carcinogenic material and unless the item that contains it is mishandled, I wouldn't worry very much about it.
I do some work with experts on toxic exposure cases and asbestos is in a number of things but it would be incorrect to assume that inhalation of a single asbestos fiber inevitably leads to a carcinogenic reaction. Everyone is different and it takes inhalation of more than one fiber to cause illness but if there is inhalation of more than one fiber it cannot be stated to a medical certainty that cancer is the inevitable result.
An example would be some floor tiles that contain asbestos but unless they are removed there is little to be concerned with. The same can also be said of some interior ceilings.
Unless the material is mishandled, such as running your hand across the surface to remove the material you should be fine.