These materials may pose a hazard to human health. The process of removal is called remediation and must be done by professionals.
Inorganic materials that are mined or quarried for use in the construction industry may sometimes be hazardous. Some of these materials have been used for thousands of years in various applications, such as insulation, glass, mortar, plaster and industrial applications.
Older, historical residential buildings may contain these materials in their structure and finishes. You may run into these materials in both hidden and obvious places. If you do see them, try not to move them - they may be relatively inert if they are undisturbed. If they must be removed, call for professional remediation.
Asbestos:
Asbestos roof tiles - photo from Construction Canada
Breathing in asbestos fibres may cause serious respiratory problems; it is a known carcinogen. It may be found in:
old building and pipe insulation,
fabricated materials such as gaskets and floor tiles,
wall panels, duct wrapping and wallboard joint compounds.
may be present in some plaster and textured stucco finishes
Asbestos fabric, for pipe insulation. Photo from Asbestos.com, The Mesothelioma Centre
Some old applications of vermiculite as insulation may contain asbestos.
Vermiculite is a mineral that was processed into insulation - it may be silver-gold to gray-brown in colour. On its own, it may not be hazardous.
In the past, vermiculite insulation was sold under the name Zonolite® Attic Insulation in Canada. The mineral for processing was taken from a mine in Libby, Montana, USA, where there was asbestos.