The Psilocybin  mushrooms are not themselves toxic and overdose potential is very low. However,  the Amanita Muscaria (Fly Agaric)  is mildly poisonous, though the poisonous level is thought to be about 10 - 15 caps* whereas the hallucinogen effect is said to be obtained from 1 - 2 caps**. There are other Amanitas (A. Virosa and A. Phalloides) which look more innocent, but are in fact highly poisonous.  

The usual suggestions that people with any history of mental illness should avoid psychedelics & hallucinogens applies to magic mushrooms of any variety.

* We don't advise that you test this!
** or this!