If the father of your child did drugs, will that take effect on your childs health and growth?

Ok well the father of my child was/is a pretty big druggie. He did pot, weed, ecstasy, and a lot of other things I probably don't know about. Well in result of all of that he's really skinny he's like 5'10 and he weighs 129 or something idk he's just skinny. I am 5'6 and I weigh 125 and people tell me I am too skinny for my height , but I do not think so. Anyways based on all of this is there a possibility the child will have health problems or trouble growing in height or will he/she be like a child that is born under weight and has problems gaining weight? It just hit me like 2 days ago that something like that could happen and I am really scared.