Will ingested semen get into milk?

My husband and I have recently become intimate again after the birth of our daughter.The problem is that I am feeding and worried that if I ingest his semen it will somehow get into my milk.I have already done this once and am now paranoid to nurse my daughter?Does anybody think its ok...I have no idea what the general rule is.