Hi Rica-
I just wanted to offer you support. I also suffered from infertility though it was all me- DH had a normal sperm count. I also had late ovulation though and clomid with monitored IUI's was able to help with that. I conceived my son on the 4th try.
Regarding your husband- Did the doctor offer you any advice or insight on what to do next? A low sperm count can often be improved by Vitamins, lifestyle changes and sometimes even surgery. I am a long time member on a fertility website called: www.tcoyf.com I suggest you go there to the message boards. There is a whole section on male factor infertility and plenty of women going through what you and your husband are going through.
Take care.