This is probably the commonest fallacious argument of all in debates about emotion-laden issues – attacking the messenger.
Addressing the qualities or qualifications of "the man" might not be fallacious if "the man" clearly displays prejudices in his/her opinion. Pointing out the clear biases of a hate propagandist such as Dr. Paul Cameron (of FRI) is legimate when, for example, he distorts statistics in an attempt to disguise a hate message as being a fact-supported argument.
In the face of dubious statistics, suspicious "facts", or claims that an unlikely position is supported by empirical research, it is worth looking into the qualifications and possible biases of the individual making the claim.
The possibility always remains that the conclusion drawn by a highly biased debater may be the correct conclusion. However, arguments that display prejudices are automatically suspect.
Equally, the conclusions of an arguer who is not an expert in the area under discussion may be correct, but such an arguer would need to make the premises and logic of his/her argument quite clear in order to compensate for the possibility that his/her argument is not authoritative. Nevertheless, to question the messenger's expertise is not necessarily an ad hominem fallacy, though it is an ad hominem - a legitimate ad hominem.
On the other hand, to call the opposing debater 'an ignorant idiot' might feel justified in view of one's frustration with his or her obdurate denial of one's own version of reality, or more likely of expert knowledge. However, it is not a good argument against his or her premises, argument, or conclusion. He or she might be correct, or you might both be mistaken. However, such an assessment ought to be based on the merits of his or her, or your argument.
Fallacious ad hominems employ a variety of attacks: directly abusive, circumstantial, and accusations of "poisoning the well".