The harm principle comes from Mill's On Liberty, Chapter I:
The object of this Essay is to assert on every simple principle, as entitled to govern absolutely the dealings of society with the individual in the way of compulsion and control, whether the means used be physical force in the form of legal penalties, or the moral coercion of public opinion. The principle is, that the sole end for which mankind are warranted, individually or collectively, in interfering with the liberty of action of any of their number, is self-protection. That the only purpose for which power can be rightfully exercised over any member of a civilised community, against his will, is to prevent harm to others.
A common misinterpretation of this, no doubt due to the label that has come to be attached to it, seems to be that this is an authorization of harm -- that is, that it says that you can interfere with liberty of action when it harms another. This is upside-down -- despite the label 'harm principle', the principle is not about harm or response to it, but about protecting liberty. And Mill explicitly rules out any interpretation of the principle as telling you when you can intervene in Chapter V:
...it must by no means be supposed, because damage, or probability of damage, to the interests of others, can alone justify the interference of society, that therefore it always does justify such interference.
In other words, the harm principle does not tell us at all when the government intervene; it just claims that it is not even a legitimate option except when there is harm to others.
It's quite obvious, and no doubt was obvious to Mill (who was a close reader of Plato, who makes the point), that much depends on exactly what counts as harm. Mill, however, deliberately avoids going into any detail, arguably because for his purpose he wants to argue that this is one of the things that needs to be openly and freely discussed in a free society. One obvious question is whether verbal harms can count, and it seems to be a common view that the harm principle allows for this. However, even setting aside the fact that this is arguably against the very spirit of On Liberty, Mill seems to me to rule this out quite definitely as well.
The harm principle draws a distinction between public and personal interests -- society cannot interfere with what pertains merely to a person himself, but only where the person's actions become a real public concern in its own right. Mill guards against a simplistic understanding of this (although how well he does so is open to dispute); almost any private action has some kind of public consequence somewhere. However, if these public consequences are such as arise only through the "free, voluntary, and undeceived consent and participation" of the others involved, this is not what is meant. However one takes this, Mill is clear that the point is to protect liberty of thought and conscience from outside interference. And this, he insists, requires also the protection of the liberty of expressing and publishing opinions:
The liberty of expressing and publishing opinions may seem to fall under a different principle, since it belongs to that part of the conduct of an individual which concerns other people; but, being almost of as much importance as the liberty of thought itself, and resting in great part on the same reasons, is practically inseparable from it.
He will go later to say that liberty of thought and liberty of speaking and writing are inseparable, and he regards treating them as such to be part of what is required to have a free society in the first place.
Thus Mill's harm principle does not allow for the possibility of verbal harm, as such. There is still some wiggle-room -- Mill thinks you can in principle have public decency laws against pornography, for instance, because he allows for the possibility that one might have good arguments that it is harmful to society at large if not kept purely private, and arguably there are some kinds of verbal harassment that he would allow to be regulated because the way they are done makes it impossible for people to opt out (and thus their exposure to it is not free and voluntary). But in neither of these cases is there actually any notion that expressing and publishing an opinion, however atrocious or offensive, could possibly be regarded as harm on its own.
A third common misinterpretation, that the harm principle only applies to government, I have dealt with before; Mill is quite clear, as you can read in the passage above, that it also applies to interventions of public opinion (shaming and boycotting would be examples).
There are a lot of people who seem to like to talk the Millian talk, but make qualifications that are actually fatal to Mill's entire argument. You cannot be a classical liberal, in the sense in which Mill's On Liberty is definitive of the position, and also hold that speech, however atrocious, can be regulated on its own, even if only by nothing more than public shaming tactics. Trying to hold both requires reading the harm principle upside-down, as a restriction of liberty rather than (as intended by Mill) a basic protection of it.
