Our sense or discernment of actions as morally good or evil, implies in it a sense or discernment of them as of good or ill desert. It may be difficult to explain this perception, so as to answer all the questions which may be asked concerning it : but every one speaks of such and such actions as deserving punishment ; and it is not, I suppose, pretended that they have absolutely no meaning at all to the expression. Now the meaning plainly is not, that we conceive it for the good of society, that the doer of such actions should be made to suffer. For if unhappily it were resolved, that a man, who, by some innocent action, was infected with the plague, should be left to perish, lest, by other people's coming near him, the infection should spread ; no one would say he deserved this treatment. Innocence and ill desert are inconsistent ideas. Ill desert always supposes guilt : and if one be no part of the other, yet they are evidently and naturally connected in our mind.
From A Dissertation on the Nature of Virtue. I like this argument quite a bit; I think it shows in a straightforward way that not all moral considerations are reducible to utilitarian considerations, even if utilitarian considerations are relevant.