The best justification of punishment is also not purely retributivist. The retributive justification of punishment is founded on two a priori norms (the guilty deserve to be punished, and no moral consideration relevant to punishment outweighs the offender’s criminal desert) and an epistemological claim (we know with reasonable certainty what punishment the guilty deserve) (Primoratz 1989, M. Moore 1987).
This is not, in fact, true of retributivists in general, though. The commitment generally shared by most people called retributivists is that punishment may only be imposed on the guilty. It is logically possible to have a retributivism committed to the claim that all the guilty should be punished, but no one seems actually ever to have held such a view. Even very hardcore retributivists usually concede something to other moral considerations -- and, indeed, not just moral considerations but practical considerations, as well, like how much trouble it would take to do the punishing, or the dangers of giving certain people the authority to punish in certain ways, or whether doing so would consistently require you to punish huge numbers of people. Retributivism in general doesn't normally tell you why you should punish this particular person but only why you can be punishing people at all; this does set up a default, but there might be any number of things that qualify this default, ranging from difficulty, to the feasibility of consistency, to the common customs of society, to general welfare.
Nor do any of these qualifications make the position less retributivist; if you consider utility, for instance, this doesn't mean that you have a utilitarian theory of punishment, since you may have a retributivist theory of punishment which recognizes that utility is a concern for the specifics of how society operates.
It is also a mistake to think of the matter in terms of "what punishment the guilty deserve". Punishments are not related to deserts in a one-to-one manner. Being deserving of punishment is something that we all recognize can be determined even when we don't know yet what punishment, specifically, should be given; while some penalties may be more appropriate than others, the question of what, precisely, the punishment should be is a different question from that of what justifies some kind of punishment in the first place. It is entirely possible to be a retributivist who thinks that there are many different ways you could legitimately go in punishing a particular kind of crime.
