William D'Alessandro has a paper, Is It Bad to Prefer Attractive Partners? (PDF), in which he argues that it is often morally bad to prefer partners who are physically attractive. That the argument fails is not surprising (and it should, in fairness, be said that the article seems put forward in a more exploratory than a probative mode), but I think it shares a number of problems with many bad discussions of ethics, and in particular ethics of discrimination, so is worth considering.
The two key concepts in the argument are harm and unfairness. (D'Alessandro also considers issues of control, but makes clear that these are secondary.)
(1) Harm is understood in a broadly consequentialist way; of it, D'Alessandro says, "If a form of discrimination favors group X and disfavors group Y, I’ll count it as harmful if it causes the members of Y to do significantly worse than the members of X along intuitively important dimensions of human wellbeing (without any compensating benefit)." It is important for evaluating the argument to recognize that this way of understanding harm narrows the field of the kind of consequentialism that we can be assuming. For instance, note that this notion of harm does not require that group X be doing badly; it just requires that they be doing worse than another group. For instance, if you introduce a kind of device or practice that provides a considerable benefit (with respect to human wellbeing) to people who use it, but people who are right-handed benefit from it much more than people who are left-handed, then by this understanding of harm, we are harming left-handed people, not because we've actually hurt them, but because we have benefited them much less in a matter relevant to human wellbeing. I think it would be often controversial to call this 'bad', and I think most even of those who consider it bad would be uncomfortable classifying it as 'harm'; in this case, it seems to be a good example of unnecessarily treating the perfect as the enemy of the good.
(2) Unfairness is understood in a broadly deontological way; D'Alessandro says of it, "An unfair behavior, as I’ll use the term, is one that treats possession of some trait as a basis for awarding or denying a benefit, even though the extent to which someone possesses that trait is unrelated to the extent to which they deserve the benefit." This requires a very strong tying-together of deserts and benefits. Most people do not accept such a strong colligation of desert and benefit, because they hold that we have many different obligations, not all of which are desert-related. For instance, most people hold that parents have obligations to children that require them to make efforts to benefit their children because they are their children, regardless of their children's behavior -- any behavior by the child may shift what kind of benefits one focuses on, but the obligation is in place regardless of the behavior or desert. If this is the case, it weakens how closely desert and benefit must be tied; we have a moral obligation to benefit beyond desert, sometimes, based on other obligations. Nor is parenting the only case in which people tend to hold that they have benefit-granting obligations that are not tied to desert; marriage, fraternity and sorority, friendship, co-citizenship, shared religion, and even humanity are contexts in which people often recognize obligations to benefit that are not tied closely, and perhaps not tied at all, to desert, but to the requirements of the relationship or community involved.
D'Alessandro's argument needs harm + unfairness in these senses to be a sufficient condition for some kind of moral badness, but I think the above considerations suffice to establish that it is improbable that they are a sufficient condition, whether separately or together. The essential mistakes are two very common ones in ethical discussions: confusing comparatively worse with simply bad, and failing to consider how a given set of obligations (like those concerned with desert) interacts with other obligations.
There are other issues with D'Alessandro's argument. For instance, the argument has to assume, even to get off the ground, that attractiveness is a definitive source of benefit, rather than a source of benefits within a complicated set of causal mechanisms. "Unattractiveness is correlated with loneliness, social anxiety, self-consciousness, stress and life dissatisfaction"; yes, but unattractiveness of itself does not cause these things, but only is (at best) a contributing factor to these effects, dependent on other factors, and this means that we can ask the question, 'What if those other factors were removed, or if they were replaced by different factors? Would unattractiveness still count as a cause of these effects then?' It's unclear whether this would be so.
Indeed, we run into a peculiarity here. D'Alessandro argues as if attractiveness were the fundamental element in attraction, but it seems clear that this is not so; it's not like attractiveness makes people to be attracted. Rather, being physically attractive is statistically emergent; someone who is physically attractive is so because people in fact tend to be attracted to them on physical grounds. Thus all of the questions we might ask about preferring attractive people can be rephrased as questions about whether is is morally acceptable to act on one's attraction to someone. Whether they are what would usually be considered 'attractive' then drops out entirely; it just happens to be the case that as a population we have rough similarities in what we are attracted to, but the real ethical question is whether we morally can ever benefit anyone based on physical attraction. Almost everyone thinks that there are situations in which this is permissible -- for instance, few people would object to a husband giving his wife flowers because he thinks she's beautiful -- and in sexual matters it is hard to see how one could avoid it, since even setting morality aside it's very difficult to have much sex with people to whom one is not at all attracted, and this carries over to many activities that are not sexual but in some way sex-adjacent.
It of course does not follow from this that all forms -- or even most forms -- of acting on physical attraction are good or reasonable; there are other obligations that are relevant and important. I'm in fact inclined to think that these other obligations are far more important than any that might concern attraction itself -- whether or not one acts on physical attraction is a matter for the virtue of moderation, which unlike justice (which is what we are usually considering in considering harm and desert) does not usually have hard-and-fact obligations, but a certain kind of flexible and approximate appropriateness within a whole life. But it is another common failure in ethical discussion to assume that in moral matters there are no cases in which we are dealing with 'better and worse' rather than 'right and wrong'.
