But as others of McGrath’s examples make clear, the problems with moral deference seem to persist in non-‐controversial cases as well: there’s something “odd”, as she says, about someone’s basing his belief in the wrongness of slavery purely on the testimony of a reliable authority (despite being in possession of all the relevant non-‐moral facts). In cases like this, however, we might well question whether someone who doesn't know on his own that slavery is wrong can be relied upon to identify a moral expert worth deferring to. As McGrath and Hills both point out, in the case of moral experts (unlike the case of medical or tax experts), it often takes one to know one.
I once tried an experiment in my Ethics course, in which I would relate the general approaches to actual historical cases where ethical reasoning played a significant role -- women's rights, slavery, civil rights, and so forth. As is so very often the case with experiments in teaching, it was something of a disaster. The students didn't have much of a sense of ethical reasons that historically contributed to progress in these areas, which is probably not surprising, but the questions were almost impossible to discuss. For instance, on the subject of slavery,
(1) Students agreed it was wrong and that it was better for a society not to have it.
(2) Their primary reason for saying it was wrong was that everybody thought so.
(3) Pressed on that, they said that in our culture it was taught as wrong.
(4) Pressed on why it was better for a culture not to have slavery, most of the reasons were either based on religious principles or on a principle of what goes around comes around ('karma' was usually the term they used, although they were very hazy about it and it had no connection to Indian thought for most of them).
(5) They were not very impressed by philosophical reasons for the wrongness of slavery unless they could be assimilated to one of the above reasons; they thought them complicated, silly things that were either just weird or attempting to say something that really boiled down to religious principles, or karma, or culture.
And so it was with other topics, with some variations. My point is that the situation being called 'odd' here seems in fact to be the normal situation of normal people in most situations: their moral views are generally views to which they have been accustomed by upbringing that are reinforced by other people apparently sharing the same views. Custom, tradition, and the agreement of others are the primary reasons for taking things to be right or wrong. Nor do I think that this is something from which anyone is wholly free; articulation of further reasons is difficult. I have seen professional ethicists break down into incoherence when trying to explain something they thought was ethically obvious to people who did not take it to be so; faced with the mind-boggling breakdown of the principle 'everybody thinks that' they couldn't come up with a way to argue for the position. (Although, it is true, part of the problem may be the tendency of ethicists to overestimate the extent to which people outside of philosophy have any idea what they are talking about, or any interest in it when they figure it out.) Think of how our moral vocabulary develops. Why do we treat a word as being morally charged? Because other people treat it as morally charged. We are deferring to people around us in taking 'hypocrite' or 'racist' to mark a bad thing. We might then come up with reasons why this is a reasonable evaluation. But we deferred first.
Nor does there seem anything wrong with this, as such. The chances are very good that any argument you make up on your own about why slavery is gravely wrong is more fragile and less reasonable as a basis for the belief than the testimony of Frederick Douglass that it is gravely wrong. If you ask ethicists why slavery is wrong, some are going to be 'everyone thinks that' ethicists; those that don't may come up with a number of arguments, but some of the reasons are arguably going to be more stupid than "Because Frederick Douglass says so." And as Phillip Hallie notes in "From Cruelty to Goodness", in order to get a good sense of how to describe cruelty accurately, you need to give some deference to the victims of it; they are the ones who actually know the harm. It doesn't follow that they are infallible; it does suggest that you need to avoid trying to base your moral views on nothing more than arguments in your head, and maybe ask how someone else sees things.
But the argument above suggests that this causes problems for identification of moral experts. This seems, I think, too quick. Suppose someone really and truly cannot figure out why slavery is morally wrong. It seems irrelevant that he cannot 'be relied upon to identify a moral expert'; he would, by the very assumptions of the argument, be better off deferring to practically anyone, moral expert or not. It's obviously nonsensical to say that you can only defer to experts; you don't have to assess your father-in-law as an expert to defer to his tax advice, you just need to recognize that he has more experience than you, or has more relevant familiarity than you, or, for that matter, just seems to know what he's talking about. Maybe you'll be burned by your father-in-law's con man ways, but that doesn't mean it wasn't a reasonable thing to do. And by the very assumptions of the argument quoted above, almost everyone should be able to figure out why slavery is wrong, so almost everyone would be a reasonable authority to defer to. This is going to be a common problem with arguments like this, on points that are not controversial: if it's so obvious that most people should be able to get it, it's not actually crucial to identify a moral expert, just as if you are completely helpless with your taxes in comparison with everyone else, you are already going to be better off getting any advice.
Likewise, there are many, many areas of taxation and medicine where ordinary people cannot certainly recognize the experts -- they have to rely on testimony from other people to find them at all. Sometimes people genuinely end up with quacks and charlatans. Perhaps there are more quacks and charlatans in moral matters than medical matters, but even if so, the posited asymmetry simply doesn't seem to exist, and could hardly exist given the apparent importance of testimony for moral education.