While it is not the only passage that comes up, the big point of contention seems to be the footnote that Hume added -- for reasons unknown to us -- to the 1753 edition of the essay "Of National Characters". in its original form it went as follows:
I am apt to suspect the negroes, and in general all the other species of men (for there are four or five different kinds) to be naturally inferior to the whites. There never was a civilized nation of any other complexion than white, nor even any individual eminent either in action or speculation. No ingenious manufactures amongst them, no arts, no sciences. On the other hand, the most rude and barbarous of the whites, such as the ancient GERMANS, the present TARTARS, have still something eminent about them, in their valour, form of government, or some other particular. Such a uniform and constant difference could not happen, in so many countries and ages, if nature had not made an original distinction betwixt these breeds of men. Not to mention our colonies, there are NEGROE slaves dispersed all over EUROPE, of which none ever discovered any symptom of ingenuity; tho’ low people, without education, will start up amongst us, and distinguish themselves in every profession. In JAMAICA, indeed, they talk of one negroe as a man of parts and learning; but ’tis likely he is admired for very slender accomplishments, like a parrot, who speaks a few words plainly.
The Jamaican man in question was Francis Williams, who, sponsored by the Duke of Montagu, went off to Cambridge; as far as I am aware we don't have actual record of his graduation, but we don't have positive reason to think he didn't graduate. He returned to Jamaica and wrote some poetry. He may or may not have taught Latin; certainly there are stories he did, but Williams was actually fairly well off, being the son of a relatively successful merchant, and we don't have much direct evidence of his life. But Hume isn't the only person to refer to him in this context. The talk of 'species' is possibly significant; it may signal that Hume accepts the polygenist thesis that different races are in fact not related (as, for instance, Lord Kames had suggested at the time).
Over the next several editions there are some revisions, although they are pretty clearly in line with Hume's ordinary revision practice, since they are purely verbal. But the final revision by Hume, published posthumously, makes some notable changes in the first two sentences:
I am apt to suspect the negroes to be naturally inferior to the whites. There scarcely ever was a civilized nation of that complexion, nor even any individual eminent either in action or speculation. No ingenious manufactures amongst them, no arts, no sciences. On the other hand, the most rude and barbarous of the whites, such as the ancient GERMANS, the present TARTARS, have still something eminent about them, in their valour, form of government, or some other particular. Such a uniform and constant difference could not happen, in so many countries and ages, if nature had not made an original distinction between these breeds of men. Not to mention our colonies, there are NEGROE slaves dispersed all over EUROPE, of whom none ever discovered any symptoms of ingenuity; though low people, without education, will start up amongst us, and distinguish themselves in every profession. In JAMAICA, indeed, they talk of one negroe as a man of parts and learning; but it is likely he is admired for slender accomplishments, like a parrot, who speaks a few words plainly.
Notice that the claim has been confined to blacks rather than "all other species of men" beside the white; note also the "scarcely ever" rather than "never". So why the change?
We have no clue. The usual answer for a long time was that it was in response to the major criticism of Hume's argument in the original footnote, by James Beattie, who devotes pages and pages of his Essay on Truth to ripping it to utter shreds, in what is one of the most thorough attacks on the position that blacks are naturally inferior to whites that the eighteenth century has to offer. There are plenty of problems with this suggestion, however. Aaron Garrett, in a 2000 article on the subject, noted the two key issues: (1) We have no reason to think Hume ever read Beattie that closely, because Hume seems to have regarded Beattie as merely trying to attack each and every single thing Hume said, regardless of what it was; and (2) The revision makes little sense as a response to Beattie. For instance, Beattie has very good objections against the dispersed slaves argument, but that part of the footnote budges not in the least. Hume qualifies the original claim, but not in a way that would be effective against any of Beattie's major arguments (which essentially argue that the comparison is rigged from the beginning). While we can't rule it out, it seems unlikely that the change is a response to Beattie. And if it's not Beattie, we have no clear idea why the revision would have been made; perhaps Hume read something, perhaps he talked to someone, perhaps he just thought about it more. We seem to have no way of knowing.
So there are a lot of things we don't know about the footnote -- why it was added, why it was revised, why it was revised so slightly. It's always possible that some new evidence or insight will come up, but at present we're faced with this footnote appearing suddenly, for reasons we can't precisely determine, and lasting through revisions, the most important of which were made for reasons we do not know, to the very end.
So that raises the question of how this fits into Hume's philosophy, and it is a tangled one. One thing we can't do is dismiss it as a mere inadvertence; Hume has a passage in which he suggests very strongly that women are mentally inferior to men, but it's not a parallel case, occurring in an essay that was withdrawn almost immediately after publication, and thus not enduring revision after revision in the way the notorious footnote does. There's also no point in trying to argue that the qualifications, either in the original or in the revision, should somehow moderate our judgment of it; they are fairly slight, do little to change the argument, and do nothing to change the injustice of it. Putting much emphasis on little verbal qualifications is mere slipperiness unless we have a solid argument that the verbal qualifications make for significant differences. Cautious expression does not make racism less culpable.
Eric Morton in his article, Race and Racism in the Works of David Hume, argued that the case was "an example of how Hume’s theory of knowledge is driven by Hume’s racism and the built-in racism in his philosophical and conceptual worldview." And Harry M. Bracken famously used it as part of his case that empiricism, while not guaranteeing racism, faced severe difficulties in ruling it out. (The contrast case being rationalism, which, while not guaranteeing the absence of racism, has shown itself historically to be more resistant to many forms of it.) Part of the basic idea in approaches that link the racism to Hume's empiricism is that, while they can avoid it by being careful, it is easy for empiricists to connive on this subject. A fairly standard sort of defense used by racists and bigots of all kinds is to protest that they don't really have anything against the people; they just are sincerely trying to analyze the facts with an open mind that doesn't prejudge the issue, courageously and thoughtfully asking questions that others shirk and following the trail of evidence to its ultimate conclusion. It's very easy for an empiricist to accept such an argument as viable in principle, because prejudging the issue is a serious failing on virtually any empiricist viewpoint. And where one hasn't adequately analyzed things it can be easy enough to take inadequate empirical evidence as if it were adequate. Prejudicial analysis can often parade around as objective analysis -- indeed we find cases of this over and over again in history. And since empiricism can't make a distinction between the prejudicial and the objective except after the fact, it can easily mistake one for the other.And even if it doesn't actually make the mistake, one might argue, and some people have argued, that the conniving is still a problem: the empiricist approach shelters the bigot by putting an extraordinary burden of proof on people who find themselves faced with bigotry -- each and every point has to be analyzed as if it were in principle a purely empirical issue suitable for serious inquirers rather than something that only prejudice could lead one to regard as a serious matter of inquiry, and while perhaps not impossible it becomes difficult to make the Beattie-like point that inquiry, the questions asked, the evidence used, the very air of the discussion, is sometimes very clearly prejudiced in its own right. The racist attacks or insinuates and calls it empirical analysis; empirical analysis can't be dismissed out of hand because there are no overarching ideas with the function of sorting out what kinds of empirical analysis are genuinely rational. This can be a serious issue: In societies where racism is common history has shown it to be common enough for the racists to try to tar the anti-racists as themselves bigots who won't listen to reason and consider actual evidence. Thus empiricism is vulnerable, in a way that other positions may not be, to racist equivocation between prejudgment and conclusion; such equivocations, after all, are not advertised in neon lights and those relatively unprotected from them are more likely to fall victim to them either when others make them or in their own reasoning. There are other arguments that have been made on this score; they are not all mutually consistent, but they all agree that Hume's racism is actually closely related to his main philosophical positions, even if not logically entailed.
Such is one view. The other view is that Hume's footnote is actually an inconsistency, a failure to think things through in his own terms. Some have argued, for instance, that it is inconsistent with his skepticism; others that Hume should have focused on the importance of the common sharing (and consequent power) of sympathy; and so forth. It is difficult to find extended developments of these arguments, however; they are usually presented in passing. If anyone has come across a more developed account along these lines, let me know.
One thing we have to be careful about, post-abolition, is too easily conflating the issues of inferiority with slavery. Hume is on record as considering slavery cruel and corrupting (of the slavers); there are some ambiguities in his claim, but they are the sort of ambiguities one occasionally finds even in the staunchest abolitionists. It is possible to oppose slavery and be a racist, and likewise possible to oppose racism but accept some kinds of slavery; and thus pro-slavery bias is not really at issue in any of this.
I have a few more things, but this is the skeleton, and I'm not sure how much of the rest of what I have fits into a summary of the state of the problem; and even some of what I have here may be cut or reworked. Is there any significant issue missing here? Are there any major resources that need to be added?
I've discussed the issue of the infamous footnote before on Siris, here and here and here and a bit more abstractly here.