But I go farther, and maintain, that none of these demonstrations can have sufficient weight to establish such a principle, as this of infinite divisibility; and that because with regard to such minute objects, they are not properly demonstrations, being built on ideas, which are not exact, and maxims, which are not precisely true. When geometry decides antyhing concerning the proportions of quantity, we ought not to look for the utmost precision and exactness. None of its proofs extend so far. It takes the dimensions and proportions of figures justly; but roughly, and with some liberty. Its errors are never considerable; nor wou'd it err at all, did it not aspire to such an absolute perfection. (1.2.4.17)
Despite the qualification in the last sentence, this is a strong position to take: that geometry is inexact, imprecise, and merely approximate in its conclusions is not a claim that is usually made. Part of Hume's argument for this interesting conclusion is an argument about the standard of equality in geometry.
If we were to think of geometric lines as composed of points, we could (in principle) simply identify geometric and arithmetic equality: Line A would be equal to Line B iff the number of points on Line A is equal to the number of points on Line B. Even setting aside the qualms we might have with treating points in this way, Hume notes that this would be "entirely useless"; no one actually identifies two lines as equal by counting their indivisible points.
Another argument, which Hume found in the mathematician Isaac Barrows, was to define geometric equality by appeal to congruity: Figure A is equal to Figure B iff, by placing the one on the other, every part in Figure A contacts every part in Figure B. Hume argues, however, that this is just an elaborate way of conflating arithmetic with geometric equality: ultimately, the congruity position reduces to the claim that for every point on Figure A there must be a corresponding point on Figure B.
Hume's own view is that "the only useful notion of equality...is deriv'd from the whole united appearance and the comparison of particular objects" (1.2.4.22). In effect, the only standard of equality in geometry is the one you use when you eyeball it.
In effect. When we look at the details, it ends up being more complicated. The general appearance can be put into doubt. When it is, "we frequently correct our first opinion by a review and reflection"; this correction may be corrected with another correction, and so forth. We use instruments of measurement that are of varying degrees of precision. At different times we exercise more or less care in the determination. Our idea of equality, therefore, is not exact. On the contrary: we form "a mix'd notion of equality deriv'd both from the looser and stricter methods of comparison" (1.2.4.24).
However, we don't stick with this. Having become accustomed to making these judgments and corrections, we get into the habit of doing so, and led on by a sort of mental momentum, we suppose an exact standard of equality. Knowing that there are bodies more minute than those that appear to the senses, we falsely suppose that there are things infinitely more minute than those that appear to the senses; and in light of that we recognize that we don't have any instrument or means of measurement that will secure us from error and uncertainty in such a context: the difference of a single mathematical point could be crucial. Because of this we suppose the corrections in our "mix'd notion" of equality to converge on the existence of a perfect but "plainly imaginary" standard of equality. What makes this "plainly imaginary," Hume thinks, is that our idea of equality is just the "mix'd notion," i.e., the appearance plus the corrections used by applying a common measure, juxtaposition, or instrument. The supposition that there is a standard of equality far beyond what we can actually measure is "a mere fiction of the mind." It's a natural fiction, since it is a result of this mental impulse or momentum whereby the mind keeps going even when it has ceased to be in touch with the facts. It is, however, a fiction.
Hume notes that this point, if true, is perfectly general: it applies not only to geometrical equality, but to equality in any sort of measurement: whether in time, or physics, or music, or art (e.g., hue). In all such cases we are led by the impulse of the mind to something far beyond the judgments of the senses. Our real notion of equality is "loose and uncertain": the exact standard of equality is more than we could possible know to be the case.
The problem this poses for the geometer is this. Either (a) equality in geometry is imprecise; or (b) it is precise. If (b), then geometrical equality is useless in practice (we can never know that a case exists) and depends on the controversial notion that lines are really and actually composed of infinitely divisible points, which Hume (and most of the geometers he would have known) thinks simply absurd. The standard they actually use, Hume thinks, is the imprecise one; but if we accept this view, many of the inferences made b geometers are ill-founded, since they assume a precision far beyond what the imagination and senses can yield. Thus, says, Hume, this shows that a geometrical demonstration of the infinite divisibility of a line is impossible.
One of the reasons I find this an interesting discussion is that in 1.4.2 he appeals to the same mechanism by which he explains how we come up with an exact standard of equality to explain how we come up with an idea of body continuing independently of our perceiving it. I presented a paper before the Hume Society a few years ago on this topic; my views have changed a bit, but I still think it's a key issue in understanding what Hume's theory of the external world really is.