Which I distinguish from Hume's philosophy of the mathematics studied by mathematicians. Hume does, in fact, say some interesting things about what mathematicians study. I've briefly discussed some of them -- for instance, Hume insists that geometry is necessarily imprecise, approximate, and inexact, and that geometers are fooling themselves if they think they work with any sort of precision. But perhaps more interesting than this is what Hume says about mathematical practice.
To discover what Hume says on this subject, we have to look at the subject of curiosity, which Hume sometimes glosses, a bit misleadingly, as love of truth. Curiosity is actually an extremely important concept for Hume. As a skeptic, he tends to think that there are no (stable) speculative or theoretical justifications for particular fields of enquiry, and, what is more, he is quite frank and straightforward about the horrors of 'philosophical melancholy' that this can induce if nothing compensates for it. So Hume's justification for intellectual endeavors is always rooted in what he considers to be stable driving passions -- most generally curiosity and vanity. This is the justification for philosophical investigation Hume gives in Treatise 1.4.7, a justification even skepticism cannot defeat (one which he calls "the first source" of his inquiries in Treatise 2.3.10. I've dramatized it a bit in a doggerel mnemonic or 'dogmonic' called The Shipwreck of David Hume. It makes for quite an adventure; but, as you can see if you compare it to the relevant section of the Treatise, I didn't have to take much poetic license to make it one.
But my interest here is to discuss what Hume's justification of mathematics by curiosity commits him to saying about mathematical practice. The relevant section of the Treatise is 2.3.10. His primary interest in this section is to argue that truth is not desired as such; rather, it is desired only insofar as it is "endow'd with certain qualities". I won't go through the arguments, although I have also discussed some of them elsewhere. Rather, I want to sum up what I think these 'qualities' are that make mathematics an object for curiosity.
(1) An occasion for exercising ingenuity. Hume calls this the "first and most considerable circumstance to render truth agreeable." It is enjoyable to exercise one's mind in a way that reflects well on one's skill. The point here is not merely that mathematics is difficult; even difficulty doesn't always exercise our "genius and capacity," as Hume calls it. What curiosity requires is not merely a difficulty but an intriguing challenge, something that requires us to "fix our attention or exert our genius."
(2) An occasion of importance. Even the potential for stimulating thought is not really adequate. It might be sufficient for a mild recreation, but it just doesn't capture the passion of an intellectual endeavor. What is necessary is that the problem or puzzle challenging us is in some way important. Hume has an interesting idea about how to understand this importance; he appeals to his account of sympathy. Imagine, he says, an engineer surveying a properly fortified city. He gets satisfaction and pleasure from seeing how well done the fortifications are. This is not, however, because he gets any immediate use from them. Rather, he gets the satisfaction because, being human, he sympathizes with those who benefit from them (and presumably with the satisfaction of the engineers who built the fortifications in order to benefit them). So by sympathy we can have a very broad sense of the utility of something indeed -- even things that don't immediately benefit us or anyone we know can still give us a sense of this broader utility, if we can have some notion that it might be of real value to someone somewhere.
(3) An occasion that admits of success. The project, after all, must not be completely futile. But Hume is very concerned to reject the claim that avoiding futility requires finding truth; which is not at all surprising, since curiosity is the primary justification of his philosophical work in the face of skepticism about whether the truth can be found in many important areas of research. To argue this, he treats study and research as analogous to pursuits like gambling and hunting. In such recreations, we actually have a dual sort of success. There's the immediate success -- in the case of gambling, winning, and in the case of hunting, catching the prey. But a person's passion for gambling simply can't be explained in terms of his desire to win; this desire may spark the passion, but the passion is for gambling itself. And so with hunting. The pursuit itself becomes its own form of success. For this to be possible we need the precondition of importance -- someone who loves hunting partridges is not going to get the same enjoyment out of hunting magpies, because he's not going to be able to value magpies at the same level; the first are fit for the table (and note that by Hume's understanding of importance, this can be operative even if the hunter himself doesn't eat partridge), whereas magpies are largely useless for things like that.
Thus the object of curiosity, the prospect that really enlivens our interest and justifies mathematical endeavors, is, according to Hume: the discovery of possibly viable solutions to puzzles that are, from the perspective of the community, important, and that are, from the perspective of the individual mind, stimulating. To the extent that a field like mathematics provides this, Hume thinks, it justifies the pursuit of truth in that field. And, of course, as a matter of empirical fact we find that mathematics can provide it in spades.
Curiosity, then, is the governing motive of mathematics, the one that shapes it into a pursuit and a passion. It is not, of course, the only motive; nor is it in every particular case the strongest motive. There is, for instance, vanity, the desire to make a name for oneself. And we should not pretend that academics are above such sordid desires; anyone who has ever had dealings with academics knows that they are often almost obsessed with the possibility of being well respected, and that this motivation in at least many cases overtops even curiosity as a driving force in their work. Academia is filled to the brim with vanity; on a Humean view, this is one of the reasons it works in the first place. But vanity, unlike curiosity, doesn't shape the endeavor itself, considered as a community effort; the goal of the community of mathematicians is, at least in principle, determined by curiosity. What vanity adds to this is merely a powerful incentive for meeting the curiosity-proposed standards of that community; it cannot shape the community because it presupposes it. Curiosity, however, really does shape the community, especially when joined with sympathy.
So that, more or less, is Hume's account of mathematical practice. It's rather general, of course, since Hume was not a mathematician and does not seem to have had much notion of mathematics itself; it's really an account of intellectual inquiry in general, in which he continually appeals to mathematics as a paradigm example. But it's interesting in its own right, I think. I started thinking about this topic again on reading one of David Corfield's posts at "The n-Category Café"; some of what he says there is similar to what Hume is trying to get at, although at least part of what Hume is saying is shaped by his attempt to build an apraxia objection to certain forms of skepticism. One of the things Hume is missing that would greatly strengthen his account is the notion mentioned in that post of a 'mathematical story'; a problem with his analysis is that if it's taken as sufficient, we just get a string of interesting puzzles. But, of course, part of what really drives our curiosity in any field is the sense of taking part in what is (so to speak) an epic adventure of the mind. Even if the 'adventure' is only minor, it still needs a bit of sweep to it, a background against which the individual problems can be placed. Hume's sympathy account of intellectual importance is interesting and carries, I think, a serious grain of truth; but on its own it is clearly not adequate. Hume is right that importance is a major necessary condition for serious, sustained intellectual inquiry; he is right that sympathy plays a role; but an adequate account has to appeal to more than this. It needs a telos, a goal or end, and this telos needs to be that not merely of the individual problem but of the field as a whole. Corfield has argued (PDF) that this is a matter of tradition-constituted inquiry. Hume, of course, doesn't think in such terms. He goes far enough that he thinks in terms of community -- an immensely important achievement to which most people discussing mathematical, scientific, and philosophical practice don't attain -- but never develops a sense of the history or narrative of the community as a community that is (so to speak) united together in common cause, of the discipline not merely as a recreation but as an adventure of progress in a form of civilization. Something like that is needed to give an adequate account of any major field of intellectual inquiry.
