While there are aspects of unphilosophical probability that are interesting (most notably some of the ways we use general rules), the real interest here lies in philosophical probability. Philosophical probability for Hume consists essentially in imperfect causal reasoning; it is distinguished from causal proof, which occurs when we are dealing with something that happens in exactly the same way with perfect regularity. Obviously, there are many cases in which we don't have such ideal conditions to go on, and this is where philosophical probability comes in. In philosophical probability, either the resemblance or the regularity (constancy of the conjunction of events) are imperfect. This brings us to the Humean analysis of analogy (I'll set aside the aspect of Hume's analysis that depends on features of his dubious theory of belief, and focus on the meat of the analysis, which is what he brings in the theory of belief to explain), which are summed up in three pregnant sentences:
1. In those probabilities of chance and causes above-explain'd, 'tis the constancy of the union, which is diminish'd; and in the probability deriv'd from analogy, 'tis the resemblance only, which is affected.
2. Without some degree of resemblance, as well as union, 'tis impossible there can be any reasoning: but as this resemblance admits of many different degrees, the reasoning becomes proportionably more or less firm and certain.
3. An experiment loses of its force, when transferr'd to instances, which are not exactly resembling; tho' 'tis evident it may still retain as much as may be the foundation of probability, as long as there is any resemblance remaining.
These three sentences combined indicate some interesting things about analogy.
First: Contrary to common philosophical myth, analogy, as such, is not a weak form of inference. The problems that arise with analogy are not that it is a weak form of inference, but that it is impossible to tell from its form alone whether it is strong or weak in any given case. In some cases analogy is a very strong form of reasoning. In others, it is not strong at all. Which is the case depends not on its being analogical, but on independent factors.
Second: Analogy holds in all cases, as long as you are not trying to draw an analogy between the existence and non-existence of something. This sounds like a strong thesis, but in fact makes considerable sense: for any case a and any case b, so long as a and b aren't related as existence to nonexistence, there is some resemblance R such that aRb at some level of description. The difficulty with analogy on this front is not that some analogies don't hold; the difficulty is to what degree an analogy holds, and in what ways it doesn't. But we can legitimately think of any case along the model of any other case at some level.
Of course, we do say that some analogies don't hold, and are being perfectly reasonable when we do. We do something similar when we talk about 'evidence': we often say that "there is no evidence at all for position p" when, strictly speaking, we mean, "there is no evidence above the relevant level of significance for position p." That is, we don't usually mean to imply that there really is nothing that could reasonably be taken under some circumstance for evidence of p, which would be a very strong position that could apply to very few positions at all. What we usually mean is that anything that could be considered evidence for p is, for whatever reasons, insignificant enough that it can effectively be discounted. And likewise, when we say an analogy doesn't hold, we don't generally mean that the two cases don't resemble each other at all, but that their resemblance really isn't significant or useful for our purposes. When we combine this with the first point, we can conclude that whether an analogy holds in this looser sense, it is entirely relative to whatever our purpose of putting it forward it is, because that is what determines whether something is significant or useful.
Third: Because of these elements, it is essentially useless to try to refute an analogy as such. What one can do is exploit the same aspect of analogy that makes it essentially irrefutable: resemblance, which always applies and admits of many degrees. Since all analogies hold at some level of description, one needs to compare it with its rival analogies. Some analogies will turn out to be much better than others for the particular thing being considered. Further, even when an analogy turns out to be better than its rivals, we need independent information to determine how strong or how weak the conclusion of the inference actually is.
Fourth: It is not possible to underestimate the importance of analogy to the Humean project. In the above statement (#2), Hume says that "Without some degree of resemblance, as well as union, 'tis impossible there can be any reasoning." This has the effect of making analogy a minimal condition for intelligibility; i.e., we can only understand something if we can at least think of it along the lines of a resembling case. Hume explicitly appeals to this aspect of analogy elsewhere. In discussing abstract ideas in 1.1.7, he says:
The most proper method, in my opinion, of giving a satisfactory explication of this act of the mind, is by producing other instances, which are analogous to it, and other principles, which facilitate its operation. To explain the ultimate causes of our mental actions is impossible. 'Tis sufficient, if we can give any satisfactory account of them from experience and analogy.
And in the Appendix, when discussing his theory of belief, he says:
For if it be not analogous to any other sentiment, we must despair of explaining its causes, and must consider it as an original principle of the human mind. If it be analogous, we may hope to explain its causes from analogy, and trace it up to more general principles.
In these passages Hume is presenting an aspect of analogical inference that is often mentioned in the early modern period: it is by analogy that we move from what we know to what we don't know. Since in the Appendix he goes on to give, as perhaps his primary argument for his theory of belief, the argument that it, unlike its rivals, makes belief analogous to other mental acts (and therefore intelligible), we can see how far wrong people are who occasionally attribute to Hume the claim that analogy is a weak form of inference. It is an uncertain form of inference, in the ways noted above; but it plays a fundamental role in our knowledge-seeking inquiries. It is, in other words, a genuine case of philosophical probability. Such are the basics of Hume's analysis of analogy.
For a snapshot of the sort of work being done in cognitive science on analogical reasoning, see Chris's fascinating post on it at Mixing Memory.