William James's The Will to Believe is in great measure a response to William K. Clifford's The Ethics of Belief. The latter is best remembered for its claim that "it is wrong always, everywhere, and for anyone, to believe anything upon insufficient evidence." What is often forgotten is that Clifford also took the trouble to specify what, precisely, he meant by sufficient evidence. The full argument of the essay could be summarized like this:
(1) We may believe what we can directly and sensibly observe.
(2) We may believe what we can rigorously and strictly demonstrate from principles we can believe.
(3) We may believe what goes beyond our experience, only when we can conclude it from that experience using the assumption that nature is (to some extent) uniform.
(4) We may believe another person, when there are grounds for supposing that he is sincere and there are grounds for thinking his belief falls into one of the above three classes.
(5) If a candidate for belief does not fall into one of the above four classes, it is always morally wrong to believe it because it can have bad consequences.
(6) It is always morally wrong to believe without prior inquiry because it can have bad consequences.
There are some notable features of this argument. It is empiricist, since everything believed ultimately has to be parsed into sensible experience, logical principles, and the assumption of the uniformity of nature. It's also unclear exactly why the assumption of the uniformity of nature gets a pass; Clifford explicitly sets aside the question of its actual nature, but it's essential to his view that we have it, because, as he notes, going beyond experience is part of the very nature of belief. The ethics part of Clifford's ethics of belief is purely utilitarian. Why is it wrong to believe in certain cases? Because the beliefs lead to actions that are morally wrong, or to bad consequences for society: every belief that is believed on what he counts as insufficient evidence tends to the harm of society, even if it doesn't on its own actually issue into any harm, so every such belief is wrong (sinful, as he sometimes calls it). It's because of this that much of Clifford's rhetoric is so melodramatic: every belief is a matter of the welfare of society, and so we have an obligation to regulate every belief as part of our duties to all mankind.
Practically everything in James's arguments is best understood as a response to features of Clifford's full argument. "The Will to Believe" is part of a larger Jamesian project, however, of naturalizing rationality. That is, James wants to give a psychological account of what it is to be rational. This is one of the root differences between Clifford and James: James wants to insist that an account of what makes belief rational must begin by looking at actual facts about how the mind works, and in this context his primary contention is that in the human mind passions and reasoning are integrated, not separated. Inquiry is governed not by pure reason but also by the interests established by our nature; a passion for truth, for instance, or a strong feeling that this or that is important. The only mentions of passions in Clifford's essay are all negative: we should reason without regard for the passions. But, James will insist, given facts about human psychology what this actually means is just the more widely accepted truth that we should reason as guided by some passions and not others. And that means that there is an intrinsic complexity in matters of belief, any belief, that Clifford ignores entirely.
James early on makes his famous distinctions among different kinds of options, by which he means a case in which we are presented with alternative candidates for belief: these options may be living or dead (a living option is one where the candidates seem like real possibilities to us), forced or avoidable (a forced option occurring where we must believe one or the other), and momentous and trivial (a momentous option occurring where the opportunity is unique, the stake momentous, or the decision irreversible). What we even bother to consider depends on how living, forced, or momentous the option is. But more than this, some options require the clear involvement of a panoply of passions. In some cases, for instance, when the option is trivial and avoidable, it makes sense to act in accordance with our passional interest in avoiding error; but even here, as in scientific discovery, one wants a balance that optimizes one's ability to pick out salient facts: "The most useful investigator, because the most sensitive observer, is always he whose eager interest in one side of the question is balanced by an equally keen nervousness lest he become deceived."
In cases of living, forced, momentous options, however, we need a different balance: hope of finding genuine truth takes priority over fear of believing error, so that we may reasonably be willing to risk being partly wrong in order to be at least partly right. And this, too, will be governed by our passional nature, our willingness to risk one given what seems to us (since we are dealing with a living option) a real opportunity for the other. Inquiry, in other words, is a practical matter, and in every practical matter we are trying to find means that yield the results that interest us, which on a Jamesian view are picked out by our passions. And avoiding error is not the only interest we can have in rational inquiry; an ethics of belief that fails to recognize this, as Clifford's does, is an irrational rule.
Such is the general shape of James's argument. His work is made easier by the fact that Clifford's argument is itself not fully coherent; for instance, he never at any point establishes that the basic principles of his account of belief are supported by sufficient evidence, and in several cases (the assumption of the uniformity of nature being one) it is difficult to see how they could -- there is nothing we can directly observe that could establish them directly or indirectly as acceptable for belief according to the account itself. James, on the other hand, can easily accommodate principles like the assumption of the uniformity of nature: we have a passional interest in practical life that requires the assumption that things don't change too much. And James can accommodate other principles of inquiry more easily than Clifford came; we have a passional interest in having simple theories and this makes it reasonable to choose, among a variety of alternatives, the simplest theory that fits the facts. We have, in his words, an "imperious inner demand on our part for ideal logical and mathematical harmonies." Clifford's ethics of belief makes it generally impossible to choose between theories that both fit the facts: we could never take the advice to believe the simplest theory that fits the evidence, but would have to suspend judgment. The list could be continued.
But the key point here is really that Clifford's empiricist criteria for what counts as sufficient evidence forces the dispute onto Jamesian turf. Since we can only believe what has been directly observed by someone or drawn out of that direct observation by strict logic or reasoning with the assumption of the uniformity of nature, the test of whether Clifford's account of rational belief can be accepted must be what is directly observed. And since we are talking about belief, that means the test of whether Clifford's account of rational belief can be accepted depends entirely on whether it fits with psychological facts about believing, and James, of course, is the psychologist of the two. More than that, James is proposing a psychological account of rationality of belief as a rival to Clifford's ethical account. This tips the field massively in James's favor from the very beginning.
In the post linked above, ArithmoQuine makes a number of objections to James's arguments that dissipate completely when put in this context. For instance, AQ says,
The irritant in reading James is that he cannot refrain from irrelevant ad hominem attacks on Clifford, James calls Clifford “nervous” and cites his “fear” of making errors and contrasts this fear with his own hopefulness. But the issue is not Clifford’s emotional state; the issue is whether it is responsible for one to believe something when there is insufficient evidence to think it’s true.
But this in itself shows a complete failure to understand both James's argument and the point actually at issue in the dispute between James and Clifford. James is not making "ad hominem attacks"; he is insisting on his point that all belief, even rational belief, is guided in part by what he calls our passional nature. On James's view it is sometimes good to be nervous about error and good to fear being wrong; these are simply psychological facts that set our interests in inquiry, and obviously they are sometimes reasonable responses. What he denies is that these are the only passions that set interests in inquiry. And James's whole point is that it is impossible to determine whether you are believing something on insufficient evidence without regard to how the passions are setting our goals for inquiry, and whether they are balanced and healthy in that context. If 'insufficient' means just 'insufficient for responsible belief' then it is trivially true that it is irresponsible to believe on insufficient evidence. But if it does not mean this, it is making a substantial claim about responsibility, as Clifford is making a substantial claim about responsibility, and this must be closely examined, not glossed over with ambiguities in the meaning of the word 'insufficient'. Throughout James's lecture he is clearly taking 'insufficient evidence' in Clifford's very restrictive sense; and he is claiming that even if the evidence is insufficient in Clifford's sense, it may still be sufficient for responsible belief. And his argument is one based largely on the importance of "emotional states" to guiding inquiry.
All of AQ's argument about sufficiency dissolves as well; it is very clear throughout that AQ has an account of sufficiency of evidence that is inconsistent with Clifford's, despite the fact that AQ never actually bothers to give in plain terms what that account would be. Clifford's account, for example, makes no exceptions for normativity. Likewise, it is not enough to hold based on evidence that people are friendly: this must be universally observed, or logically required by what we do observe, or logically required by what we do observe if we add the assumption that nature is more or less uniform. That is, the evidence must rigorously prove (allowing for only the little bit of slip in the uniformity assumption) that the person in question is friendly. Clifford's account, unlike James's, leaves no room for mere probabilities (except by way of the uniformity assumption).
AQ's moon example, by the way, is an odd argument against James. It's obviously not Jamesian at all, but AQ seems to forget this and argue as if it were, and makes criticisms that clearly depend entirely on features specific to that sort of example. A better example would be a scientist tentatively accepting a theory because it is the simpler of two theories that fit some important facts, or accepting a theory because, although it explains some facts less well than another theory, nonetheless explains facts that the scientist thinks are probably more important. The scientists in these scenarios are being irresponsible and immoral on Clifford's account and perfectly rational on James's. When physicists say that the beauty of a theory is a factor in their scientific judgment, they are being sinful on Clifford's account. Another example would be a non-scientist accepting a scientist's judgment on an extremely difficult matter, where that judgment might not be purely a matter of what scientists have directly observed or logically deduced from what they observed using at most the additional assumption of the uniformity of nature. On Clifford's view this is sinful, for even if the scientist may reasonably be judged sincere, the non-scientist is in no position to be able to believe that the scientists' inquiry actually conforms to Clifford's strictures, since even scientific testimony can only be accepted if its truth is required, directly or indirectly, by what the non-scientist has directly observed. Thus there is really nothing in AQ's arguments that actually does anything against James; AQ's arguments ignore context completely and thus attack a straw man.
If we want to abstract from context, then we have to recognize that the real issue between James and Clifford is not so much religious belief but the legitimacy of accepting hypotheses that have merely been confirmed and not rigorously established by empirical proof. Religion comes up as a key issue because Clifford treats certain very general religious doctrines as being, at most, that, while James treats the same religious doctrines as being, at least, that, and thus it is a good context for discussion. It is also a subject in which both have a certain amount of 'passional' interest as James might say. But the actual crux of the dispute is not religious at all, but James's claim, based on his insistence on a naturalized account of rationality, that whether or not accepting a hypothesis is rational depends entirely on its context and, moreover, a context that includes rather than ignores our 'passional tendencies'.
