Classifications are regulated by the Idea of Likeness; but to get a natural classification you have to go beyond resemblance to something else. This something else is what Whewell calls "natural affinity". As Whewell says (vol. 1, p. 488):
The assumption that there is a Natural System, an assumption made by all philosophical botanists, implies a belief in the existence of Natural Affinity, and is carried into effect by means of principles which are involved in that Idea.
This is not immediately obvious, though, and Whewell, as one might expect of the father of the history of science, is quite aware of the fact that people have attemped natural classification without a clear notion of natural affinity. Following Decandolle, Whewell distinguishes such attempts into three types: blind trial, general comparison, and subordination of characters. He summarizes them very conveniently in an aphorism (vol. 2, p. 462):
XCV. The attempts at Natural Classification are of three sorts; according as they are made by the process of blind trial, of general comparison, or of subordination of characters. The process of Blind Trial professes to make its classes by attention to all the characters, but without proceeding methodically. The process of General Comparison professes to enumerate all the characters, and forms its classes by the majority. Neither of tehse methods can really be carried into effect. The method of Subordination of Characters considers some characters as more important than others; and this method gives more consistent results than the others. This method, however, does not depend upon the Idea of Likeness only, but introduces the Idea of Organization or Function. (VIII. 2.)
The problem that arises with finding natural classification is this. We have some notion of natural classes that organize resembling kinds of things and of affinity among the classes; but this notion is very vague. In fact, it is nothing more than an "obscure feeling of a resemblance on the wohle, an affinity of an indefinite kind" (vol. 1, p. 500). Linnaeus is a good example of this. Linnaeus denied that there was any a priori rule for natural classification, and held that the only way to classify into a natural system was to take into account the symmetries of all the parts of the plant. Thus Linnaeus proposes various classes as natural, but can give no reason to ground this classification as natural; indeed, he even declares it impossible to identify a system of characters that could do so. Whewell diagnoses this situation as a failure to look beyond resemblance for the purposes of classification (vol. 1, p. 500):
This persuasion was the result of his having refused to admit into his mind any Idea more profound than that notion of Resemblance of which he had made so much and such successful use; he would not attempt to unravel the Ideas of Symmetry and of Function on which the clear establishment of natural relations must depend.
Needless to say, classification according to a general feeling of resemblance gives us something whose justification is very vague, so it was inevitable that people would try to find a more systematic way of doing it. This way was hit upon by Adanson, who, trying to classify plants in Senegal, found himself stymied again and again in his attempt to classify newly identified vegetation. Because of this, he started examining all the parts of the plants, building up as complete a description of each as possible, noting all the similarities and differences with other plants. By this aggregation of comparative descriptions, he found that the plants more or less arranged themselves into classes. To put it another way, we construct many different artifical systems, each based on some part, and then classify together those plants that resemble each other in the greatest number of artificial systems.
It's an ingenious approach, but it suffers right from a fatal flaw, right from the beginning. The amount of work you have to do even to begin to get it off the ground is utterly immense. Adanson, for instance, created sixty-five artificial systems; the sixty-fifth, to take one example, consists of ten classes that are further subdivided into ninety-three sections. Of these, only thirty-five are retained in the end result. And, at the end of the day, you can never be sure you've considered all the relevant aspects of the plant. Even more importantly, the principle of the method of general comparison if flawed, because it assumes that all possible artificial systems are of equal importance, which is obviously false. Without at least a dim feeling for natural affinity, this method is simply not practicable, and cannot reach the natural system.
For general comparison to become practicable it must introduce some ideas other than resemblance alone, and, in particular, it must identify some resemblances as more important than others. That is, general comparison must be transmogrified into subordination of characters. Subordination of characters, however, leaves us with a serious puzzle (vol. 1, p. 536):
It is easy to see that some organs are more essential than others to the existence of an organized being; the organs of nutrition, for example, more essential than those of locomotion. But at the same time it is clear that any arbitrary assumption of a certain scale of relative values of different kinds of characters will lead only to an Artificial System....It is clear that this relation of importance of organs and functions must be collected by the study of the organized beings; and cannot be determined a priori, without depriving us of all right to expect a general accordance between our system and the arrangment of nature. We see, therefore, that our notion of Natural Affinity involves in it this consequence;--that it is not to be made out by an arbitrary subordination of characters.
Whewell asks us to consider the integration of living things. Each organism is a system in which functions play an essential part; they facilitate each other, and it is often difficult to pry them apart without being arbitrary. This is often viewed in light of adaptation or conditions of existence; but Whewell argues that it is also an illustration of affinity. His argument is rather important, so I might be forgiven for quoting it in full (vol. 1, p. 537):
It has sometimes been asserted that if we were to classify any of the departments of organized nature by means of one function, and then by means of another, the two classifications, if each strictly consistent with itself, would be consistent with each other. Such an assertion is perhaps more than we are entitled to make with confidence; but it shows very well what is meant by Affinity. The disposition to believe such a general identity of all partial natural classifications, shows how readily we fix upon the notion of Affinity, as a general result of the causes which determine the forms of living things. When these causes or principles, of whatever nature they are conceived to be, vary so as to modify one part of the organization of being, they also modify another: and thus the groups which exhibit this variation of the fundamental principles of form, are the same, whether the manifestation of the change be sought in one part or in another of the organized structure. The groups thus formed are related by Affinity; and in proportion as we find the evidence of more functions and more organs to the propriety of our groups, we are more and more satisfied that they are Natural Classes. It appears, then, that our Idea of Affinity involves the conviction of the coincidence of natural arrangments formed on different functions; and this, rather than the principle of the subordination of some characters to others, is the true ground of the natural method of Classification.
Or, as he summarizes it in an aphorism (vol. 2, p. 463):
C. The basis of all Natural Systems of Classification is the Idea of Natural Affinity. The Principle which this Idea involves is this:--Natural arrangments, obtained from different sets of characters, must coincide with each other. (VIII. 4.)
In other words, you find natural classes not by arbitrarily assuming that one characteristic is more important than another, but by showing that there is a similarity of gradation among several different characteristics. Correspondence of classification among functions is indicative of natural affinity. Thus Whewell formulates a maxim for testing whether a system is natural or not; and the test is this: the arrangement obtained from one set of characters coincides with the arrangement obtained from another set. By seeking out this coincidence of arrangement, he thinks, we can find the natural connections among various classes in our classification, whether we are organizing organic or inorganic things.