From "A Defence of Detective Stories":
By dealing with the unsleeping sentinels who guard the outposts of society, it tends to remind us that we live in an armed camp, making war with a chaotic world, and that the criminals, the children of chaos, are nothing but the traitors within our gates. When the detective in a police romance stands alone, and somewhat fatuously fearless amid the knives and fists of a thieves’ kitchen, it does certainly serve to make us remember that it is the agent of social justice who is the original and poetic figure; while the burglars and footpads are merely placid old cosmic conservatives, happy in the immemorial respectability of apes and wolves. The romance of the police force is thus the whole romance of man. It is based on the fact that morality is the most dark and daring of conspiracies.
From "Errors about Detective Stories" we get probably his most interesting ideas about detective stories -- for instance, that they reverse dramatic conventions:
The two methods of concealment are exactly contrary, for the drama depends on what was called the Greek irony – that is, on the knowledge of the audience, and not ignorance of the audience. In the detective story it is the hero (or villain) who knows, and the outsider who is deceived. In the drama it is the outsider (or spectator) who knows, and the hero who is deceived. The one keeps a secret from the actors, and the other from the audience.
Or that only bad detective stories try to confuse the reader rather than focus on making things clear to them:
The true object of an intelligent detective story is not to baffle the reader, but to enlighten the reader; but to enlighten him in such a manner that each successive portion of the truth comes as a surprise. In this, as in much nobler types of mystery, the object of the true mystic is not merely to mystify, but to illuminate. The object is not darkness, but light; but light in the form of lightning.
He says something similar in "The Ideal Detective Story":
The detective story differs from every other story in this: that the reader is only happy if he feels a fool. At the end of more philosophic works he may wish to feel a philosopher. But the former view of himself may be more wholesome – and more correct. The sharp transition from ignorance may be good for humility. It is very largely a matter of the order in which things are mentioned, rather than of the nature of the things themselves. The essence of a mystery tale is that we are suddenly confronted with a truth which we have never suspected and yet can see to be true.
And he also builds on this theme in his "How to Write a Detective Story", since it is the first principle for which he argues, using "Silver Blaze" as the example -- the success of the story is that the death is caused by one whom nobody suspects but in retrospect was the only completely reasonable suspect. The second principle is that the explanation of the mystery should be more simple than the mystery itself. The third is that the guilty party should already be on the stage for a plausible reason that has nothing to do with the fact that you need a guilty party:
The art of narrative consists in convincing the reader for a time, not only that the character might have come on the premises with no intention to commit a felony, but that the author has put him there with some intention that is not felonious. For the detective story is only a game; and in that game the reader is not really wrestling with the criminal but with the author.
And the fourth, related, principle of detective fiction that Chesterton identifies is that the author should keep in mind that the reader is really trying to out-think the author, not the criminal:
The instinct of the reader, playing hide-and-seek with the writer, who is his real enemy, is always to say with suspicion, "Yes, I know a surveyor might climb a tree; I am quite aware that there are trees and that there are surveyors, but what are you doing with them? Why did you make this particular surveyor climb this particular tree in this particular tale, you cunning and evil-minded man?"
And the last principle is that writing a detective story starts with an idea rather than going in search of one:
Where the story turns upon detection, it is still necessary that the writer should begin from the inside, though the detective approaches from the outside. Every good problem of this type originates in a positive notion, which is in itself a simple notion; some fact of daily life that the writer can remember and the reader can forget. But anyhow, a tale has to be founded on a truth; and though opium may be added to it, it must not merely be an opium dream.