Building material is not the proper material of architecture. The material of architecture, rather, is the building material already organized into elements, each one of which is a response to some prime construction problem. Such are, among others, the wall, the angle formed by the juncture of two walls, the openings made in the wall in the form of doors and windows, the roof or the vault, in short, those parts of an edifice required to define and isolate a closed portion of space.
Etienne Gilson, Forms and Substances in the Arts. Salvator Attanasio, tr. Charles Scribner's Sons (New York: 1966) p. 55.
In other words: The materials to which architects give form are not things like bricks, wood, etc. Rather, the materials of architecture are design solutions; these design solutions, considered abstractly, are organized into the plans for buildings, and, considered concretely, are organized into the buildings themselves.
Indeed, since I regard architecture as a form of engineering, I think this is true, if we consider design solutions for a broader range of things than just buildings, of engineering in general. I think the primary difference between an architect and what we usually call an engineer is one of expectation, not a difference fundamental to the field. Roughly, when we are considering buildings, what we call architects are engineering generalists, and usually the expectation is that they will have a broader training that includes topics in art and finance as well as applied science, whereas what we call engineers are architectural specialists, usually focused on practical scientific issues involved in putting together particular design solutions. But this is not a fundamental difference. The reason we should treat engineering as broader than 'architecture', of course, is that there are engineers who don't work with buildings at all. (This, incidentally, is a relatively recent terminological innovation. 'Architecture' literally means that art which organizes all other arts, and would once have been much closer to meaning what our term 'engineering' means than it does now. 'Engineering' on the other hand, suggests a focus on practical mechanical problems. But such is the shifting of language.) The key point, in any case, is that architecture and engineering are both architectonic -- they rule or govern the application of other arts. And that is another way of saying that the materials of architecture and engineering are design solutions or responses to construction problems.