Frege says somewhere that, if a single contradiction were to be discovered in mathematics, "the whole building would collapse like a House of Cards". Please, why? This claim seems largely an artefact of the wrong metaphor. Mathematics is not a house with foundations which have to bear the whole weight. It is rather a planetary system of different theories entering into various relationships, and happily spinning together in logical space. Damage one, and the system will continue, maybe with some debris orbiting here and there. Or, here is another metaphor for mathematics, equally attractive, due to Chaim Perelman: it is a wonderful tapestry of many strands woven together by the great mathematicians. Pull out one strand, and the tapestry may be weaker by an epsilon, but tears can mended. And this brings me to my most central objection: we know from the history of mathematics and the sciences that contradictions are never the end of a story.
Johan van Benthem in Philosophy of Mathematics: 5 Questions, Hendricks & Leitgeb, eds., Automatic Press (2008) p. 38
