Thought experiments in science are generally illuminating or, at the least, benign. It is not so with thought experiments in philosophy. They are a locus of misdirection and deception. We are supposed to derive important conclusions about fundamental matters from bizarre imaginings of zombies, who behave exactly like conscious humans, but are not conscious; or of substances that share exactly all the physical properties of water, but are not water. The narrative conventions of a thought experiment authorizes us to contemplate hokum that would otherwise never survive scrutiny.
John D. Norton, "The Worst Thought Experiment" (p. 11n). (Despite this footnote, the article is actually a criticism of Szilard's thought experiment in thermodynamics adapting Maxwell's Demon -- the worst thought experiment in science, according to Norton. As with all of Norton's philosophy of science work, the article is well worth reading.)
I think the 'important conclusions' is important here. Thought experiments in philosophy are benign, and sometimes illuminating, if they are of modest aim -- to illustrate a purely logical point, or to identify a conceptual distinction, or to exhibit a similarity between two fields of thought, or to summarize a more complicated response to a very specific point. More than this they cannot really bear. But, of course, they are made to bear more than this all the time -- when people use zombie experiments, they aren't merely sharpening the conceputal distinction between conscious experience and behavior indicative of it but treating the stories they are telling as establishing truths about consciousness itself. When you come across a philosophical thought experiment with major conclusions, always ask: What is the rational account that authorizes the inferences required by this story or description? And I think this is where they so often go wrong -- they are taken as establishing things when, at best, they can usually be doing no more than gesturing at a more sophisticated account. And, of course, very often there is no sophisticated account at all, in which case they can at best suggest a line or two of further inquiry in the same way that any metaphor or analogy might.
This is remarkably difficult to get across to some people. Just as even ordinary counterexamples must be analyzed to determine (1) that they are not merely apparent; and (2) that they are not more limited in scope than they might seem, the interior logic of a thought experiment also requires analysis, and each assumption made in building it requires examination -- it is a scaffold for further inquiry, not a primary result in its own right. But people have a weird tendency to throw out alleged counterexamples and move on, or to take thought experiments to establish things on their own. When people do this, I often just start asking the ordinary questions any sort of reader or writer of stories might reasonably ask about the logic of a tale; over and over again, one finds that the proposers of these thought experiments haven't even thought through the basic story-logic of their example, much less started working out any rigorous argument. It's a case of scaffoldings being confused with cathedrals, and means being confused with ends.
