falsifiability

falsifiability

navigational aids:

News ticker:

Giant space monster sprouts wings, fangs
Woman gives birth to MP3s
Yahoo! asked to explain alleged perjury
Giant space monster continues its approach
Yahoos in suits choose cash flow over human rights
Junior high textbook publishers act like junior high students
Giant space monster is watching us
Murdoch conspiracy uncovered
Web site induces massive loss in worker productivity
Porn averaged
Giant space monster threat continues

topics covered:

30 October 05.

One of the oft-given responses to the question 'what is the key to scientific endeavor?' is falsifiability. It is what distinguishes science from religion, which makes claims which are clearly designed to be impossible to falsify (He meant it to be that way, regardless of how it is).

It is also what distinguishes science from mathematics. One can make false statements in both the world of math and the world of chemistry or physics or what-have-you, but one can verify the validity of a mathematical statement entirely based on the system you have written down. Through the appropriate manipulation of symbols, you can determine the statement's validity [Gödel aside], and once you've determined internal validity, you're done.

Compare with a statement to be falsified experimentally. Talk to any researcher (save for the few divinely blessed by the above-mentioned Creator) and they'll tell you about how often they wrote down a model, verified that it was consistent and true and that the numbers checked out, and then they built the darn project and it did nothing. It was falsified by something outside of the mathematical system as written down. The room was too warm, or the variance on the laser's power was too great, or there were odd impurities in the solution, or any of a hundred other things which you don't write down in a model until you try it out and find out that it's really important.

Figure One: False and falsifiable are not related.

Figure One makes one more attempt to make the point painfully clear: step one is writing down a mathematical model and checking whether the statement is false. This is a symbol-shunting exercise, where we check that we didn't add two and two and get five anywhere. Now we are somewhere inside the white circle, and can move on to step two: verifying whether our claim works experimentally. This is an entirely independent process from the symbol-shunting.

So there are loads of people who will tell you that if you're not doing something falsifiable, you're not doing science. This is not to disparage either mathematics or religion, but they ain't science. Some do indeed use the distinction disparagingly: if you're just writing down equations which are internally consistent, you're a theorist, which is kind of like a scientist but without any credibility.

Economists

can not do experiments. There are the ethical problems of how quickly one could screw up people or an economy, and there are the practical problems that getting everybody to all comply with your dumb idea may require billions of dollars. Therefore, all we have to falsify our claims is econometric techniques on existing data, and this is why it is of the utmost importance that economists remain firmly honest and clear and theoretically correct regarding all statistical techniques. If all falsifiability relies on your regressions, your regression techniques had better be sufficiently disciplined to falsify things. But this is a digression.

Computer scientists

, conversely, have it relatively easy. Somewhere up there, I described the two steps as internal verification and external verification, and when designing a computing system, there is no external world. We write down a system where i=i+1 always increments i by one, no matter what the weather or the state of k or the day of the week. That is, there is no external verification step. If you did the internal verification correctly and it doesn't work, it's the engineer's fault; buy new memory and run it again.

Of course, external data sneaks in some times. There are two means of falsifying the algorithm in regards to external data. One is the problem of dealing with formats and user data and boilerplate boringness like that. The other is based on claims of data transformation of the form 'this algorithm will take data of the given input form (ignoring the boilerplate boringness) and will then produce output which matches another form. I.e., another mathematical theorem, which just happens to be based on data from outside the system. This gives us a level of semi-internal falsification: all external verification depends on the external data stream, and if we define the data stream as part of the system then we're back to 100% internal verification.

IP implications

Patent law also distinguishes between mathematics and chemistry. A mathematical equation can not be patented; a chemical formula can. Many people who have written on patent law can't tell the difference between the two: well, they reason, you could write down a chemical formula just like you can write down a mathematical formula. It's a bunch of symbols, all identical, and therefore should all be identical under the law. When people make such arguments, I can't tell whether they're being disingenuous or sincerely don't get it.

But the falsifiability criterion shows a clear difference. Once the mathematical formula is verified to be true, in symbol-shunting terms, you're done; the true & verified circle has no falsified cut-out in math-land. In chemistry, the falsified cutout covers nearly the entirety of the white circle, leaving a small sliver for results which are both symbolically true and empirically verified. One can write down a thousand chemical equations such that the number of hydrogen, oxygen, and carbon atoms on the left equal the number of such atoms on the right, but the conversion from one to the other happens here in the real world for only a few of them. Helping people find which are the ones which will work is a key (dare I say the key) goal of the patent.

Now, some people say that if you write down a complete enough system, then it's all internal verification. The weather affects your chemical reaction? Then include it in your equations. Can't do the reaction in space? Include gravity. Et cetera.

Two ways of expressing the problem. First, the calculation may soon become more difficult than the plain old experiment, in which case you won't bother with internal verification and will just jump to external experimentation. Second, there are some things we human folk just don't know. The mechanics of Brownian motion are in some ways cataloged and written down, and in some ways are just anybody's guess. The patenting literature often refers to a person having ordinary skill in the art (phosita), and the question of external verification via experiment can theoretically be reduced to a sufficiently complex symbol-shunting exercise, but can only rarely be reduced to a symbol-shunting exercise that a phosita can actually write down and solve. Maybe it's all equivalent in theory (and this is a philosophy of science kind of question which is in no way resolved), but the practical reality is that an equation and an experiment are not identical.

I claim that under correctly interpreted patent law, only externally falsifiable inventions could be patented. Under current law, this is no longer the case. If you write down an equation with the appropriate variable names, you are free to patent. By not distinguishing between internal and external verification, it has become difficult to distinguish between pure math and chemistry.

I feel that about everybody who works in the sciences has no difficulty distinguishing between theoretical work and experimental work. That is, we have a concept which is a no-brainer among the practitioners but which patent law ignores. Is patent law correct to ignore it?

From the economic perspective, ignoring the distinction is folly, because writing down a good design which needs to stand up only to internal verification is an order of magnitude or two easier than a good design which needs to stand up to external verification. If you can ignore the rest of the world, points to ya, but why are we subsidizing your work in the easy stuff that you'd prefer to do anyway? If you write down an internally consistent system, what have the rest of us learned about the world at large?

Some make ethical arguments that both designing an internally consistent system and testing against the world at large are the same sort of work, engaging in some spark of creativity and then verifying the results. Who cares if it's internal or external verification, which is often an arbitrary distinction anyway? In fact, it's the theorists who are really being creative; the experimentalists are just copying the theorists or at best using an informed shotgun approach and trying everything. My apologies to the reader, but I'm counting this as a separate topic, which I'll discuss next time.

In terms of advancing the sciences, the internal and external are very different. Designing better methods for building an internally consistent system is a logic problem: given the premises of the system I've written down, what are the conclusions? Designing better methods for interacting with the world at large involves learning the rules of the natural world and how its rules can be mastered. In a generation, those rules will be folded into the phosita's internal system, but when first gleaned from the outside world, it is a real contribution.

The theorists are often characterized as the revolutionary ones, but it's often the case that the experimentalists are the ones actually speaking a new word, because they are more prone to haphazard external influences. This is where all the true apple-fell-on-my-head stories are: the guy who invented Velcro, the guy who worked out that his uselessly weak glue won't harm paper and thus invented Post-it notes, the guy whose gas cylinder kept getting gummed up with a slick polymer now known as Teflon, the guy whose sloppy lab technique led to the discovery that penicillin kills bacteria.

Better math and better designed systems are good things. We like having fewer moving parts. But this is a substantively different contribution from that made by systems which must interact with the outside world, and it is a mastery of the unexpected and the interaction with the natural world that requires the big cash and dedicated experimentation, and is most likely to provide benefit to others who are designing systems of their own. The patent literature that I have seen makes no mention of the concept of external verification, and I'm not sure if one could actually base a patent system on it. But it gives us another mechanism by which we can see that in the history of patenting was right to agree that mathematical results are not patentable, by giving us another line to draw between pure mathematical equations and chemical formulæ. It should come as no surprise that software falls on the unpatentable, only-internally-verified side of the line.

[link] [A comment]
[Previous entry: "physical property v intellectual property"]
[Next entry: "The cult of the inventor"]

Replies: A comment

on Thursday, November 3rd, the author said

People actually _do_ read this stuff and even comment; I just get all the comments by email. E.g., Ms. SW of Laurel, MD, whose job title is mathematician, points out that falsifiability in mathematics is not as simple as I characterize it to be. For example the four-color theorem was first posed as a question in 1852, but nobody knew whether it was true until a proof 124 years later—a proof which involved a computer trying to color in around 1500 maps. At which point, it starts to look a lot like experimentation. So the distinction is hazier than I characterize it to be here.