April 30, 2013 by startupengineering .

[image movie replica]

When I had a motorcycle, I also had, inevitably, a theory about motorcycle safety. The theory went like this: Motorcycle’s are about as safe as cars. On the plus side, you sit higher than in a car seat, so you can see better. Also a motorcycle is smaller, more maneuverable, and quicker than a car, all features that give the alert cyclist a better chance to maneuver out of sticky situations and avoid accidents. On the minus side, you’re not protected on a bike, so if you do have an accident, you’re likely to be much worse off than the driver of a car.

This theory served its purpose admirably. I liked riding motorcycles, and needed a reason to believe I wasn’t putting myself or my family in harm’s way. At the same time, I wanted to be realistic, which required some acknowledgement of danger.

Adducing evidence for this theory was easy. Good view? Quick? Maneuverable? Check. Check. Check. I (thankfully) never tested the part about the consequences of an accident at speed, but that stood to reason anyway. It never occurred to me, or if it did, I suppressed the thought, that I could test my theory in some other way, like by checking motorcycle accident statistics.

Startups operate on theories too. Actual businesses may or may not operate on a theoretical basis. It’s possible to have a business that simply does x and customers spend money for it, with virtually no broader view. Most of the time though, and all the time with startups, companies operate on the basis of theories. “We believe (because of X, Y, or Z) that if we do A, customers will do B.

Startups easily fall into the trap of testing the validity of the hypotheses in their business model canvas the same way I tested the validity of my theory about motorcycle safety. They take the factors that led them to the theory in the first place, and look for further evidence to support them. That evidence is almost certainly available – if it wasn’t, the entrepreneur probably wouldn’t have come up with it in the first place. Sadly, it’s not evidence that the theory is true.

This is a reason why Flashpoint asks entrepreneurs to do three things that aren’t intuitive or easy.

1. Distinguish between having a theory about an aspect of your business and having a hypothesis about it. A theory describes the mechanism of how a process or transaction important to the business works (e.g. I can get people to buy my service by advertising in bus shelters.) A hypothesis is part of a plan you use to test whether the theory is correct.
2. Validating your theory involves not only coming up with a good hypothesis, but also setting validation conditions before the test, rather than trying to make sense of what happens afterwards. For example, “I will put ads in three bus shelters. Each one will show a special URL with a counter. If at least ten people per week come to our site through that URL and purchase a product, I’ll judge the theory to be correct.”
3. Seek to disconfirm rather than confirm the theory. In my motorcycle theory, “good view,” “quickness,” and “maneuverability” are hypothesized to be evidence that I’d be less likely to get into an accident. But I could confirm that motorcycles met those criteria all day long, and be no closer to actually knowing whether that was true. If instead I’d been (like my mother) equipped with a theory that motorcycles are VERY DANGEROUS because they tend to skid out on wet roads, I could have found plenty of evidence for that too, and been no closer to valid knowledge on the subject.A much better approach, if my theory is that motorcycles are less accident prone than cars, is to look for evidence that that isn’t true. Evidence that I’m wrong (like,say,data showing more accidents per miles driven) forces me to ask “is this piece of evidence an anomaly, or should I trust it, reject my theory, and come up with a different one?” If I go in the other direction and look for confirming evidence, than whenever I have it, I have to ask “is this piece of evidence enough to confirm my theory, or do I need to keep testing and find more?” Generally, coming up with anomalies will poke holes in your theory and lead you to abandon it, a lot faster than coming up with confirmations will prove your theory true. In practice, entrepreneurs rarely even stop to think whether the confirming evidence is enough to rely on – they believe their theory to begin with, and even minimal evidence is plenty.

The ubiquitous lean entrepreneurship catchphrase, “get out of the building and test” is a great start. But following that advice doesn’t make your business valid, any more than getting accepted into medical school makes you a doctor.