Y-Combinator matters. It’s a big deal. Whether it’s the #1 ranking in the 2013 accelerator rankings[1] or teaching a class on startups at Stanford[2], or Paul Graham, one of its founders, stating that, “I have in my head more data about that [startups] than anybody else,”[3] it matters. And that’s why the words spoken and written by the people at Y-Combinator matter too. Young (and not so young) entrepreneurs listen to what the people at Y-Combinator say.
But good research design matters too. Without good research design we can’t do good analysis and come to useful conclusions. And some of the people at Y-Combinator need help with research design. In a blog post I wrote for Etohum back in January 2014[4], I already criticized Sam Altman for engaging in poor research design in an analysis of the defining qualities of successful startups he had done and then written about on his blog[5]. Since Sam is the top dog at Y-Combinator this matters. Since, as of 27th January 2015, the blog post had been viewed over 129,000 times, this matters. Since, as a result of his poor research design, the list of defining qualities was essentially meaningless this matters. But I’ve already written about this so why am I bothering to write again? Let me explain.
Well not only does Sam Altman have a very popular blog but so does Paul Graham. A month or so ago, the startup ‘chattering classes’ were talking about a blog post of his on meanness[6] (the title is ‘Mean People Fail’)and while I didn’t have enough time to read it when he posted it, I flagged it as a ‘must read later’ blog post. I finally got round to reading it last week and couldn’t believe what I was reading. The blog post was in the same spirit as many are, i.e. trying to find the key characteristics of founders / startups etc. In this one he was arguing, as the title of the post suggests, that mean people fail and nice people succeed. He qualified this by stating that he was only focusing on the startup world and not drug barons or hedge fund managers and then gave a ton of reasons why he thought mean people failed as startup founders while nice people succeeded. Controversial? Maybe. Correct? We have absolutely no way of knowing from the analysis that Paul Graham has done. Leaving aside definitional issues of what it means to be mean[7], yet again, as Sam Altman did for his post on successful startups, the research design is lousy. Yet again we see someone from Y-Combinator choosing on the dependent variable[8]. To quote the man’s own words, “And yet while there are clearly a lot of mean people out there, there are next to none among the most successful people I know. What’s going on here? Are meanness and success inversely correlated?” Well, the way to answer that question is to look at a random sample of all founders, successful or not, and find some way to measure how mean they are and then measure success and see if there is a correlation. But that’s not what Paul Graham has done. He just looked around at all the successful people (founders, professors and programmers) he knows (that’s the choosing on the dependent variable part), compared this to the fact that the World is full of mean people while most of the successful people he knows are nice, and extrapolated from this that mean people fail as founders while nice people succeed. Well, all we can really conclude from this is that the successful founders, programmers and professors that Paul Graham knows are generally nice. He’s a lucky guy.
Now you might disregard this post of mine as the rantings of a frustrated academic and that’s fine. However, this doesn’t change the fact that the issue I have now raised twice really matters. Paul Graham and Sam Altman are not two no-name bloggers with 25 people reading their blogs. They are two of the most influential people in the startup world with a huge audience and as such they have a huge responsibility. What they say and write matters. And in these two posts they have failed to live up to this responsibility. I’m sure they write these blog posts because they want to be helpful and informative, the problem is that in these two instance they are neither and all because of lousy research design. What they need to realize is that good research design matters.
[2] http://techcrunch.com/2014/09/16/y-combinator-will-teach-a-class-on-startups-at-stanford-this-fall/
[3] https://www.youtube.com/watch?v=ii1jcLg-eIQ
[4] http://english.etohum.com/what-drives-success-in-startups-dont-ask-sam-altman
[5] http://blog.samaltman.com/super-successful-companies
[6] http://paulgraham.com/mean.html
[7] http://www.inc.com/jeff-bercovici/paul-graham-mean-people-fail.html
[8] Somewhat ironically, in the blog post Graham mentions that he may be guilty of selection bias but he then describes a different selection bias not the really important one that he is guilty of committing
