How to write good software

I have a thesis on how to write good software that I would like to persuade you of. It's not an easy process to follow - indeed, much to my shame, I have never successfully followed it, but I think that if more people tried to follow it it would gradually become easier to achieve, and at the end of it what we would have would be a much better software ecosystem.



It's a bit of a complicated thesis though, so I'm going to have to ease you into it. Lets start with two hopefully uncontroversial premises:




* Anything you do that you did not set out to achieve is achieved by accident

* You are more likely to achieve things if you do them deliberately than you are if you do them accidentally



In particular, if writing good software is not your goal then it is something that happens by accident. Thus if writing good software is not your goal, you are less likely to write good software.



This seems fairly plausible as a starting point.



But now I would like you to ask yourself: When waas the last time that writing good software was your goal?



Thought of it? OK.



Now I'd like to ask you this:



Was it really your goal?



Consider the following three scenarios:



* I could spend time working on making our software more reliable for our existing users, or I could spend time exploring this cool new feature that adds way more valuable IP and will probably add like 2% to our acquisition price.

* I could spend time on making our software more reliable for our existing users, but actually our existing users are pretty much a captive audience, but if I add this feature we can capture 10% of the market from our competitor software.

* I could spend time on making our software more reliable for our existing users, but actually I've written just about all the papers I can on this idea, so if I want to have something to write about I'll have to go do something else.



Which branch of these decisions do you think people are likely to take? Which branch do you think would produce better software?



Sure, sometimes adding that feature will produce better software because it will radically change the utility of your system, but if you're making these decisions over and over again, you're probably going to be producing some pretty awful software.



Sure you might want to produce better software, but ultimately that isn't your goal. Or at least it's not your organisation's goal. You can subvert the organisation's goal to a certain degree, but if you subvert it too far then you're not doing your job and you get fired. There is room for a certain amount of individual subversion, but as a group it only goes so far before the corporate overmind realises that its drone are deviating the company realises one of its teams are off-mission and they step in to do something about it.



A long time ago when I was young and naive and working for a company I shall not name I had a conversation with a colleague I shall refer to as B. It went roughly as follows:



Me: We write really buggy software. We should probably not do that.



B: Why?



Me: Well, bugs are bad. They make users unhappy.



B: Great. That means they'll pay us to fix the bugs.



To the best of my knowledge this was not an official company policy, but it was still remarkably educational in terms of how incentives work.



This is the thing: When your goal is to make money, you will generally speaking do the thing that causes you to make money. While the idea of a corporate obligation to maximize profits is nonsense, nevertheless generally speaking behaviours which cause you to make more money will be tolerated and behaviours which cause you to make less won't. You can add other constraints, such as morality and good taste, but generally those constraints will be rubbed away at the edges by the structure of your incentives.



Further, money breeds money - by increasing your wealth you increase your capabilities, which increases your ability to make more money. So by constraining yourself you are in danger of ceding the playing field to someone who hasn't - if they decide they want your share of the market they will usually out-compete you by being able to throw money at the problem.



Sometimes this is not harmful to the goal of producing good software. It can be the case that the best way to make money is to write good software.



But normally the best way to make money is to write good-ish software. You know that thing where getting 80% of the way there takes 80% of the work and then the remaining 20% takes the other 80% of the work? Well, it turns out that users are probably not going to pay twice as much for something which is only 25% better (100% / 80% = 125%). Especially now that we've been doing this for decades and have conditioned them to believe that this is just what software is like. And investors and acquiring companies? They literally couldn't care less. You think they use your software? Nah. They just care about the IP, the talent, and the data you've built up. Turns out none of that relies on quality software.



"But David!", you cry. "What about open source? Doesn't open source solve these problems? Can't we write quality open source software?"



So first off, it's worth noting that regardless of whether we can write quality open source software, the reality seems to be that we don't. Consider Linux on the desktop vs OS X or even Windows. Sure, I use and sorta like it, but I don't think I could keep a straight face if I tried to tell you it was higher quality.



But secondlythis is one of the major misconceptions of open source. You know what a successful large open source project looks like? It looks like one where a great deal of money has been spent on its development. Successful large scale software development needs a whole bunch of people spending a lot of time working on it, and it turns out that people need to eat. And it turns out that in order to eat you generally need money. So open source development succeeds not as a way for individuals to collaborate on making quality software, but as a way for corporations to collaborate on mutually making more profits. This makes the incentive structure of open source remarkably similar to the incentive structure ofproprietary software.



There's of course a second type of open source - the small single (or at most several) person project which does one very specific thing. Most commonly exemplified in the ecosystem of libraries that most programming languages have.



Given the average quality of those ecosystems, we're still not producing much evidence that we can write good software this way.



The thing is, in order for this to result in good software, the author must:



* Have enough free time to work on it

* Care enough about the problem that they will devote as much of their free time as they need to to work on it

* Actually want to write quality software rather than just software which scratches their own itch

* Actually be good at writing quality software

* Understand the user base of their software well enough that their idea of quality lines up with their users' ideas of quality



It definitely happens. There are some pretty good libraries out there maintained by one or two developers apiece. But these people are basically unicorns, and relying on an adequate supply of unicorns is probably a sign that your approach is in trouble.



And even if we could solve this problem it still wouldn't be good enough, because most problems are too large to be solved by individuals and need teams working on it.



So how do you write good software? You get together teams of talented people, you give them the capability to work on it without worrying about where their next meal is from, and you give them the goal of producing good software. You do this without them having to worry about their project drying up due to lack of funding, or the source of their funding telling them yes it's very nice that they want things to work but maybe could they make them work now?



So, now that we've smashed the capitalist structure of dominance done away with the corrupting influence of financial incentives produced an environment in which people can focus on the task of writing quality software, are we done?



Oh, if only.



Lets recall a line I slipped into a bullet point list up above. In order to write quality software you need to "Understand the user base of their software well enough that [your] idea of quality lines up with [your] users' ideas of quality".



How well do you think that currently happens?



The answer is that it happens in exactly one case: When your users are exactly like you. Sure, you can, do and should talk to your users and find out what they're like, and in the course of doing so you may educate yourself into seeing through their eyes to some degree, but ultimately your vision of what quality software looks like is deeply tied in with your view of the world.



One of the core features of the feminist concept of is that your view of the world is intrinsically shaped by who you are and what you've experienced. If you've not experienced a problem, or don't know anyone who has experienced that problem, then you probably don't see that problem as acutely as those who have (and may not see it at all). This is how we end up with software that only works in certain time zones, or doesn't handle foreign languages, or imposes a real name policy, or asks for your gender as "Male or Female?". The authors didn't experience those problems, so didn't think they mattered.



(This is also a function of financial incentives: e.g. If < 1% of our users are non-binary and we can save effort on data modelling and do a bunch of "charming" gendered things for ad targeting if we just ask male/female, which way does the path of more money lie? Unicode is hard, and if we're not interested in the foreign market, which way does the money lie? But we've already solved that problem, right?)



As we're fond of smugly declaring as an industry, software is eating the world. More and more things are getting automated, and more and more automation is going to be just a function of software.



What does this mean? It means that our user base is everyone. Even if they're not using our software, they're using things that are using our software, or they're using things that will be "disrupted" by our software.



Good software is software that makes lives better. If your software is making things worse it doesn't matter how solid an execution it is. You may have written the best spyware engine in the world, but I'm not going to think well of you for it and I'm not going to call your software good.



So in order to write good software you need to understand your users. And your users are everyone.



So what do you do?



Well, I'll tell you what you don't do. You don't assemble your team of software developers from a group that looks remarkably similar to the top end of the existing structures of social dominance. Because if you do then what you are developing is software that solves peoples' problems in direct proportion to how well society already solves their problems.



Good software is not software which reinforces the entrenched power structure.



Look around your software company. If the team is any like the ones I've worked in, it's mostly white, mostly male, mostly middle class. If you're lucky you're working in a team which is at least not perpetuating this problem, but the problem is so endemic in our entire industry that right now software absolutely is mostly written by the people who are socially more or less at the top (and mostly in service to the people who are actually at the top).



So in order to produce good software we need to stop doing that. Good software is produced by teams who represent the people they're building software for.



So now that we've formed a socialist utopia removed the financial incentives that prevent writing good software and smashed the patriarchy solved the fundamental lack of representation in our industry, can we write good software?



Well, maybe. At the very least, now we can start to learn how.
Full Post