Will the Real Singularity stand up [we need a software revolution]

We all know about the coming of the Technological Superiority, that magical moment in the future when machine intelligence will outstrip human intelligence – first by bit, then by more, and then accelerating to All the Intelligence in Universe – and we will be rendered...obsolete? fodder for the machines? Whatever. If you look around on New Savanna you’ll find places here and there where I say, Nonsense! We’re living the Singularity Now! I’m not entirely serious when I say that, nor am I entirely unserious. I’m mostly tired of hearing about some Magical Moment in the Future when Shazaammm! Everything Changes.

Now it’s time to get serious.

Back in 1975 Fred Brooks published The Mythical Man-Month: Essays on Software Engineering. It’s about the difficulty of writing good software. From the Wikipedia entry:

Brooks' observations are based on his experiences at IBM while managing the development of OS/360. He had added more programmers to a project falling behind schedule, a decision that he would later conclude had, counter-intuitively, delayed the project even further. He also made the mistake of asserting that one project—involved in writing an ALGOL compiler—would require six months, regardless of the number of workers involved (it required longer). The tendency for managers to repeat such errors in project development led Brooks to quip that his book is called "The Bible of Software Engineering", because "everybody quotes it, some people read it, and a few people go by it". The book is widely regarded as a classic on the human elements of software engineering.

Such problems persist.

Here’s Alan Kay’s keynote address for OOPSLA 1997 [Object-Oriented Programming, Systems, Languages, and Applications]

One major topic is how the underlying software architecture of the web, noting the HTML is what happens when physicists (Tim Berners-Lee has an undergraduate degree in physics) design software. His general sentiment is that we’re doing it wrong.

Now, just the other day, Patrick Collison, co-founder of Stripe, gave an interview for The Torch of Progress (an online course in technological progress for high school students).

Starting at roughly 45:20 he remarks about the software:

We're still just so bad at building software. Our software sucks, our tools for building software suck, and, maybe for some deep cognitive reasons or something we can't do very synthetic better than we currently are, I hope that's not true and my strong supposition is that it's not true, and so the sense that we're somehow kind of plateauing, we're maximizing in depth as possible, it just feels misguided to me. It stills kills me that we're building, you know, MS-DOS programs and there are so many many layers beyond that which we're currently realizing.

So, 45 years after Fred Brooks argued that we don’t know what we’re doing when we build software, we’re still at it, bumbling around in the dark.

Over the same period, of course, hardware has become vastly improved – if by that you mean cheaper, more powerful, and more reliable. Hardware has done most of the heavy-lifting in the so-called computer revolution. Without this hardware smart phones, server farms, and all of the rest of it would be impossible. Of course, yes, we need software running on the hardware, but the software is clunky in comparison to the underlying hardware. Hardware is so cheap that we can get by the software that wastes CPU cycles and memory.

I don’t care so much about the CPU cycles and the memory. They’re cheap. But I do care about the time wasted in producing the software, which often doesn’t work as designed or desired. That’s where we need a revolution.

A fundamental improvement in software technology – that would constitute a real technological singularity. It has always been the case that a few superb programmers are able to craft high-quality software (given time and resources of course). We need software development technology and methods that allow programmers of medium capability to craft high quality software.