Why Software Development Requires an Evolution of Consciousness

I was recently sent a link to a dating from 1992. He was a proponent of the idea that software was ALL design, even down to the area of testing, comparing it with traditional engineering processes which had a significant focus on the phase of building something. In software development building it is a miniscule part of the process, now frequently done under automatic control, especially if you are using This means that the traditional build phase in software is almost a non-operation (called a no-op, or nop, by techies). Click an icon and - bang - its built.



However there was a little nugget in one of Jack's essays which caused a rather big "Aha!" moment for me. I recently gave a talk at in April () which was based on my research into phenomenology and software, words you don't usually see together, as well as my interest in the struggle between seeing the whole and the parts, as wonderfully researched by . Jack Reeves made a point which connected with these ideas of the whole and the parts, monism and dualism, etc. His point is fairly obvious to anyone who has done any significant amount of software development as he identified that the high-level software design influences the detailed low-level design (to be expected), but ALSO that the detailed design will influence (or should be allowed to influence) the high-level design.




To quote Jack:



"The high level structural design is not a complete software design; it is just a structural framework for the detailed design. We have very limited capabilities for rigorously validating a high level design. The detailed design will ultimately influence (or should be allowed to influence) the high level design at least as much as other factors. Refining all the aspects of a design is a process that should be happening throughout the design cycle. If any aspect of the design is frozen out of the refinement process, it is hardly surprising that the final design will be poor or even unworkable."



I can hear the programmers among my readers saying: "So what?", but seen from the perspective of how difficult it is to grasp the relationship between the whole and the parts, I now realise that it is this issue that causes such difficulty when writing software systems and why traditional hierarchical methods of management will not work. This should ring significant alarm bells for anyone wishing to manage a software project because due to the inherent complexity of systems it means that the smallest details down in the low-level design can trip up the whole project effort.



This is why I am so sceptical about large system initiatives managed in the traditional way. One of the latest casualties being the which was abandoned after a project spend of EUR98 million. Ouch.



This does NOT mean that we cannot make large systems, it just means that we must factor in this tight link between the whole and the parts, the high and low level design. This is hard, even requiring a significant evolution of consciousness if we take on the ideas of Henri Bortoft and the Goethean scientific approach.



So much of our culture sits on assumptions that hierarchical structures work. Indeed a top-down approach is the usual way that I have done projects in the past. But I have to jump to the risky items FIRST, many of which will be low-level design problems that, if not solved, will mean the whole effort is a waste of time. This is why I much prefer attacking problems incrementally, which is a much more organic process and also explains the rise of Agile software development techniques.



In the end it means that as a programmer, if I am to be successful, I must be aware of the way I think and how I perceive the world. Maybe this is why I have recently become interested in watercolour painting - but that is another story
Full Post