Monday, March 1, 2010

Documentation and the philosophical distinction between memory and reminiscence

The title of this blog, and the 'About' text about the parallels between life in general and coding specifically, aren't just meant as a feuilletonistic nicety. Over the past years I have reflected occasionally about both my job as a software programmer and development coach, and about philosophy (especially ancient ethics); and I've noticed connections between these two fields again and again. So my goals for this blog include exploring those parallels; today I want to try and get started with one such exploration. I'll introduce you to a thought I recently came across, and then link it with the analogies I see in our profession.

1) At her thoughtful and scientifically well-informed blog on the philosophy of memory, Susannah Kate Devitt writes: "I'm fascinated by the degree of information we record about our lives today and the issues around what use such records will play in our lives at a later stage of life. Perhaps we spend too much time documenting our lives and not enough time improving our abilities to live them."[1]

The line of thought that leads to this general statement is about the distinction between memory and reminiscence, as it is made in ancient philosophy. Memory, on the one hand, is seen as a mental function that is useful in gaining knowledge: it enables you to think independently, to understand complex arguments and examine them for their truth or falsity, and to teach your insights successfully to others. Reminiscence, on the other hand, is a mere replay of something you've experienced, and there is nothing really to gain from it — it's good for some amusement at best.

One notable thing about the distinction between memory and reminiscence is that it isn't just descriptive: it's not only that a difference is made between two functions of the mind; they are not simply neutrally described as we find them in people. On the contrary, there is a normative dimension here. The two functions are examined with respect to their value. It is made very clear that striving for knowledge is a good thing for us, while indulging in reminiscence isn't what a well-oriented person would sensibly do with her time.[2]

2) Let's now turn to the question of 'documenting our lives' and 'our abilities to live them' in the quote above. It is an important element of ancient ethics that your ability to live your life generally improves with the skills and knowledge that you gain from exercising your rational capacities: you become more successful by using reason and good sense. The two functions discussed here are then easily classified: memory would help you in living your life well, while reminiscence doesn't repay the time you're investing in it. (In fact, that is what makes them good or bad respectively.) Apply this to our habit of recording experiences in our lives: the mere documentation of what happened, the heaping of images, audio and video clips, status updates, tweets and diggs (and what have you), won't contribute much to the success of your life, because they have no potential for growth and development — they're mere pictures, they're mirroring what happens in your life, but nothing substantial can be built on them. Real memory is about insight into what happened as much as it is about re-presenting it; it's about learning from it, it connects your present actions, thoughts and feelings with those past experiences. And we are perhaps not putting much effort into that.

Once more, stating this isn't just about describing what happens. It is evaluative: that means it's about finding out what's good and what's bad to do — or, to put it less dramatically, it's about deciding what to do with your time, how to invest it wisely.

3) I've elaborated this line of thought in the hope that it makes some intuitive sense to you; it may seem rather 'philosophical' at first glance. And of course, that's exactly what it is. But that doesn't make it irrelevant; it also doesn't make it a topic that we may only think about in moments of reflection (which some people seem to picture as relaxed and calm situations involving an armchair and a bottle of wine). On the contrary, thinking about these things is perhaps the single most important activity there is in our lives. (I'm not going to argue for this claim here; it would take way too much space. But think about it for a moment: if you don't reflect about these topics, how would you know that you're investing your time well, the time of that single life that you've got? Are you steering your own course or are you merely on automatic pilot, as programmed by your environment, your childhood experiences, the suggestions you receive from the media, and other such influences?)

But my goal is to draw some connections between these reflections and similar issues we find in software projects. Let's change over into that area now.

4) In the quote I gave above, the word that captured my attention was of course 'documenting'. If you have spent some time in a software project, that is a topic you will likely have come across; and probably you have encountered a good deal of controversy around it. Documentation has been an undue favorite with many managers and process designers in the past, and there has been a powerful counter-movement in the agile community, which even made it a core value of its practice that "working software is more valuable than comprehensive documentation."[3]

(However, notice that this is about business value; you have to read it this way: "For our customers, we believe that working software has a higher business value than extensive documentation". So there is a slightly different concept of value involved here than the one I've used above in section 1).)

Before I start outlining a parallel between the philosophical distinctions in memory and the area of software documentation, I want to put aside some uses of 'documentation' in that area which I don't have in mind here. One is user documentation: the audience for that sort of documentation is end users who work with some released version of a software product, and obviously they have to be taught how to use it. Then there is tracking and reporting: as in any other sort of project, software projects need to display, both for people inside the project team and for stakeholders outside it, what the current status is and how the reality of the situation compares to what was planned. Furthermore, information is collected about changes made in technical detail, mostly by means of revision control tools: this might be called a form of documentation (in management speak, it is sometimes said that "it is documented who made which change, when, and for what purpose"), but mostly it is merely a kind of backing up snapshots in order to be able to roll back to them when it turns out that some change wasn't a good idea. If used in the latter way, it is a pragmatic tool to support daily work; if used in the former way, it turns into a control instrument typically employed for putting blame somewhere (or for pre-empting blame).

In contrast to these, the concept of documentation that I have primarily in mind here is technical documentation, as it is for example introduced in the Wikipedia article on software documentation: "When creating software, code alone is insufficient. There must be some text along with it to describe various aspects of its intended operation."

We have to unpack this claim first before we can discuss it.

When developers are working on a software product, i.e. when they extend its functionality (or implement it in the first place), fix program errors or clean up its internal structure (a process known as 'refactoring'), they know how to deal with the source code: they know which passages to look at, they know which consequences a change in the code may have, and so on. Thus in an ongoing software project, besides the code itself there is vital information sedimented in the skills and knowledge of the developers. That is why it would not do to replace these people with others (even if equally educated), at least not without allowing for a period of orientation and training. When the Wikipedia article claims that the actual code alone is not enough, it is that extra information which it refers to. However, instead of directly talking about it as skills and knowledge (something that would be attributed to people), it assumes that it can be codified, just as the source code is, in additional text (i.e. it assumes that there is a largely context-independent format, perhaps prose or one of various visualization formats, which can capture this sort of knowledge). More precisely, then, the claim of the Wikipedia article, when made fully explicit, is something like this:

i) Software creation requires both
a) code
b) skills and knowledge with the developers who
create and work with the code.
ii) These skills and knowledge can be codified
in text.
iii) It is not only possible, but also necessary
for software creation to codify these skills and knowledge
in text.
Therefore: "When creating software, code alone
is insufficient.
There must be some text along with it [...]".
It is not obvious why we should accept premises ii) and iii), though. (Not obvious, at any rate, from the cited Wikipedia article; and to be sure, both these premises are contentious, to say the least, in recent discussions. It's not my goal in this article to argue for or against them. But it should be clear already from my formulation that they are not self-evident once made explicit; it would take some argument to make them acceptable, and thus make the overall claim sustainable.)[4]

Technical documentation, in the discussed sense, is an attempt to externalize the competence of the developers currently working on the code. It takes many forms, including code annotations (e.g. in the well-known doxygen or javadoc formats), class design or control flow diagrams, verbal descriptions of module responsibilities, explanations of intentions behind design and architecture decisions, and more. The objective is to record something that is present in skills and knowledge, in order to re-use those records, later on, with the purpose of re-instating such skills and knowledge.

5) We are now in a position to see the connection between this and the philosophical distinction made earlier. We might say that documentation is for a software product what memory is for people; and our ability to carry out software projects depends on skills and knowledge in a similar way as our ability to live our lives well does. (Although obviously we're talking about two different sets of skills and knowledge here — I'm just pointing out an analogy.) But from the philosophical discussion above we can learn that there are two different sorts of recording the past: one that has a constructive role in our personal development and therefore contributes to our ability to live our lives well, and one that is just a waste of time and energy. In the discussion above, these were called 'memory' and 'reminiscence' respectively.

If we read 'documentation' for 'reminiscence', we might rewrite the quote I gave above, with respect to software development: "There's such a high amount of information we record about our product; and so much discussion about the use such records will play in our work at a later time. Perhaps we spend too much time documenting our software and not enough time improving our abilities to work with it." Have you ever had a thought like this in one of the projects you were working in?

Just as genuine memory builds the knowledge required to live your life successfully, and reminiscence doesn't do that for you, there are also two ways of preserving technical information in the software world: one that builds the knowledge and competence required to work successfully on your software, and one that doesn't. Consequently, it's not enough to just write down what you know somehow: you have to find a better way than that. That way will certainly have to do with building skills and gathering insight about what you're doing (and how well it works); it won't be easily externalized or formalized, however. In other words, it won't resemble 'documentation', it will rather be like a constant learning process, an
exercise in self-examination.

In contrast with the attitude from the agile movement that I quoted above, however, this insight doesn't stem from the goal of maximizing business value. It comes from a totally different way of looking at things, one that connects with more fundamental principles.

6) Sometimes things that look alike are really different; by bringing conceptual distinctions to the fore, philosophy can teach us to see the differences, and that's especially important if the things in question aren't simply different in themselves, but also have different values, that is, if they are more or less good or bad for us. Insofar as philosophy helps us to make the necessary distinctions, it improves our abilities to deal with things, and be more successful — be it in living our lives in general, or simply in a more restricted field, such as developing software.



[2] The text quoted by Devitt is Plato's Phaedrus, at 274c-275b; Plato makes this contrast rather sharply and is more severe than most other ancient authors. The harsh judgment isn't restricted to ancient writing, though. It can equally be found in contemporary aesthetics. For instance, there is a similarly decided, and judgmental, formulation of an analogous point in Roger Scruton's Beauty, in the section about fantasy vs. imagination. See Scruton, Beauty, Oxford UP, 2009, ch. 5 on 'Artistic Beauty'.

[3] As expressed in the Agile Manifesto;

[4] The Wikipedia article emphatically notes several times how important it is to have technical documentation in the discussed sense; however, insisting that something is in your view important is not the same as demonstrating that it is true.