Suddenly everyone seems to be talking about SCRUM. In fact it's become (rather amazingly, given what it is) a requirement for employment in a lot of the jobs I am currently looking at. I therefore thought it best to investigate what SCRUM was to see whether I could sensibly put "has SCRUM experience" on my CV.

Following a short article by Robert Martin comparing methodologies I decided that I could, though after looking at it in some detail myself (see the references below) I'm not so sure it's such a good thing.

Because I have a bit of a problem with it - not so much about what the Methodology is saying (which isn't very much), but about what it omits.

If we cast our minds back to XP, those of us who were around at the time might recall how it was stated that XP's principles enabled one another and worked off one another when they were all being put to use at once.

I wonder now whether XP was just a bit too big a pill to swallow for most management teams so that SCRUM was presented as a sort of "acceptable face". But - it's all very well talking about a light-weight method, SCRUM simply doesn't work without that one thing that all of these methodologies need and that one thing which accounts for 99% of the difficulty and effort of working in this fashion:

Flattening the cost of change curve.

Ken Schwaber in his presentation below, bless him, does allude to the fact that your speed might just slow down as time goes on, but he makes sure he describes this as happening between releases (i.e. outside SCRUM) rather than between Sprints. I'd like to know what Scrum Masters do when they start plotting their Burndown Chart and find that they have a curve, rather than a line, and that that curve rather disturbingly decreases in gradient as time goes on.

You could rather blithely answer that it is the Scrum Master's job as QA guy or whatever to make sure that code maintains its beautiful integrity across Sprints. This, however, is where *all* the effort in software engineering happens, and it had better be your best engineer that is Scrum Master if that's going to be his particular job.

And even then I don't think anyone ever believes you will flatten the curve completely. Rules of interoperability (c.f. The Mythical Man-Month) still state that as time goes on the same step-wise increment in functionality will take longer and longer to implement. Keeping code's integrity clean will stop you getting into a mess, but it wont prevent your code getting more complex.

And even if you could still somehow do it you still have to factor in requirements-churn, which is not going to be constant throughout the lifetime of the project.

And (and I'm sorry about all the "and"s) you also need to measure how much extra deprecated work you are having to produce in order to sustain your short iterations and rapid release. I do most certainly believe that some of this has to happen, however I do not believe that time should be your only consideration in determining your iteration life-cycles. You have to consider the certainty of requirements and the risk of implementation as well. If you're sub-optimal, you could end up doing a lot of un-necessary refactoring.

Everyone wants predictability and everyone wants increased productivity. SCRUM and XP both recommend the mega-sensible approach of taking vertical slices through a product and developing iteratively. I'm not quite sure about SCRUM insisting on "release" - sometimes that's simply not possible - just "completion" is enough for me. Time-boxing meetings just seems like a bit of a sales-pitch to all those managers who think their engineers waste a lot of time arguing (those managers should read PeopleWare or start managing something else). I certainly wouldn't want anyone ringing the bell and saying time's up if I'm still not sure what I should be doing that day. Of course, you need structure, but SCRUM doesn't have to lay that down - decent team leaders do that already in whatever way suits them and their team best.

After that, you have to look at XP or other techniques for the best way to improve your engineering practices.

Ask yourself: if 65% of software doesn't make it out the door, why is that? Is it because programmers aren't typing fast enough? Is it because programmers are spending too much time chatting about TV or something? Is it because project management is up the spout?

Or is it because programmers are producing too many defects?

SCRUM, well, ok, maybe it's a good way to get XP through the door. Introducing iterative development is fantastic but you really need to go elsewhere to make the biggest improvements in productivity and predictability. I strongly suspect that where claims of SCRUMs great successes are being made that actually it's something else, not actually part of SCRUM itself but something that SCRUM has helped introduce, that is making all the difference.

Richard

References:

Robert Martin's article can be found here.

The original SCRUM white paper can be downloaded from here.

The author himself talks about it here:

And Jeff Sutherland adds his views to it here:

At first hand this may not seem like such a big deal. After all, we had methods, we had data structures - what is an object if not just a simple collection of methods, each of which takes a pointer to a data structure as a parameter, all gathered together somewhere nice and convenient.

Indeed, from a basic programming mechanics point of view that is all it is, at least until you get into polymorphism and the next principle "Interface" (and more on that in the next post).

However the concept of the Object and Object Based Programming introduced a whole new approach to programming which was, in my opinion, a far greater Paradigm Shift than the later much more lauded Object Oriented Programming.

Objects brought with them Object Design, and Object Design allowed designers to construct solutions on the basis of a model of the problem domain, rather than approaching the problem from the point of view of, well, a computer.

The programmer's job prior to the arrival of, let's call it, "Object Based Thinking", was along the lines of a well know University Textbook of the 80s called "Algorithms + Data Structures = Programs". Although a programmer would quite sensibly use names from the problem domain in the solution domain in order to make understanding of the code easier, the two domains were still very much separate. The solution solved the problem using its own artifacts - i.e. data structures and functions. Programming was procedural in the main because people generally think procedurally - i.e. in order to solve the problem we do this thing first, then the next, and so on. Keeping track of what was going on came down to Flow-Charts and Data-Flow Diagrams, and there was always a "main loop" somewhere, processing events and farming the work out.

Object Based Thinking turned that on its head completely. Now the solution domain consisted of interacting objects bouncing of each other like billiard balls on a table. No more main loops. No more data flows. Stack-tracing, that vital debugging technique of the procedural days, more or less lost its value as objects got smaller and better defined (cohesive and loosely-coupled) and as the number of billiard balls knocked and re-knocked around the table in response to one external input increased.

Indeed, in Object Design, you never think beyond what is using you and what you use - i.e. your immediate neighbours. Later on (next principle) you stopped even thinking of them (and them of you) beyond your interfaces. Reusability was just around the corner.

And with Object Design came the rather lovely realisation that this flexibility allowed you to build your solution as a close model of your problem. If the problem domain has a Customer asking for a Loan from a Broker, then the solution domain could have a Customer object call a Loan_Request function (now called method) from an object called a Broker. If the Broker has to check bank references on the customer before agreeing, again the software can have that Broker object call a Bank object's Check_References method passing in the Customer object. And so on.

And this makes software so much easier to understand that it is probably the most important development in software engineering that we have ever had.

But - there's something rather counter-intuitive to the engineer, who likes to think in terms of machines sorting out problems, which has definately caused a slowness in the acceptance of problem-domain modelling as a technique. Although it is clear that it cannot be taken to the n'th degree, the temptation to use algorithmic artifacts rather than problem artifacts in design is quite clearly strong (as I have witnessed in my career). However in my opinion, Problem-Domain-Objects are the linchpins on which the whole future of software development hangs - a future based on the steady increase of the ready-comprehensibility of the software that we write.

Richard

The Principles of Software Engineering: start previous

The reason for this is that I'm on the last furlongs of getting QuSheet finished and I haven't wanted anything to distract me.

Things are likely to be a little quiet here for a couple more weeks until I get QuSheet out to my Beta testers.

Just out of interest, after more or less finishing the development work on QuSheet (I've got about a day's worth of bits to clear up), I still had the following to do:

1) Write all the help text.
2) Sort out installation (using ClickOnce)
3) Sort out activation code so that installations are legitimate
4) Sort out icon and logo
5) Script and write tutorials (still doing this - massive job in lieu of writing user manuals (another massive job))
6) Sort out web-site (to-do)
7) Script and write over-view presentation

Then there's beta testing. I'm hoping I'm not going to get a lot of problems thrown up by this, as I've been quite meticulous testing out all my functionality as I wrote my help text and tutorials.

Time ticks on and I'm rapidly running out funds. I'm pretty confident I'll finish before I run out completely, though then it'll be time to either get some work, funding or quick sales.

All the best

Richard

From the point of view of building a compiler this isn't exactly rocket-science, but just to recap on the fundamentals a Function is a named block of code which:

1. may be executed from some select area, maybe everywhere, in the rest of the code, and always returns control to the caller, possibly passing back a return value,
2. takes parameters, which may be positional or named, by value or reference, mandatory, optional or even of undetermined number
3. has its own scope and can declare local variables

Functions are the way (and for a long time were the only way) to manage complexity via Functional Decomposition (FD), the process by which blocks of code are split into functions in order to make the code easier to understand (and therefore easier to get working). FD aids comprehensibility by reducing duplication (i.e. removing clutter) and adding english language explanation in the function and parameter names. FD helps if, and only if, it is done well. Done badly it makes things worse.

Functions also introduce a concept which is so fundamental and mind-blowingly important to programming that it is all too easy to miss. Every function has a user and a provider: a separation of concerns which is the beginnings of Client-Server programming. User and provider need to communicate effectively since neither should know what is going on in the other's domain (i.e. for an effective separation and simplification of complexity the function writer should know nothing about how and where his function will be used and the function user should equally know nothing about how the function has been implemented). Programming languages give very little help in easing this communication. Documentation is the frequent resort of the inexperienced programmer but the best result comes from good FD (i.e. design) and the best possible choice in function and parameter names.

Steve Maguire in his book "Writing Solid Code" was the first person to take a close look at FD from the point of view of reducing bug-risk. I had been programming in C for 9 years when I came across this and I found it a real eye-opener. In particular until then the attitude of the function writer to the function user was one of "caveat emptor" (or "user-or"), even if the user and writer were one and the same person. Steve Maguire suggested that the writer should try to help the user not make mistakes when using his code by not writing functions which tripped the user up. Where Functions had divided the programming world with a big fence (frequently called the function library), this book tried to build a bridge and persuade the two groups of programmers to be nice to each other!

The section on Assertions is the only part of the book that I'm not sure I agree with, as my friend Sean once pointed out to me you shouldn't change the behaviour of a function between debug and release versions. I accept that, but the rest of the book is faultless and excellent.

Ask any group of programmers what language they program in and you'll get the expected answer (C, C++, C#, Java, etc). The truth is that we all program in english (or whatever our spoken language is). We punctuate in a computer language, but 90% of the information we gain when we read through a program comes to us in our native language. Although I shall come back to this later, I strongly believe that our native language skills are fundamentally important in the way we program, and that we need to think more about how we use our vocabulary and such things as prepositions, adjectives and so on. For now let me just say that trying to understand code when you have to refer to documentation every time you come across a function is like trying to read a book with a dictionary at your side - cumbersome and slow.

Functions are generally named according to "intent". One debugging technique I have found very useful is renaming functions from "intent" to names which reflect what the functions actually do, even if these names are somewhat long (e.g. rename "register_customer" to "register_customer_name_if_not_present_or_out_of_date"). The latter might seem cumbersome, but follow the trail of functions up and you may spot where user and writer have made different assumptions about what the function is doing. I wish we could also do this with parameter names but unfortunately there has been a decline in the provision of keyword, or named, parameters in programming languages (e.g. compare "register_customer(true,false)" with "register_customer(account_holder=true,overseas=false)", though I note that C# is thinking about bringing this back on a future release).

Before the days of OO all you could do by way of code design was FD plus gathering functions together into modules and libraries. This latter activity was "nice" but not really what design was all about - that was the FD. Now that we have, wisely, moved away from the design-up-front approaches of the past, FD has become even more important as an ongoing refactoring activity, as functions coalesce and explode in our drive to produce a solution which is rational and comprehensible.

Richard

The Principles of Software Engineering: start previous next

In Chinese philosophy, the concept of yin yang is used to describe how seemingly disjunct or opposing forces are interconnected and interdependent in the natural world, giving rise to each other in turn. (Wikipedia)

Yang (the more dynamic element) and Yin (the more meticulous) appear all over the place, including Computer Programming.

People, and hence programmers, tend to be more one or the other. I'm more Yang than Yin. Currently, with QuSheet, I'm going through a Yin phase, and I'm starting to chew the carpets, but since I'm on my own with this I have to concentrate on my Yin side and just get on with it.

In programming, Yang is the Creative phase: the time when you get your hands on that keyboard and start making things happen. It can be new functionality, but it can also be re-designs, refactorings and debugging. You kind of *buzz* when you do this sort of thing. Fly by the seat of your pants. Shoot from the hip. And other well-worn cliches.

On the other hand, Yin is the Tidying up phase: the time when you read through designs, if you're doing design-up-front stuff, or walk though code, or check through test scripts to make sure you are being thorough. It's when you pick up your note book with that list of all the things you need to sort out, which you've been putting off for a while, and start ticking them off.

The most important Yin activity, however, is providing Quality Assurance, rejecting all the code which is not clear or succinct, even if it "works".

To a Yang person, Yin removes the "bu" from Yang's buzz and leaves "zzzzzz". For a Yin person, Yang is just crazy hackery.

Yang and Yin is a little bit like the labels "doers" and "finishers"- but not quite. First of all, you need Yin and Yang all the way through a project. Secondly there is a suggestion that Yang does all the work and Yin tidies up, whereas it could be the Yin doing all the work and the Yang just producing disposable prototypes.

Both activities are necessary. In my experience Yang tends to precede Yin, which is why I've called it YangYin, as periods of creativity are followed by consolidation. Programming teams need a mixture.

The reason teams aren't properly balanced is generally down to the Development Manager or Team Leader that's in charge. These guys inevitably tend to favour either Yin or Yang programmers and if not careful will recruit invariably the one type. Since, in true Yin Yang style, the two programming forces are opposite it's very easy for the Yin (or Yang) Manager to think that the Yang (or Yin) programmers are useless and not want to touch them with a barge-pole.

Too much Yin means nothing gets done (short of a mountain of paperwork). Too much Yang means nothing ever works.

Richard