To document or not to document; that is the question

Whether 'tis nobler in the company to suffer 

The numbers and writing of outrageously large reports

Or to take arms against the sea of papers

And by opposing, go Agile?

    I'm sure most people have been confronted with the problem of lengthy documentation that no one reads versus cutting out documentation so that no one knows what's really going on.  To give my readers some background though, there is bit of a struggle on how to move on a project with the two forces (they sometimes oppose each other but they don't need to) being:

1.  Traditional Waterfall Methodology, which encompasses lengthy planning, execution, documentation, reporting, and structure.  Generally the tasks will look like a waterfall on the Gantt chart, hence the name.

2.  Agile methodology, captured in programs such as Extreme Programming, Scrum and Kanban.  This takes a much more adaptive approach and responds much quicker to change.  It is generally characterized by much less documentation, much shorter iterations, and much more focus on team collaboration and accountability.

So, documentation.  This is the rough breakdown of the arguments put forward

1.  Supporting Waterfall

• Requirements are VERY traceable
• Planning is documented with all factors kept in mind
• Covering your back is easier
• Progress is much more straightforward and easier to measure
• The entire process is under much more control and can quickly detect problems in scope, time , cost, etc.

2.  Supporting Agile

• The documentation is short enough that people will actually READ it and know what's inside
• Conversations and working code are emphasized more than comprehensive documentation(see Agile Manifesto)
• Having less documentation makes it easier to respond to change, saves time

Notice first that these arguments may have some flaws in them and second that they both address needs of different bodies.  Medical and legal implementations obviously need to be better documented than other fields.

Let me share you some experiences from both worlds.  In a course in project management, we were asked to split into teams and create bids for improving an airport.  Each team turned in a recommendation and supporting documentation of some 20 - 50 pages.  Who won?  My team won for the very simple reason that I summed up all 20 -50 pages of work into one summary paragraph at the top.  The instructor read the paragraph, skimmed the rest to see it was there, and gave it to us.  He didn't even read all the proofs and whatnots!
Second experience, in agile this time:  I was assigned in a company to work on some schedule management stories.  They looked like they had been dropped for a year and were coming up again.  I looked at the progress that was done.  All I could find were some mocks.  I asked an agile leader.  Blank stare.  No one even remembered what the tasks were about!

I'm not going to get into a lengthy solution today, but I do want to show that while both have good points, improperly managed, they can both be disastrous.

What I do want to propose is this.  Documentation should be integrated in, not apart from, tasks as much as possible.  QA is probably the most advanced in this sort of thinking.  For example, they will automate tests and use the automation itself to document and show what the code is supposed to do.  Simple idea but it combines a host of ideas together such as agile, TDD, management of technical debt, etc into a simple way of documenting and maintaining what the code actually does.
While this is the golden example, and may not be possible to this extent in many situations, the idea should be that by the very structure of the program you are presenting you should have documentation built in.  Just like a design artist wants the user to not even be aware of the work of navigating, a Project manager should, to his fullest extent, have stakeholders not even aware of the work of digging up numbers, reports, etc.

We'll talk more on this later and suggest some practical implementations.