Agile Documentation

The author periodically meets top-level managers, professors, and team leaders, who claim that the Agile mindset exempts any documentation, except for the content of the task tracking system.

Luckily, there are also many with the opposite opinion among business owners, certified Agile professionals, and other business-oriented professionals.

In short

Many are messing up different types of documentation. Initial requirements and living product design are not a waste. Duplicating actual code in words and diagrams as a standalone document, or even in code comments, is a waste, for sure. There is always an exception, like proper documentation of a sketch or temporary demo is a waste. A Product’s public API requires documentation in code to auto-generate a nice manual. However, user documentation is a separate discussion aside from this article.

In any case, if a product has proper Quality Assurance, it must have consolidated refined requirements and product design materials for UI, public and system protocols, public and system data structure and file formats, etc.
Any critical system must have well-defined and validated models for Security, Performance, and Throughput at a minimum.

Despite acting Agile, a mature business must think about traceability and business continuation in the event of “the team” vanishing for any reason. This directly derives from well-known international standards, which are modern and also Agile by design.

Retrospection of Agile deviation

Let’s start from the Agile Manifesto, which states “comprehensive documentation”. Its meaning becomes clear from the Twelve Principles:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Yes, “development team”, not QA, not Operations, and not even business people. So, the main pain point was detailed functional specifications of the era when it was really faster and cheaper to write and plan ahead than to implement that iteratively on ancient hardware with a deficit of allocated time for a programmer.

Before the Manifesto was published in 2001, “Extreme Programming Explained” (1999) had already been written, and then the second edition cleared up any questions in 2004 (ISBN: 978–0321278654):

Extensive internal documentation of software is an example of a practice that violates mutual benefit. I am supposed to slow down my development considerably so some unknown person in a potential future will have an easier time maintaining this code.
…
Software development is full of the waste of overproduction: fat requirements documents that rapidly grow obsolete; … While all of these activities are important to software development, we need to use their output immediately in order to get the feedback we need to eliminate waste. Requirements gathering, for instance, will not improve by having ever more elaborate requirements-gathering processes but by shortening the path between the production of requirements detail and the deployment of the software specified. Using requirements detail immediately implies that requirements gathering isn’t a phase that produces a static document; but an activity producing detail, just before it is needed, throughout development.
…
If you just stop writing documentation and use XP as an excuse, you will be called on your behavior by the community. Belligerently saying, “We don’t have to write documentation because we’re extreme,” shows contempt for communication, not an embracing of communication as a value.

This just nails what Kent Beck meant, and confirms the assumption above. To be sure, let’s look at other authors’ opinions in “Lean Software Development: An Agile Toolkit” (2003, ISBN: 978–0321150783):

Do you ever ask, Is all that paperwork really necessary? … Consider writing customer tests instead of requirements. In general, delay documenting the details of desired features until the iteration in which they are implemented.
…
Let’s assume that at the beginning of the iteration, the customers or customer representatives write down descriptions of features they need on index cards. There are many other ways to document what the customers want, but index cards are a lot like kanban cards, so for now we will assume index cards.

Here we finally come to the “point of the sabotage”. The Poppendecks actually suggested to optimize and postpone requirements document creation after the fact of related feature implementation, while capturing details in the kanban system as the first step. Apparently, that quickly turned into a “we do not need documentation” movement because a chaotic pile of cards is seen as sufficient.

What we are trying to solve

With this article, the author shows a different angle to the problem, especially nowadays, when technical features allow simultaneous working on documents and maintaining true live information just-in-time.

Typical issues:

1. Large systems waste the effort of each individual developer to reverse engineer requirements, raising the bar for entry of new team members, and harming shared code practice.
2. Incorrect individual reverse engineering of requirements leads to multiple new bugs for a single change.
3. Non-developer persons have difficulties participating in product discussions while forcing their inappropriate decisions.
4. Quality Assurance cannot properly validate requirements and changes according to the best practices of the industry.
5. Lost knowledge due to staff turnover.
6. Forgotten knowledge due to the high cost of context switching between projects and in stop-start projects.
7. Inability to oversee and assess interference with existing or planned features at communication time.
8. Jumping through tickets to acquire a holistic view is inefficient and requires a human-managed index, which would always be outdated.

The facts:

• Requirements are still written somewhere — this process can be adjusted.
• Despite the exalted values of Agile, there are still many team members who want to work on prepared tasks instead of defining them.
• It is naive and utopic to think in “small teams” and “always available” paradigm.

Solution: Agile Documentation

Main concepts:

1. Define a global reference identifier system for each document, section, and individual items. Fixed, self-explanatory, comes from standards’ compliance, best practices, and practical considerations.
2. Shared access, concurrent editing as feasible. Any general online document editing or wiki system fits the purpose. Convenient integration with a task tracking system is a huge bonus. A specialized requirements management system with a holistic linear view is a fantastic choice.
3. Each item should have:
   a) unique reference identifier;
   b) short title — updatable;
   c) status(es): “Draft”, “Done in <Version>”, “Rejected”, “Removed in <Version>”, other applicable;
   d) arbitrary details as applicable;
   e) author(s);
   f) references to task items;
   g) unified color coding of text labels like orange for Drafts, green for Done items, and red for Removed/Rejected items.
4. Tasks track activities, but do not serve as a source of requirements. This is a separation of concerns and elimination of waste of an extra process for maintaining documentation, which would never be actual and up-to-date otherwise.

Case of a new feature

Normal flow:

1. Discuss with stakeholders.
2. Capture the requirements and designs in collaboration using real-time editing and/or shared screen inside the live document as Draft.
3. Create a task for implementation referencing the document items and cross-referencing in the document.
4. Create any sub-tasks for dependencies like UI design or architectural spike deliverables.
5. Update the documented items with findings and deviations during implementation.
6. Before testing by QA, mark as Done in a specific release (automate as much as possible).

If at any stage after creating a requirement it becomes abandoned and/or rejected, still leave that intact, just mark it so. The main text can be grayed and struck out. The team or business may still return to the idea. If the idea is a dead-end, then it still must be documented to prevent going in circles and wasting the effort of other team members.

The feature requirement description can be as simple as a user or a job story, or quite complex. A Software Architect should help the team to properly split complex features into smaller, more manageable individual requirements. As applicable, it is desired to keep structured discussions and to attach or link their protocol in any form to the requirement. That aids future archeological activities.

The QA team should be involved as early as possible to validate the requirement itself and its fitness in the holistic scope. Otherwise, the requirement may still need to be further refined during validation of the task. In the worst case, the task may need to be re-worked.

Unlike tasks, which can easily jump from one planned version to another, the document is updated by the fact of a change.

Case of a feature elimination

1. Discuss elimination.
2. Create a task, cross-reference in the document.
3. Remove functionality.
4. Mark Removed in specific release in the document.
5. QA performs regression testing for other features.

Case of a feature update:

1. Discuss with stakeholders.
2. Update the requirements in collaboration, keeping both the existing behavior and the new behavior, clearly marking.
3. Create a related task and any required sub-tasks.
4. Refine as applicable during implementation, as with a new task.
5. Before testing by QA, merge the changes in the live document, clearly marking original features, new features since the specific version, and removed parts.

The format of updated features can vary and depends on the significance of the change. For the entire redesign, it is advised to mark the existing feature as removed and to add new requirements. That would reflect what actually happens. If the requirement is extended, then the extended part can be marked with “[Since <Version>]”. Similar for the removed part. In case of minor but significant updates for users, these can be marked in the form of a note.

A specialized requirements management system should mark changes more conveniently.

Capturing of legacy code requirements

There are situations when existing code becomes such spaghetti because of mis-Agile development that it either requires a deep refactoring or a complete rewrite from scratch. Sometimes, that may also be required for a technological stack change. Therefore, any existing unit tests may not be helpful at all.

Reverse engineering of requirements can be done in a rapid manner by team members designated for that. For such items, the authorship can be marked as “AS-IS” to minimize effort needed without digging into historical tasks, if they are traceable and/or available at all. Then, such requirements can be reviewed with stakeholders to identify candidates for elimination or refinement for a new implementation.

Skipping such a step is a guaranteed disaster for any mature project in production.

Conclusion

There is no way that Agile processes were meant to remove critical input documents with requirements and other design materials. The information was continued to be captured in tasks, but it became inconvenient for processing context on a holistic level. Agile is meant to view projects at that level, but not as individual tasks or features.

Therefore, it is essential to return to the benefits of historical requirements management approaches. That does not require new activities, but rather just adjusting what many already do with individual tasks. As soon as the mindset gets shifted, it is easy to add plugins/apps for automation to existing task and document management systems, and even create tailored systems for that.

Ultimately, collaboration for discussions of new and existing features can have a living requirements document as a metaphor of the existing software in a form that is readable and understandable to all stakeholders.