Measuring Agile Software Productivity

Outdated Waterfall metrics still in wide use over 20 years later

Girl with ComputerHOLLYWOOD, CA (GoshRobin) 2022/08/17 – Back in the 1990s, I was on the lecture circuit with Grady Booch, Kent Beck and other best practices code evangelists who spoke of a new rapid software development process. In 2001 that process would become known as Agile. The Agile software development process was much inspired by Linux and other successful open source projects that ignored traditional Waterfall practices.

To the shock of Waterfall managers, Agile developers often write code with incomplete or no engineering specifications. It’s not that producing specifications is forbidden. The point is to quickly build a small Proof of Concept (PoC) prototype to test with users. Avoid writing extensive documentation, which may be wrongheaded, before beginning testing for customer satisfaction.

When a PoC is successful it may evolve, or is often thrown away to start over, in order to create a Minimum Viable Product (MVP). An MVP is the smallest subset of features to make a useful program, to create a platform that quickly gets into the hands of users. This contrasts with Waterfall methodology, that hopes to anticipate and completely specify a finished product before the first line of code is written.

When it comes to measuring projects using Waterfall or Agile methodology, the metrics differ.

Waterfall Metrics

  • Meetings
  • Reports
  • Specifications
  • KLOC

The primary metrics for Waterfall are effort.

It may be assumed that if we have enough meetings, reports and a precise plan, that what will result is good product. KLOC, or thousands of lines of code, becomes a metric once code development eventually gets underway. The architect may use Function Point Analysis or other estimation methods to plan how many lines of code we will have. Progress is expected to proceed until 100% KLOC is reached when the software may be released.

Implementing Waterfall may be thought of like filling a bucket. If we estimate how big the bucket is and how much water we’re pouring, it will fill, right? Not if our bucket has a hole in it. With Waterfall, too often the specifications were wrong.

Agile Metrics

  • Commits
  • Tickets
  • Releases
  • Users

The primary metrics for Agile are not efforts, but results.

Git Commits

Requiring all developers to check in code changes at least as often as the end of their day is a typical git strategy.

Agile code gets written right away and committed to a git repo daily. Git is a content management system invented by Linus Torvolds of Linux fame. Some older version control systems are still in use (Bitbucket, Subversion, CVS), but git dominates. Linus not only pioneered Linux and Git, but also Internet dating. The woman who became his wife asked Linus for a date via email.

Branching, or Not

Developers may work on separate git code branches to merge later, or on the same main branch. If a project does much branching, a build master may have to be assigned to manage the merges and rectify merge collisions where two programmers changed the same line of code. Smaller teams without a build master often work in the same branch. They may use CI/CD automation to check nightly builds will still compile and pass unit tests.

Ticketing

JIRA is a popular ticketing system for tracking software issues. A ticket may hold a bug report or a feature request. Ideally, git commits are labelled to refer to the ticket they address. Although JIRA is in wide use, and so are other bug trackers such as the ancient Bugzilla system, it isn’t necessary to get a separate ticketing system. Github and Gitlab offer ticketing systems built into the repo software.

With Agile, a developer writes a ticket. It may be assigned to a developer, or if the Agile process is Balanced Team, any developer may pick which ticket to work on, may self-assign. After the developer makes the bug fix or feature enhancement, she or he checks in the code with a git commit. Then closes the ticket as having been resolved.

These code commits and issues tickets are the measurable productivity of an Agile programmer. Agile managers may expect at least one ticket or code commit will get done each day by each programmer. A programmer who produces no tickets or commits during a work day may be presumed to be on vacation.

MR2 Bug Reporting: Minimum, Reproducible, Readable

A bug report should not be an email or chat message that merely says, “It didn’t work for me.” It should be a ticket and should be complete, that is, be a Minimum Reproducible and Readable (MR2). For our own project, we will rarely create example code, because we already have the code in front of us in git. However, if we are so stuck that we would ask for help on StackOverflow, example code will be necessary.

When writing a bug report, the report should be clearly named and specify the steps to repeat the bug so any programmer can find and work on it. If a bug happens at random, it should still be reported, with note that it is a random crash that can’t be consistently repeated. In tough cases like this, the bug report should include any information that might be relevant to track down the issue, such as hardware platform or context of when it happens.

Here’s an example of a bug report that isn’t MR2 compliant. The subject, C2065 : After building Zoom+ repos from Gitlab , encountered attached issue, is both too vague and too specific. It has a number of steps, like installing the Visual Studio compiler or that this is a Zoom+, not Minimum.

Unnecessary screenshot in bug report

Unnecessary screenshot in bug report

The screenshot of the compilation error shows what the error is, but that info isn’t in the text of the bug report. We wouldn’t include a screenshot typically, unless to show a visual GUI  bug, and it would be a screenshot of our app misbehaving.

This bug report is for the same issue, but more specific and concise.

Do We Have Closure?

A problem with our first bug report was closure. The developer closed the issue, marked it as fixed, without fixing anything. Unfortunately, there’s no way to block this happening in Gitlab, to enable only a project owner or maintainer to close issues. What should happen is the developer signs off, then QA will close the issue after testing.

Using Waterfall Metrics with Agile

As with Waterfall, an Agile manager may choose to write specifications or to track KLOC. What is important to note is that writing specifications doesn’t count as Agile progress unless ticketed. KLOC doesn’t count either until the code is committed.

Why isn’t KLOC a great metric compared to commits? Consider IBM OS/2.

OS/2

During the development of OS/2 by Microsoft, IBM was measuring progress with KLOC. Because OS/2 was performance-bound, Microsoft had to start paying attention to doing code refactoring even before OS/2 was completed. During refactoring, inefficient code would be rewritten to be tighter and redundant code could be eliminated entirely. The result, better software with fewer lines of code.

Reducing KLOC during refactoring was judged by IBM as negative productivity. IBM was measuring only the quantity and not the quality of code. IBM directed Microsoft to stop doing refactoring.

OS/2 never became performant, was a flop in the market in part because it was big and slow. OS/2 was never made open source, was abandoned. However, the IBM JFS file system of OS/2 became open source and is part of Linux.

Resistance to Embracing Agile

Many developers are comfortable in Waterfall methodology, or prefer to have no methodology or metrics at all. They may be beyond reluctant, may view Agile with alarm. Tickets and commits are metrics that can measure programmer productivity on a daily basis. It’s like the leaderboard in a video game. It is easy to see who is a star.

Programmers accustomed to being admired for expending heroic effort or for talking a good game in meetings can feel cramped and uncomfortable in this new world where their effort counts for nothing, that only results matter.

As I have done so many times before, I launched a new software development team recently. I need to convince them to embrace Agile.

In Outlook, I scheduled a separate half-hour Teams meeting to brief each developer on my team, what I expect and to explain Agile metrics. When one has a mix of female and male programmers. or senior and junior programmers, there might be a power dynamic in the group. If one person dominates it can prevent building good 1:1 rapport. I met separately with each developer.

So, what’s happening with my Agile initiative? A week later, no tickets and no code commits. I have a potential mutiny?

Resistance may be expected when pushing others out of their comfort zone. We are not going to panic or get angry. A way forward is to make everyone feel more comfortable, without accommodating their resistance to change. We will explain again, and explain better, what is expected.

Sabotage of change can be expected from those who fear change absolutely. It is often someone senior who fears change most. If the system they currently rule changes to new methods, they may lose power, or even be replaced.

We can identify who is resisting change out of fear. They want to equate change with risk. They will say, let’s not risk changing anything, assume inaction is always the best course. Of course, in real life, taking decisive corrective action is often necessary to prevent disaster.

About Robin Rowe

Robin Rowe

I’m Robin Rowe. People call me Robin or Rob, sometimes professor. I work in advanced product design and innovation management.

As UN WHO Augmented Reality Group Manager and XR Games Producer, I developed a medical metaverse, an AR/VR hospital simulation to train doctors worldwide to save lives.

As chief technologist at multi-billion dollar systems integrator SAIC, I was the founding director of their AI research lab and an enterprise manager with P&L responsibility for commercial and defense divisions. I constructed the Chicago NBC-TV studios, advanced national critical infrastructure by adding Smart Cities capability to the U.S. traffic control system, created real-time AI for DoD national defense crisis management, and developed the real-time motion-capture animation system used to create digital stunt doubles in superhero films such as Disney Marvel Spider-Man.

I led the team that won the Novartis Biome innovation prize in 2019, using AI computer vision to analyze skin disease. I’ve taught C++ as a Computer Science professor at the Naval Postgraduate School and the University of Washington. Former DARPA PI for AI and digital video. Former navy research scientist for VR war gaming and vision research. I’ve chaired innovation committees at ANSI/ISO and The CFO Alliance. I’ve developed financial software for Fortune 500 companies and large NGOs. I’m a thesis advisor to graduate and doctoral students in metaverse R&D.

I founded my first start-up, a car company, when I was 16. My family is in real estate and agriculture, owns the largest organic farm in Illinois. Not wishing to run the family business, I went another way. I have 30 years experience in product design and financial systems. And, with trading stocks and now crypto on my own account, I’ve made high ROI year over year.