To win a hackathon, being Agile 2.0 is key
HOLLYWOOD, CA (GoshRobin) 2022/11/4 – To win in any competitive engineering contest, where time and money are dear, first and foremost we care about engineering process. Winning may be our goal, but the point of entering a hackathon is to learn new ways and processes.
We must resist our first temptation, driven by an aggressive deadline, to jump right into it without considering the overall efficiency of our approach. Not let our process decision be no decision. Not doing out of habit whatever we’ve always done, without even thinking.
Let’s look through the lens of history to understand the evolution of the software development process to where we are today. First came Chaos.
Chaos Theory, No, Not That Chaos Theory
To paraphrase Genesis… In the beginning, software was without form. Void and darkness was upon the face of software development.
The Manhattan Project was the first time anyone used IBM computers in a big serious way. The lead programmer decided it would be really cool to write some mathematical function that would calculate tables of tangents or something like that. Interesting, but useless to the Manhattan Project.
The programmer was off on a caprice instead of doing his work calculating the energy yield of atomic fission. He had found a cute computer science problem to work on and was off. That was only the first time that anybody did something like that. Computers can be fun. Distraction kept happening. It was chaos. Software project failure caused by the distraction of something shiny. Software development lost focus, went over budget, had delays, would get canceled and written off.
Eventually, the government said we can’t have this anymore. We must have some kind of structure and order. Make programmers work on what they’re supposed to be working on, not doing whatever random thing interests them. In battle, if our software is wrong, or non-existent for being late, that hurts our people. Bad safety-critical software can injure or kill us.
Waterfall Saves the Day, Well Sometimes
The government decided, we’ll require a top-down management system. Because we’re the government, and that’s how we do things. The government being the Department of Defense, the military. Top-down is how we go. Just follow orders.
This top-down software process would evolve into a set of rules of project management known as Waterfall. In this system, we have a leader at the top. He, or rarely she, is deciding what happens in an orderly way. Figuring out the tasks to be done, assigning the roles, and delegating sub-tasks to sub-leaders under them. It’s very much a military chain of command. Orders cascade from the top like a Waterfall.
Waterfall seems better than Chaos. So, everyone did that for a while. What we found was that some Waterfall projects, particularly those with outstanding leaders, succeeded. However, many Waterfall projects went chronically over budget, both in time and money. Bad leadership or just the bureaucracy of Waterfall can be crushing. By the time the leader communicates an order all the way down the chain, and hopefully back up again to hear the result, much time has elapsed.
So many big Waterfall projects were failing, in such spectacular fashion. The wrong orders. Miscommunication. Errors being made. Rework. Entire projects failing to deliver anything.
Waterfall, which was expected to solve everything with its rigorous top-down methodology, could be as much of a disaster as Chaos. At the end of the day, people thought, I remember the good ol’ days of Chaos. At least then, it was fun working here. Creatives resented the straight-jacket thinking of Waterfall rules.
Agile Emerges from Open Source
A new system evolved from the Open Source software world. Out of Open Source programming came this concept of Agile. I was a leader in this movement, and spoke at computer conferences promoting it in the 1990s. I didn’t write a book on it, so am not as well known as those who wrote books. Well known Agile proponents like Grady Booch and others on the Agile lecture circuit with me.
The reason that Agile grew out of Open Source is its capability to operate leaderless. In an Open Source project, even if you have a leader, she can’t really lead anyone. Open Source developers are volunteers who do what they feel like working on. An Open Source project manager can’t allow the process to be Chaos, because we won’t complete the project. We can’t do Waterfall either. No one is willing to take orders when we’re all volunteers.
Not Chaos. Not Waterfall. So, now what?
Agile 1.0 Scrum
A scrum is a sort of huddle in sports parlance. The idea is we share our thoughts on IRC or Slack channels, where it doesn’t matter anyone’s time zone. Open Source programmers may be all over the world. As video teleconferencing became available, some added daily scrum meetings, where we gather everybody together at the same time. As we hash ideas, with colleagues constantly talking to each other, a decision consensus and teamwork may emerge.
Linux got built this way. Linux is a humongous engineering project. Git also got built this way. A lot of things that are our software infrastructure today got built with Agile.
Agile is an improvement. We’re getting better at creating software. We started with Chaos. We tried Waterfall. While Waterfall succeeded for many projects, for a great number of other projects Waterfall was a total disaster. Waterfall was inconsistent. Very dependent upon who is the leader.
Then came Agile Scrum. Many Agile projects failed on the vine. And surprising to Waterfall proponents, Agile leaders said that’s good, let’s see if we can fail early. Because if an effort is going to fail anyway, we would rather know it now. With Waterfall, we wouldn’t find out until the last integration step that the whole design can’t work. Waterfall could waste 18 months of development time, when we could have tried a different design. So we have this “fail early” mentality that emerged out of Open Source and became Agile.
Outsourcing Breaks Agile 1.0, Waterfall Pushes Back
Then outsourcing came in. Outside of Open Source, companies resisted abandoning Waterfall because Agile requires trust in the team while Waterfall only trusts the leader. Many leaders are uncomfortable trusting their team.
With offshoring, leaders had to lead teams all over the world. Project managers would do a full day of work in America, then stay up late or arise early to boss teams in India or China. Burnout was inevitable. Waterfall failures increased over time with outsource projects. Leaders who had kept up a death march pace at first, fell apart, became apathetic. Offshore teams, often new to programming and writing buggy code, disintegrated into Chaos.
The business people that were hiring offshore teams mostly believed in Waterfall. It was all they knew. They had not come from the Open Source community. At systems integrators and consulting firms, Waterfall was still king. Trying to manage internationally with Waterfall didn’t work. But then, they could argue Agile Scrum didn’t always work either. It didn’t help that there were those who wanted Agile to fail, so everyone could go back to normal, to Waterfall.
Agile 1.0 Side Effects
Agile was supposed to be the cure, but this cure had unpleasant side effects on scrum. We wanted everybody on a daily video scrum call across all time zones. It may be noon here, but it’s midnight in Beijing. Who will we make suffer from lack of sleep? Some managers were democratic, said let’s make everyone equally suffer by picking the time that is equally inconvenient to everyone. In which case, everyone gets tired on the team. Or, the American managers might say, let’s have it be at our convenience, when’s best for me. Not worry about the offshore team becoming sleep-deprived.
Zombies or walking wounded, either way, the team is making silly sleepy mistakes. That’s not good.
Agile 2.0 Balanced Team
Bugzilla, an Open Source bug tracking system, had been developed where anybody could post a bug on an Open Source project and then anybody could fix it. Open Source needed bug tracking because Open Source projects were full of bugs created by student programmers. Our programmers are in different time zones spread out all over the world, are volunteers who resist fixing their own bugs, and we don’t even have a project leader in some cases. How can we handle bugs? We ticket them Bugzilla or a more modern equivalent such as Zira or Gitlab Issues.
What if we ran our project around trouble tickets instead of scrums?
The Agile Balanced Team concept is that as with bugs, we can develop features using the same ticketing process. Have people randomly deciding on their own to be self-actualized. Someone may think, “I know how to fix that bug,” or “I don’t have any idea how to fix that bug, but I’ll just pick it because I want to learn how to fix bugs.” People assign themselves do stuff, and may get other people to help them. The bug gets fixed. Excellent. What if we do that with features?
Are new product features really so different than bugs? In fact, in many cases, when we file bug report we may be reporting that something is missing. A bug that’s a missing feature only seems like a bug. Why not post feature requests just like bugs? Then, have programmers assign features to do to themselves.
Agile Balanced Team can be incredibly rapid. If people will embrace it. Because unnecessary delays and extraneous decision-making are avoided. I can post a bug, go to sleep and in the morning someone in another time zone has already fixed it.
Whereas if I did this bug fix with Chaos, I would just start working on it. I’d be totally distracted by it. Not doing what I’m supposed to be doing as project manager. If I did this with Waterfall, I’d first consider the skills of everybody on my team. I’d think, “who’s an expert in fixing this?” Maybe I only think Kayla has done something like this before. Maybe I’ve got that wrong. Maybe it was actually Nitin who did this before. But I think it’s Kayla, so I’m going to assign it to Kayla.
Tomorrow, when Kayla wakes up, she emails or chats me with, “Nitin worked on this before, wasn’t me”. And I’ll reply with, “Oh, Ok. Sorry”. Next, I write a note to Nitin, “Will you please fix this bug?” The next day, Nitin writes back to say, “I would but I’m doing this other urgent thing so you may want to ask someone else if you expect it soon”. I try somebody else, and four days later I finally have someone assigned to start work on this bug.
Obviously, this top-down decision-maker process is much less Agile than if the right person would simply select themselves. Do that, and we would already have the task done in the time it took to assign it. It can take an incredible amount of effort for a manager to track down the right person, to assign tasks. Self-assignment, people just do it. It’s magic.
Software Process Evolution, Increasing Efficiency
Software process evolved from Chaos, to Waterfall, to Agile 1.0 Scrum to Agile. 2.0 Balanced Team where we are today.
A challenge facing any engineering team is no one on the team may know anything about the different development process. Or care. They’ve never heard of it before. Wasn’t taught to them in school. No manager has ever explained it before. Why bother? They say, “Well, I’ve always done it my way, and I’m still here, so I won’t change anything, because this will be something way to different and difficult for me to learn.”
In a hackathon, we only have a few days to complete a project, from concept definition to completed code to demo presentation. Chaos does sometimes win hackathons. To succeed that way means one person did almost all the work, and everyone gets the credit for it. Trying to Waterfall a hackathon is a guaranteed fail. Waterfall won’t even get everyone assigned before the project is over.
How to Win
To win in a competitive engineering endeavor, we want the fastest process that gets results. Today, that’s Agile Balanced Team. In the future, AI will help us write software and our engineering process will evolve again to accommodate advancements that increase efficiency.