I have read a number of studies that take a look at different careers and how stress accompanies many professions. Air traffic controller? Lots of lives in the hands of people who must think and act quickly in all manner of situations. If job one is to avoid collisions with the ground and any other thing airborne, I can imagine significant stress. First responders? Prepared to risk life and limb to carry me out of a burning building. Dedicated to resuscitate me in dire health. Will place themselves between criminals and the innocent. Lots of ways stress is present every day. Military volunteers for their country? Enough said.
Software engineering typically resides in the list somewhere a bit more stressful than things like actuary, professor and librarian. So why would fear be a factor in a field that works in a safe environment, contains plenty of time to do thoughtful work, and is well-compensated? Part of the issue is a perception of the stresses in the career.
Uncertainty is a word I would use to describe new software development. Here are some of the mysteries of software development that can be additive and compounding. Let’s compare some of the common steps in a software engineering project:
- The concept of an automated system that will solve a business problem is proposed.
- For an engineer, a concept is not a concrete thing that can be understood accurately for delivery.
- The concept, however, demands a high-level estimate for leadership. Engineers provide an estimate with uncertainty.
- Who will participate in the project and what is the level of expertise for the key contributors? Will the team be an all-star team or be some eager but green engineers?
- The newest software engineering projects may contain some element of new technology. Can engineers predict just how that new technology will impact an estimate? “Never used the software yet” adds a big uncertainty.
- At some point on the project timeline, the more detailed business, user experience, interfaces, data definitions and non-functional requirements will begin to appear. If we are in an Agile-style project, the requirements may not become concrete until many sprints down the line.
So far, these steps are fairly normal. Engineers can communicate with leadership that a lack of requirements, the mystery of new technology, and the number and type of engineers add up to a significant variables to solve in the equation that should yield a single integer – how many days until I receive this work?
A friend gave me this interesting analogy. Based on concept, could I predict how complicated the requirements will be half the time? Ok, as an experienced engineer, perhaps. Could I predict how long it will take our engineers to get good at a new tech half of the time? Well, I don’t have experience with that new tech, but I could get close. Could I guess which experts on my team will be available for the important project half the time? Perhaps. Can I guess and how many others will be available half the time? Maybe.
If these are the factors of getting reasonably close in the high-level estimate, then the number of times my integer for the high level estimate is “close” to actual is 16 to 1. I propose that even the most learned engineers at the time of the high level estimate are going to be “off” by some significantly measurable amount much of the time.
As an IT leader, I know that many of my peers in leadership know that the high-level estimate is likely to be far off. Our safety net? The Agile process. Here is an over-simplified description of how this works. The high-level estimate says 4 months for the project, but the Agile team will gather stories with detailed requirements, estimate the difficulty of that story, and assign the story to engineers and quality assurance to work through the task in an iteration. At the end of the iteration, we add up all of the points of all completed stories and get a “velocity.” After a few iterations, we can get an average velocity of points the team is able to complete. Once all stories have been assigned estimated points, the project manager can do the math. If our team remains together, we expect the velocity to continue until we exhaust story points.
Sounds quite reasonable. Here is the problem. When leaders, sponsors or outside clients are planning, they are making decisions based on the value of the automation result. If well done, there is a strategic advantage to the business that turns into some dollars and cents that will be saved or gained when the application is delivered. Leaders have to determine if we should build that app, and a vital lever in making that decision is the high-level estimate. What is the cost to build this app? The high level estimate says 4 months at some amount of hours and that equates to some dollar amount with zeros behind it.
The sponsors and IT can agree on a statement of work that is intended as a guideline, but the uncertainty of cost is a real problem. Let’s say the 15 of 16 times that the high-level estimate is off, a majority of those will not be “less time” to deliver. Why is that? Why would nearly all estimates fall short of the real amount? Here is an opinion I share: when an experienced engineer estimates a project and correctly presents some of the possible uncertainties in a larger high-level estimate, that cost for the project can outweigh the proposed value that should be realized. Irony goes here. Leaders tend to challenge the high-level estimate that is large with the senior engineers. Why? The engineers admit that the estimate is likely to be way off. More than once, I have seen an executive chop a number of hours from the high-level to have it meet the value expected. Otherwise, IT would not be permitted to try the project. Or, the project may be given to some outside entity that feels more comfortable undercutting internal IT. Honestly, I cannot say that this is an incorrect approach. After all, the high-level estimate is likely off, and perhaps is over-inflated.
The feeling in the pit of the stomach for the people who will work on the effort is not a good one. I would describe the feeling as being similar to the one when we are on a large family vacation. My wife’s gang likes to invite a large branch of the family tree to rent a big house in the NC Outer Banks. Once we are all at the house for the first day, the ladies pile into the largest vehicle and make the big grocery shopping run. The men-folk are left with all children and no supervision. Right away, the party with the dads and kids begins. After all, the grocery shopping is likely to take at least 2 hours.
Now, this big load of supplies that will arrive at the house is really heavy. When the ladies return, they will be in need of some serious bag-carrying help and are likely very low on patience. Imagine me and the other guys hanging by the pool and having recreational beverages in hand when the gals need our muscle. I can guarantee that the last of the patience is exhausted.
It is this feeling of uncertainty that causes me to wait on the front porch of the house after the first hour, ready to alert the other men that the bosses have returned. Even though I could use some of the time to have a modest amount of fun, I find that I am totally preoccupied with the time that the big van will arrive. Since I am on the porch and can summon the guys in a moment’s notice, I expect a big “My hero” from the Mrs. Deb is as practical as always and skips that part, instead choosing “take all the cold stuff in first.” When all of the groceries are correctly moved and stashed, we can all relax, and the guys can resume recreational beveraging.
Like the dads in NC, our engineer team is not sure how bad this project will be will be when the van-load of actual requirements arrives.
When the ok is given to start the project and get started accumulating stories, start work, get an idea of velocity, the Agile leads begin to see the backlog of story points, the velocity of the team and the first factual metrics on a delivery. Many, many times, the project goes instant yellow because the velocity shows the date of an eventual full delivery is in the distant future and not near the deadline. What levers do we as leaders have? Adding resources does add cost, and the value of the project does not increase. Cutting scope can cut value from the eventual delivery.
This is where the uncertainty stress raises its ugly head. As a leader, I have one button to push. It is the dreaded “additional hours for IT professionals is an expectation of this career” speech – one that I hate to give. And, in order to hold the costs closer to the high-level estimate, I should not pile more work on the contract or hourly engineers. The added hours have to go to the exempt team.
Now we can see the stress. Since that high-level estimate will continue to be held as a cost high-water mark, I must ask portions of the team, those exempt, internal ones, to shoulder a disproportionate amount of work to try to keep a difficult and flawed commitment. As a leader, I feel my own disappointment. I truly believe that only the team can win. All must contribute in the right role and do their best work. Then I contradict my principle by treating teammates differently. Oh, and just to make matters worse, the exempt group with the most senior abilities are super-critical to getting the project done, and I have made them quite grouchy. I have put the company in danger of losing an All-Star or top contributor due to stress caused by uncertainty and a feeling of an unfair workplace.
What is my role as a leader in this case? I recommend a few tactics:
- Communicate early and often. I like to site actual requirements with an actual good-faith estimate. It is better for my team and my long-term reputation to raise that initial deadline as a big risk. Raising the risk early gives me more time to adjust.
- Can I bring in more talent to help meet the date? This does sound blatantly obvious, but I have led teams that dislike the uncertainty of having new talent “making a mess” in their applications. The challenge is two-fold. As a leader, I need to bring in talent that will truly help the team and have the ability to increase the Agile velocity.
- Propose a stepped-delivery with dropping the most value first. In any major systems, there are critical features and nice-to-have features. By delivering the most critical first, and negotiating later deliveries of less important features, can I help “make the initial date” with a minimum viable solution? It is a way to save face in IT and grant more time to work on the extra scope.
- Do not hit the “instant overtime” button. If I assign additional tasks to those exempt employees only, I will be in danger of losing those best resources. If I have to go to the extra hours well, I need to expect hourly, contract and exempt teammates to all contribute.
In my next article, I will share a method I use frequently to have senior leadership, sponsors and other decision-makers help engineers with the initial estimate. The trick is this. Answer estimation questions with questions, not estimates.
If this was the grocery-hauling example, the technique works this way. Deb asks me to “take in the cold stuff first,” and I answer “can you point at the first ones to take?” As with software estimating, guessing in grocery bag selection is a bad idea.