The average software developer reads less than one professional book per year (not including manuals) and subscribes to no professional magazines. These developers are not developing or advancing themselves professionally. About 75% of these people do not have a degree in computer science or a related field. They learn by trial-and-error and on-the-job training, which means that they risk learning other people’s bad habits rather than industry best practices. This method of professional development perpetuates ineffective, inefficient practices that hinder the success of software projects.

— Construx.com

Tell me something I didn’t know!

Thus spoke The Programmer.

I hold in my hand a 63-page requirements doc . . .

We spend a lot of time reworking the requirements doc to reflect the reality of the system that we’re actually building. We also spend a lot of time reworking all the docs that derive from the requirements doc — design docs, UI docs, test plans, etc. — to reflect the changes in the requirements doc.

Another way to think of this is that the project is driving the requirements, rather than the requirements driving the project.

So why did we create all these incredibly detailed documents?

We have a vast collection of well-documented ignorance . . .

Thus spoke The Programmer.

My first 10 years in the software business, I had great managers. They did the management thing and I did the programming thing and we got great results together.

Then, after the dot-com boom torpedoed industry hiring standards, I got tired of working for managers who should not have been allowed anywhere near a software project, people who were not fit to direct a professional software developer to a table at the Olive Garden, much less direct their activities on a complex project.

I couldn’t possibly have continued to work for people like that — it just made a mockery of all the work I’d done over the years to actually learn something — but I still miss being a developer . . .

Thus spoke The Programmer.

I work with a company that has the following set of milestones in its standard project methodology:

• Vision/Scope Complete
• Requirements Complete
• Design Complete
• Definition Complete
• Build Complete
• Test Complete
• Rollout Complete

I’ve noticed an interesting pattern at the weekly enterprise status meetings: a significant number of projects report being exactly on schedule for each milestone — not one single day ahead or behind! — until they get to rollout, at which point they suddenly go several months late.

Some things can be faked and some things can’t. As long as you have milestones that can be met simply by declaring them done, or by signing off on a document, you can always hit them on time.

But when it comes to putting actual working software in front of a customer, that’s when you really have to deliver the goods, and that’s when the milestones start getting missed.

This is a very high-risk approach to software projects. Deferring testing to the end of a project guarantees that if your project fails for any reason — and if your testing is honest, there’s always some non-zero probability that it will fail — you will have already invested in the entire cost of construction.

That’s why the history of software engineering is littered with big-ticket disasters. You never really know what you’ve got until the end, after you’ve spent all the money.

It’s also a good argument for iterative, incremental development. If you have to deliver working software early and often, you can’t fake it.

Thus spoke The Programmer.

An ironclad but widely ignored law of software development — you could probably substitute “product development” as well:

Customers never understand how something is going to work until they see it in action.

Thus spoke The Programmer.

IT has got to be the world’s most unprofessional profession . . .

I’m sitting through a dry run of a presentation for our CEO, explaining why the company’s flagship IT project has taken so long and cost so much, with no end in sight.

It seems that, compared with two years ago when the project started, headcount is up 60 percent, sales volume has more than doubled, the product mix has changed significantly, as a result of of which, requirements and assumptions have had to be constantly changed.

Of course! Two years ago, who could have anticipated that the business might look a lot different two years down the road?

If you said “Uh, pretty much anybody,” you’re right.

Thus spoke The Programmer.

When you’re meeting with the people responsible for a company’s software development process being the way it is, any suggestion that it could be improved is going to be a tough sell.

Prepare to hear something like this: “We didn’t get where we are today by doing things wrong.”

Thus spoke The Programmer.

Project X hit a milestone called Vision/Scope seven months ago, 99 days late. It’s 312 days late on the current milestone, which is called Definition.

To date, the project has consumed 36,000 labor hours — 18 person-years — and $2.5 million.

At this morning’s enterprise-level status meeting, it was decided that Project X will be put on indefinite hold, as it is no longer a strategic priority.

This reminded me a lot of an article I read a few days ago:

What the waterfall does well is to keep useless projects from resulting in useless code that needs to be maintained. I’m not sure if that’s the real purpose, but it’s certainly a great side benefit. It may sound inefficient to pay a lot of engineers to get started on projects, do a bunch of analysis and design, and finally abandon the whole thing when something else becomes a higher priority, but every line of code they don’t write is another line that can’t break!

OK . . . you could make a case that waterfall “worked” here — clearly if, after 18 years of effort, people can’t even define the project, that sounds like a project that has no chance of success and shouldn’t be attempted — but it worked at a cost of $2.5 million.

That doesn’t seem very efficient.

What I find is that if you put the customer, the technical team and other appropriate representatives together for as little as four to eight hours, à la a Sprint Planning Meeting, it should be obvious whether or not anyone understands the problem well enough to go ahead and attempt a software solution.

Thus spoke The Programmer.

This is an excerpt from a job posting for a Sr. IT Project Manager:

• Develop project plans and ensure that deadlines are met on time and projects are delivered within budget constraints. You will use standard project management tools to define requirements and track project status.

• Manage and prioritize projects for the division using standard project planning methods and software through all phases of the project/development lifecycle.

OK, let me get this straight: You want projects delivered on time and within budget — you don’t mention whether or not you want the software to actually work, but I assume you do — and you want it done with standard tools and standard methods.

It may have escaped your attention, but that is not a standard result. The standard IT project is either late or over budget or fails to meet customer expectations, or all three.

If it were possible to deliver high-quality software on time and within budget with standard tools and methods, then everybody would be doing it.

So you want someone to achieve exceptional results — non-standard results — with standard tools and methods. How is that supposed to happen?

I can deliver exceptional results, but you have to let me do it my own way. If I have to use the standard tools and methods, I’ll get the same crappy results everyone else gets.

Thus spoke The Programmer.

A family member had surgery recently and had to sign a consent form:

I have been advised that all surgery involves general risks, including but not limited to bleeding, infection, nerve or tissue damage and rarely, cardiac arrest, death or other serious bodily injury. I acknowledge that no guarantees or assurances have been made as to the results that may be obtained.

And so on . . . Don’t say you weren’t warned!

Medical professionals are very good at setting realistic expectations with the customer. Software professionals are not. We take customers into projects with glib assurances and wishful thinking.

If you work in corporate IT, as I do, your project risks don’t include death or bodily injury, but how may times have you heard a project manager say to a customer even something as simple as this:

This project — like all projects — has a number of possible outcomes, and not all of them are good. Let’s go over some of the more likely scenarios . . .

I don’t know if I’ve ever heard it, other than when I was saying it myself.

I’ve never seen a really good explanation for this . . .

Thus spoke The Programmer.

I swear to God this is a real email from a once-promising manager with degrees from Brown and Princeton, who recently accepted a new position as Chief of Staff to the CEO, and now uses her Ivy League education to put out emails like this:

Effective immediately please ensure that all written communications at [insert company name here] have a minimum font size of 12. In particular, [insert CEO's name here] has asked me to convey that he will be ‘throwing away’ any communication he receives (over email or on paper) that does not meet this criteria [sic].

Please call me with any questions or comments, and hope everyone has a great weekend!

I always say if you’re going to misuse the word “criteria,” at least do it in a highly readable 12-point Verdana font . . .

Thus spoke The Programmer.

It ain’t over till it’s over.

— Yogi Berra

A project manager reports that her project is “48 percent complete.” In terms of what, I wonder? Calendar time? Cost? Effort?

I know it’s not 48 percent complete in terms of functionality because there hasn’t been any working code delivered, just a bunch of documents.

One approach that makes sense to me is to express completion percentages in terms of implemented requirements.

For example, if you have 100 functional requirements, and 48 of them have been successfully implemented, then you’re 48 percent complete!

Actually, I oversimplified that a little . . .

All requirements are not created equal: Because some requirements cost more to implement than others, and some requirements have a greater business value than others, you could assign relative cost and relative value numbers to each requirement, and calculate completion percentages accordingly.

This is good both for measuring the value of work already completed, and for estimating time to completion on the work remaining.

Completion percentages continue to be one of the enduring fictions of our business. I wish I had a dollar for every time someone reported that work is 90 percent complete, it continued to be 90 percent complete until a week before the due date, at which time the date was pushed out another six months because nothing actually worked . . .

Thus spoke The Programmer.

Our Sr. EVP dropped by today for a meet and greet . . . he was 45 minutes late, and when he arrived, it was obvious he had no idea who he was talking to.

“Is this the IT group?” he asked.

It was explained to him that some of the people were from IT, but some were from the call center and tech support.

“And do they all report to you?” he asked the senior manager in the room.

Here’s a little trick I’ve picked up over the years: When you’re addressing a group of people, take a few minutes beforehand to learn who they are. It will make them feel less insignificant.

After this fiasco, he went off to a catered meeting with other highly compensated executives, and I went out to buy my own lunch.

Prediction: This meet and greet will be mentioned in at least two exit interviews in the not-too-distant future . . .

Thus spoke The Programmer.

More software projects have gone awry for lack of calendar time than for all other causes combined.

— Fred Brooks, The Mythical Man-Month

As a corollary to this, I’d say that lack of calendar time very often forces us to admit that our projects have gone awry.

Denial is a viable strategy when delivery dates are far in the future, but when the deadline is staring you right in the teeth, the time for sunny optimism is over and the time for the Day of Reckoning (DoR) meeting is at hand.

I attended one such DoR meeting yesterday afternoon . . .

This particular meeting broke down into a battle between the Designers and the Implementers. The Designers — who happen to be the more senior members of the team — felt that they had written the specs in such excruciating detail that the system should pretty much have coded itself from that point on, despite the relative inexperience of the Implementers.

The fact that this didn’t happen was frustrating to both factions. The Designers think the Implementers are incompetent; the Implementers think the Designers are arrogant.

Let me say here that if my life depended on delivering a project successfully, I would want to follow at least two simple rules, namely:

1. Keep it simple.
2. Deliver early and often.

At every DoR meeting I’ve ever been to — and I’ve been to quite a few over the years — one or both of these rules has been violated.

Yesterday was no exception.

Rule One violated

The company I’m working with is a believer in big upfront designs. On this project, the Designers were allowed to generate hundreds of pages of design docs, models, diagrams — then hand it off to the Implementers to implement.

The design is way beyond the ability of human beings to comprehend in its entirety. You’d have to be some kind of savant to keep that much information in your head and feel a sense of mastery of it. Software development is a technical process, but it’s also a people process, and you have to work within the limits of human comprehension.

Rule Two violated

You can’t really assess the efficacy of a design until you try to implement it — except to say that it’s almost certainly wrong in some respects — for the following reason:

If you were assigned to model an existing system of moderate complexity in perfect detail — well, I don’t know you, but I feel safe in saying that you probably couldn’t do it.

You’d get some things wrong, you’d leave some things out . . .

Now if you were assigned to model a system that hasn’t been built yet in perfect detail, just working from a set of requirements — well, that is an impossible task.

And even if you could do it, you’d find that in the course of building the system, forces beyond your control would inevitably lead to requirements changes, thus invalidating your model.

Bubbles and Lines

Bubbles and lines don’t crash.

— Bertrand Meyer

Nothing ever breaks during the design process. Bubbles and lines don’t crash — code crashes!

So if you defer the writing of code until the big upfront design is done, you wind up learning things late that you would have been much better off learning early, when you still had time to do something about them.

Again, if my life depended on a successful project delivery, I’d start with early delivery of a working system that did almost nothing, but had the basic infrastructure in place — the essential hardware, software and network connections — then start adding to it, one piece at a time.

Summary

1. Keep it simple.
2. Deliver early and often.

Thus spoke The Programmer.

Man plans, God laughs.

— Yiddish proverb

A VP has asked me to review a Microsoft Project schedule printed out on 16 legal-size pages. The first thing that jumps out at me is that the level of detail in the schedule far exceeds the quality of information we have about the state of the world and the project at the future dates and times represented.

I don’t see how you can break tasks down to this level of detail, add in dependencies, and state that Task XYZ is going to start at 2 o’clock in the afternoon on some date 14 months from now.

And I would say that if the success of your project depends on your ability to forecast the future to that degree of precision, you are DOOMED from the outset . . .

Thus spoke The Programmer.

It’s amazing how much havoc a person can wreak in the workplace by applying a certain kind of inspired idiocy to every situation: follow all procedures to the letter, do exactly what you’re told, and respond to all questions exactly as asked. One-word answers are ideal.

The latter technique is especially effective via email.

Thus spoke The Programmer.

A friend of mine asked me the other day, “Do you think an organization really values a good manager?”

He asked me that because he’s moving from a position as lead developer on a high-visibility system (lots of job security) to a position managing the developers of that system.

And I had to say that in general, I think the answer is no, which is why you see managers generating a lot of useless paperwork to make their work visible: project plans, Gantt charts, spreadsheets, flowcharts . . .

Does this help? I haven’t found that it does, but it does provide an illusion of control and an acceptable way of failing: the manager can point to all the paperwork and say, “Well, I followed the accepted process right down the line, so the fact that we failed can’t be my fault!”

An analogy

Our local basketball team is coached by a guy named Phil Jackson. He’s not nearly as animated as most coaches . . . he spends most of each game sitting quietly in his chair on the sidelines, even when things seem to be falling apart for the team.

When things aren’t going well, he gets a lot of criticism for this:

They lost the game and he didn’t do anything!

I think this is why most coaches spend the whole game jumping up and down, yelling, tearing their hair out . . . they want to be seen as having done something.

Even if the team loses, people say, “Boy, he really coached his ass off.”

A second analogy

I grew up in Southern California during the years that John Wooden was coaching the UCLA basketball team. Like Phil Jackson, Wooden was distinctly non-animated during games.

He sat on the sidelines holding a rolled-up program in his hand, rarely called time-outs . . . his teams won 10 national championships and he hardly even got out of his chair the whole time.

He believed that the real work was done behind the scenes — preparation, attention to detail. Bill Walton summed up Wooden’s coaching style like this:

Don’t confuse activity with achievement.

Summary

Managing a software project is not a process of managing dependencies . . . it’s more a process of managing uncertainty, complexity and change. That’s why the Gantt charts and flowcharts don’t help, but they do allow the manager to show that he did something.

A well-managed team should have a clear, common vision, a robust flow of ideas, a reputation for high-quality work . . . but it may not be obvious to an observer what role, if any, the manager had in developing these qualities, because a lot of what a good manager does is not visible . . .

Thus spoke The Programmer.

Do you want innovation?

What are you willing to give up in terms of efficiency, predictability and control?

Thus spoke The Programmer.

Ill-specified systems are as common today as they were when we first began to talk about Requirements Engineering twenty or more years ago. Yet the task of creating complete and perfect specifications is not rocket science. We have adequate and comprehensible theories at our disposal for specification of finite state automata. We have proceeded over the past decades to develop and refine a discipline of applying these theories to real-world systems. In our methodological focus, we may have lost sight of some endemic problems that plague not the process but the people who do the process. Is it possible that an engineering approach to requirements is as badly suited to our real need as would be an engineering approach to raising teenagers? I’m beginning to think so . . .

— Tom DeMarco, “Requirements Engineering: Why Aren’t We Better at It?”, 2nd International Conference on Requirements Engineering

There are zillions of books on how to raise kids, and I think my wife has read most of them, along with countless magazine articles . . .

In fact, she’s far more open to taking child-rearing advice from books and magazines than she is from me, despite the fact that none of the authors has ever met our kid, and can therefore offer no insight into our particular situation.

But how comforting must it be to think that there’s a “methodology” for raising kids or for building good software — that someone has already solved all the hard problems for us . . . that we don’t have to rely solely on our own judgment to make critical decisions when we have only a limited amount of time, a limited amount of information, and no certain knowledge of the consequences . . .

Thus spoke The Programmer.

Related Links

• Peopleware
  If I could require that everyone in the IT business read one book, this would be it. Tom DeMarco (see above) is one of the co-authors.

The number of students majoring in computer science is falling, even at the elite universities. So [Bill] Gates went stumping at the University of Illinois at Urbana-Champaign, Carnegie Mellon, Cornell, M.I.T. and Harvard, telling students that they could still make a good living in America, even as the nation’s industry is sending some jobs, like software programming, abroad.

— The New York Times, “Microsoft, Amid Dwindling Interest, Talks Up Computing as a Career”

My brother is a doctor.

He doesn’t encourage his kids to go into medicine though, because he’s incredibly frustrated by the fact that you go to school for 20 years to learn something, only to have clerks from insurance companies decide if a procedure you’ve recommended is or is not “medically necessary.”

I’ve worked in computing for 20 years.

I don’t push my kid to get into it because during that time, it’s become less and less like a professional business and more like a big class project, full of people who have no aptitude, no education and no role models.

A friend of mine teaches a computer science class at a local community college. He loves it.

I don’t think I could bring myself to stand up in front of a group of young people and encourage them to be programmers. I’d probably wind up yelling at them to go be flight attendants or meeting planners and stop wasting their time.

Where are you going to go as a programmer to do interesting, influential work with bright, educated people? The list of possibilities is very short.

Microsoft is on the list — but the fact that Bill Gates is out recruiting at Illinois, Carnegie Mellon, Cornell, M.I.T. and Harvard while you’re sitting here in a community college class suggests that a Microsoft career may not be in the cards for you.

Thus spoke The Programmer.