Tuesday, June 19, 2007

Creating My Own Personal Hell

In today's excellent post, Jeff Atwood blogs about the classic development process mistakes. He provides this handy list, which I will blatantly repeat here:

People Mistakes Process Mistakes Product Mistakes Technology Mistakes
  1. Undermined motivation
  2. Weak personnel
  3. Uncontrolled problem employees
  4. Heroics
  5. Adding people to a late project
  6. Noisy, crowded offices
  7. Friction between developers and customers
  8. Unrealistic expectations
  9. Lack of effective project sponsorship
  10. Lack of stakeholder buy-in
  11. Lack of user input
  12. Politics placed over substance
  13. Wishful thinking
  1. Overly optimistic schedules
  2. Insufficient risk management
  3. Contractor failure
  4. Insufficient planning
  5. Abandonment of planning under pressure
  6. Wasted time during the fuzzy front end
  7. Shortchanged upstream activities
  8. Inadequate design
  9. Shortchanged quality assurance
  10. Insufficient management controls
  11. Premature or too frequent convergence
  12. Omitting necessary tasks from estimates
  13. Planning to catch up later
  14. Code-like-hell programming
  1. Requirements gold-plating
  2. Feature creep
  3. Developer gold-plating
  4. Push me, pull me negotiation
  5. Research-oriented development
  1. Silver-bullet syndrome
  2. Overestimated savings from new tools or methods
  3. Switching tools in the middle of a project
  4. Lack of automated source control

I was horrified when I saw that list. I've seen it before, of course, being a big fan of Steve McConnell's works. But seeing the list put there like that, so succinctly, gives me this sinking feeling in the pit of my stomach. I think the project I'm on is guilty of almost every one of those. The problem? I'm the sole developer on it (admittedly, however, not by choice).

I was thrust into this project from day one with a completely unrealistic schedule, no requirements, no goal, no vision, no contact with the users, no management team, and no freedom of movement in the tool selection. When it comes down to it, I have to fill all the roles in the project: I have to be the DBA, the software engineer, the business analyst, the architect, the web designer, the project manager, the customer support representative, QA engineer, the risk manager, the technical writer, the release engineer, and the software developer.

I get paid to be the software developer. It's my job title. I'm not particularly very good at much else; I slog my way through the rest of the roles. In three years, I've been pulling my hair out to delivery *everything* related to the project, and as a result, every facet of the project has suffered. The code quality has never been what I would have liked it to be. In this particular case, the old adage, "Jack of all trades, master of none," has certainly proved to be true. My inability to specialize in my chosen field has prevented me from being able to deliver code of a superior quality; instead, I've delivered substandard products across the board: code, product, documentation, and customer service.

Our process is haphazard at best. I am constantly rushing to get the builds out according to a quarterly schedule. The customer seems to want to get the builds as quickly as possible. I never have time to fully regression test the builds. The customer wants to constantly shortchange the testing time in order to get the builds in their hands. So, I get them the builds when they want them, giving them the caveat that I *know* there are bugs in it. I ask them to fully test the build from end to end. But they never do. They find a handful of bugs and throw it back to me, insisting that the bugs they've found must be fixed before they will continue testing. The company will not hire a QA or testing staff because they believe that I can do it all. So I have to stop my work, resolve the handful of defects the customer has found, and rush that bug fix build out the door. Again, the customer wants that build as quickly as possible--without sufficient time allocated to me to regression test it. So I give it back to them, and they shortchange their own testing again. And the cycle repeats itself.

The customer seems to think that the entire problem rests at my feet. Perhaps it does. I don't know. I'm just tired of being shortchanged on the testing time, the lack of hands to help with the testing (even if it's temp assistance), and the proper prioritization of testing in the development process. You CANNOT shortchange testing in software development. The number of defects in this project has soared out of control because the customer will not allow me to take the time to test it fully, nor will they fully test it themselves.

Each time they get a build, they insist on getting an update to the test scripts, which I dutifully try to update--but they don't give me enough time to complete them--so they get partially updated test scripts. Even then, I know that they aren't using them. How do I know this? Because when I use the test scripts to test the software, I find defects that aren't reported in our defect tracking system. I know they aren't using them. So why do they demand that I prepare them for them? Are they using them as some sort of ego-bloating paperweight on their desks? I know the work has to be done, but dammit, why isn't sufficient time allocated for it?

I cannot tell you how many weeknights and weekends I've given up for this project. And it's not like I'm getting paid overtime for this stuff. I have no personal life. And "thank you" and "good job" is simply not enough. I don't get paid for all the other hats I wear. I get paid to be a software developer. I don't get paid to be the the DBA, the software engineer, the business analyst, the architect, the web designer, the project manager, the customer support representative, QA engineer, the risk manager, the technical writer, and the release engineer. And management doesn't see that. All they see, every time I bring this up, is that the customer is apparently "happy" and that I can always be relied upon to get the work done.

Of course I can. I'm killing myself to get it done. It's Classic Mistake #4: Heroics. The company thinks I'm freaking Superman.

My God. What have I done?

To a large degree, I led the company to that opinion. I took on too much work. I let them believe that I could do it. But eventually, I started pushing back when I realized my mistake. I set realistic expectations about what could and could not be done. The course was reasonably corrected.

But in every project, in every organization, there is a certain Wall. Beyond that Wall ye shall not pass. It is the Wall of Absolute Resistance, and no amount of pushing, pleading, or reasoning will avail you. The forces on the other side are simply not reachable.

You can preach best practices until you're blue in the face, and some organizations will simply not see the light of day. You can talk about automated unit testing and they won't care. You can talk about investing more time in testing, to reduce the costs of defects as early as possible, and they'll still be more interested in beating a competitor to the market with a bug-riddled application than they will be in sparing users a frustrating experience and corrupting their data. You can tell them that you are going to have to bring on more hands earlier in a project to ensure its success, and they'll be more worried about the costs of the new employees' benefits in the short term than they will be about the rapidly escalating costs of the software's development and maintenance in the long term.

What do you do in an organization like that? What should I have done? The short answer is this: I don't know what I should have done in the past. It's been a valuable learning experience for me, that's for sure. In newsgroups, people have told me they envy me my position, that it's great job security. My response to them these days is this: I can live without this kind of security.

In the future, if a company offers me a job and tells me I'll be the sole developer, I'll walk away from the offer. I'll never do this again.

Some folks have claimed that it presents the great opportunity to establish your own process. In my experience, there is no process in a team of one. There's nothing in place to hold off the torrents of work that come your way. There's no one to correct you when the urge to gold-plate the code comes along. There's no one to review your code. There's no one to ensure that your code is checked in on time, labeled properly, unit tested regularly. There's no one to ensure that you're following a coding standard. There's no one to monitor your timeliness on defect correction. There's no one to verify that you're not just marking defects as "not reproducible" when, in fact, they are. There's no one to double-check your estimates, and call you on it when you're just yanking something out of your ass.

There's no one to pick up the slack when you're sick, or away on a business trip. There's no one to help out when you're overworked, sidetracked with phone calls, pointless meetings, and menial tasks that someone springs on you at the last minute and absolutely must be done right now. There's no one to bounce ideas off of, no one to help you figure your way out of a bind, no one to collaborate with on designs, architectures or technologies. You're working in a vacuum. And in a vacuum, no one can hear you scream.

The insane hours, the stress, the loss of a productive life outside of your job--it's not worth it. I'm willing to accept all that I have done to create this scenario. And you can bet that I'll be doing everything in my power in the future to ensure that it doesn't happen again.

If anyone's reading this, let this be a lesson to you. Think hard before you accept a job as the sole developer at a company. It's a whole new kind of hell. If given the chance, take the job working with other developers, where you can at least work with others who can mentor you and help you develop your skill set, and keep you abreast of current technology.

Take my advice. Don't open that door. It leads straight to Hell.

23 comments:

Chuck Pergiel said...

If the powers that be think you are invaluable, then you are set. Only do as much as you want. Deliver when you are ready. They want it right now? Well, that is just too bad. It is not ready. Take a two week vacation and see if it makes any difference. I will bet that it does not. They will not even notice that you are gone, but when you get back their relentless, unceasing, unreasonable demands will be back in full song. On the other hand, you might be out of a job, but given your feelings about your situation, that might not be such a bad thing.

Anonymous said...

See, now that's just the opinion that naive posters in the newsgroups take.

Reality, however is a far different landscape.

Suppose I did what you suggest. That would only make my situation worse, since I am ultimately responsible for the work that must be delivered. I created this hell, and I must ultimately live in it. Why would I want to fan the flames of my own torment?

And if I did get fired, I'd need something to fall back on. Over the last 3 years, due to a lack of access to current technologies, my skill set has grown out of date, and I know little to nothing about modern methodologies (Agile, for instance). How could I compete in the marketplace?

The point is this: Fix what I can, repair the damage, and don't repeat the mistakes of the past. Don't take this sort of job in the future. Know what I'm responsible for screwing up, and learn from it.

And DON'T take the advice of people who seem to think I've got it made in cowboy coder Nirvana, and can do whatever the hell I want to do.

Anonymous said...

1. Extreme SOA
2. Paranoia Production Support on Pilot
3. Final Mess up by Tiger teams
4. Starting a battle finally with Information security
5. Over engineered Build / deployment process
6. Regular release vs maintenance overlap with version management nightmares
7. Remote DBA / Application server admins
8. Marketing and Wall street nexus

Babu Manoharan

Anonymous said...

I actually agree with the first comment to a certain extent. Don't make them want to fire you, but you sound like you're doing a great job -- make sure that they know that people who do great jobs are hard to find. Talk to them about how 40-hour work weeks and taking a break improves productivity. Tell them that if they really want those builds fast, they need to bring in more people. Or at least make it worth your while to live in your "personal hell."

Chris said...

Really - push back a little bit! Or quit... I was there about a year ago with a job in an engineering organization that had always contracted out their software development but decided to bring it in-house (that was me). I was lucky in that the boss did recognize some of your issues and pushed back over some of it. So I was able to produce at least in my mind (still) decent quality code that still is running fairly well. But yeah all the other aspects of the project (like database development) and testing are all "part of the job" and what I knew I did (like DB development), but what I didn't (like testing) just didn't get done very well. At least I told my boss.

Either which way, the travel was more than I thought it would be and I quit... but they asked if I wanted to continue to "assist" part-time hours on the side which I do now. The thought was they would hire someone and I could help her/him out, and get them up to speed. Well, they hired someone a few months ago, and I did try to get him up to speed, but he ended up quitting sometime like a month ago.

Needless to say, I have as much contract hours I want for awhile...

Chris said...

I hadn't fully read through your post until now, but from what you are describing, I find it really hard to believe you don't have solid skills that are NOT competitive in the marketplace! So you aren't up on the latest? I am not sure where you live, but do some contracting gigs for a bit using more recent technology. That job I worked at where I was the sole developer I was doing Java! Not exactly where I personally wanted to reside in technically speaking. I took a contract doing some C# (.NET is where I am more comfortable), and then got hired on at another place doing VB.NET (5-man IT shop with 1/2 other developers in a state gov't dept - very nice).

Don't shortchange yourself! At least drop some feelers on what other options you have. Experience life man! It's more than just a job...

Drew @ Cook Like Your Grandmother said...

Lots of managers have heard the advice to immediately fire anyone who becomes indispensible. If they really depend on you as heavily as you say, someone, somewhere knows it. And they don't like it.

Maybe the reason the customer doesn't care about testing is they're woring with another vendor on a replacement. Maybe they're developing something new in-house. Fact is, by not letting you do better work, they're making you more replaceable.

Start looking for something new. Leave work every day at 5 and spend at least an hour a day looking. Get out ASAP.

Rezlaj said...

I hear you, I'm in the exact same situation. The only difference is that I apply new methodologies and new technologies even if the customer doesn't want them. That's the only way to stay on the curve (don't even think about being ahead of it) at least.

You should always get something out of every job. If you want to work on new technologies then either apply them or leave. You might have to take a job that doesn't correspond with your years of experience, but because you are experienced you can learn quickly and move ahead faster than the rest.

So, to sum up, 3 years is 2 years too long! start looking for jobs that you like, in less than 6 months you'll be working with the technologies you want.

Anonymous said...

And I thought I was the only one with this problem!

Working with other people - when it "clicks" - is a wonderful, energizing experience. Working alone for a long period of time can be very draining.

I like my job as a whole, so I'm trying to compensate for the loneliness in other ways. Tech blogs, books, podcasts. I'm a huge fan of DotNetRocks and se-radio. Hearing the enthusiasm in the voices of people on these podcasts has really reinvigorated me.

I agree with the other comments to your post that at some point we all have to set some limits with our bosses/customers about what we can accomplish in what amount of time. Setting limits is key to surviving in any job. It helps to be willing to walk away if your conditions are not met. It's a mental game. I'm not always succesful at doing this, but I'm convinced it is the answer.

All the best to you, and thanks for posting.

Jacob said...

Find a new job. Do it now. It doesn't matter how "outdated" you think you are, your skill set is valuable. The tech marketplace is humming right now and chances are you can find yourself a raise as well as some relief. Leave some other schmuck to deal with unreasonable management (and make no mistake, your systemic problems are the result of poor management).

Michael Foord said...

Hmmm... here (in London) there is high demand for developers - *especially* those who care about quality.

If you care about qualitythen (in the right environment) agile will come naturally.

I would definitely put feelers out - plus the gradual process of pushing back and making more noise. You've made it too easy for your employer!

Good luck.

Anonymous said...

Powerful post...I think you COULD bail and just fall back on being clever. Perhaps you're not up to date on the latest stuff, but these thigns can be learned.

However, I think you have the right idea. While you're in hell, learn how to handle heat, and avoid going to hell in the future. Just make sure you make it out with your soul intact. ;)

Keith said...

Man...

I read your post to my wife and she asked if I had written it. I love the autonomy of being a Lone Ranger. I love the fulfillment of being solely responsible for something that works well. Beyond that, it's a nightmare. At least I don't feel so alone now. Thanks.

Keith

Unknown said...
This comment has been removed by the author.
Unknown said...

In the end, it's your work. When it craters, it will be on your name. No one will care that you were overworked, overstressed and raised valuable and valid issues. The only memory will be that you failed. "And to think they had so much faith in you..."

Your own competence can be your worst enemy. Because you are good, They can maintain this fiction. They don't have to invest in testers and a DBA. Because They have you.

Good luck with this one, from someone that traveled the same road long ago. Don't quit, you have a duty to see this through, given your own responsibility in the situation. Next time, remember that your name is on it, and insist from the jump that it be done right. No project is worth losing a family or a life over.

Anonymous said...

My current project is basically the company's golden goose for both the present and the future, so when I outgrew what I could do myself, I started getting more people. I count myself pretty lucky on that front.

If there's anyone in management who is at all receptive, talk frankly with management about how your superhuman workload is burning you out and resulting in diminishing returns for them. Start delivering timetables that don't require 80-hour workweeks to meet, and ask for additional resources to accelerate the timetable if they need it faster. Try to manage expectations back to a more realistic level.

In most workplaces, if you've proven yourself like that, you will have developed enormous credibility that you can use to convince your management. It's hard to make yourself take a step back from killing yourself, especially when you've made commitments to people. But if the people you work for are even remotely empathetic, it's very likely that they will understand and support you far more than you think they will.

Best of luck.

Farley Knight said...

After reading your post, writing a lengthy comment, then deleting it, I'm slowly beginning to wonder: Is the problem that you are writing bad software, or is the problem that you have a bad working relationship with your superiors? Did you make too many promises that you can't keep? Are you continuing to make them? Is there an expectation, both inside of you, and from the company, that you continue to work the way that you do? You won't realize it until your tongue slips and makes that same exact promise you told yourself you won't make. If the people you work for are subtly helping you reproduce these mistakes, then you might have a people problem, not a software problem.

dasil003 said...

Wow. I am right there with you. I think my situation is better that I'm working for a small company where I have a financial stake. I get to use modern frameworks and methodologies, I believe in the project from a business perspective, and most of the positions I'm filling are all within my specialty.

But ultimately I feel exactly the same way you do. What's missing are your reasons for enduring this kind of position. In my case it's because the project is really cool, and worth the suffering if it pans out. Also, my management actually does get it; we've been trying to hire more developers for 3 months and can't find anyone qualified in the area.

If it went on for 3 years though I would have to do a serious reality check. I've been suffering for about 4 months, and it's taking a heavy toll. But you can bet that every lesson learned will be applied in full for the rest of my life. I'm performing well under pressure, but also making sure it never happens again. Are you?

Anonymous said...

i feel you man. I am there, moved across the country and I am still have the same lone gunman job, but now its worse, as I am working from home. Being bribed into staying 2 more months, 2 more painful months and still have no real chance of getting the project done.

It use to be fun, I use to have other developers to work with, a QA Team, a Project Manager, but they all bailed and now its just me. Every bug, every build, every feature, every meeting, every change.

Gotta find a new gig. 8 weeks left.

Take care bud.

Joe Grossberg said...

I have a million other reasons for not working solo (at least for *someone else*), but this sealed the deal.

Thanks for saving my life, courtesy of the spared blood pressure.

Ed said...

It sounds like you are doing one hell of a job doing everything, something that most developers can't do, and so will pay dearly for.

IMHO, you need to set expectations on your users to prevent the mad scramble during and at the end of each release. My 2c suggestion is to start blocking off time in your schedule to do proper testing and QA and update unit tests by scaling back on the deliverables per quarterly release. You could start with 3 or 4 weeks/Q for these activities.

To do this, you have the users prioritize their wants and needs for the latest release. Estimate the time for bug fixes, estimate how long each of the tasks will take, and then figure out based on how much time you are leaving yourself for testing/implementation which tasks will get done and which will be in the next Q's release. Hopefully they will realize that a) not everything they need will make it in, and b) many of their needs are actually nice-to-have's. Figure on a couple of rounds of prioritizing at the start of work on a release.

It is guaranteed that the users will be skeptical at first and I guarantee that they will accuse you of padding the sh*t out of your schedule, but if it goes well for a couple of quarters they will like it better, or at least grudgingly approve of it. It's not like they have much choice, right? :-)

I have used this method successfully for a number of large projects. I've received pushback almost every time, but persistence plus quality, on-time releases have convinced them, and hopefully it will work for you.

Anonymous said...

I've been in and out of these situations quite a few times now. What I've learned is to ensure that you push back on management early, let things (as a previous poster suggested) slip a little early on, as you take on some new burden, even if you could have managed the task by doing a bit more.

I am not advocating being a slacker. But once you take on that extra role, it isn't going to get passed on to someone else, it's just going to be added to the list of things that you do, as part of your normal week. Then, when something else comes up, you'll be asked to see if you can fit that in. You sound smart, you probably worked something out to save a bit of time and get the first 2 jobs done, and were getting away with it. So there must be some slack to get this 3rd job done too, right? And so it escalates.

Mark your boundaries. Protect them.

If a project manager is needed, it's your duty to tell management that the project needs one. If you need a tester (because you can't reasonably test your own code anyway - by definition, you can't find what you missed, because you missed it!)

There are a lot of posts suggesting that you find a new job. It sounds like that will leave the current project in the lurch. I happen to be in the same situation myself at the moment (just over 2 weeks till I leave - not that I'm counting the days, of course!) It's not a nice thing to do, and I do feel bad that (because of the structure of my current employer) I won't be able to do a full handover to someone who will have the time to properly take over what I have been doing. But at the end of the day, I'm a person, with a career, and my employer is a business, which does not care about my sanity, or anyone else's, it only cares that I get stuff done for them, in return for a salary.

Things will fall apart when I leave; I'll do my best to make sure that the damage is as little as possible, but there will be a fallout. It's not my problem, it's their problem. We techies can get too involved, too passionate about the projects we work on.

If your employer couldn't cope if you got hit by a bus, that's their problem, and not yours.

Anonymous said...

As you are working there for three years, you must now have a lot of domain knowledge which a new person will not have. Use that to your advantage.