Watching a bad system perpetuate itself

The following is inspired by annoyance, plain and simple. I’m watching a company I’ve worked with over the past 3 years do the same thing over and over. It’s tempting to quote that old saw about the definition of insanity but in that example the insane individual continues to expect different results. I don’t think this company expects different results. They expect the same bad, cheap work they’ve always paid for. And they expected me to do it.

Since 1999, I’ve been building websites. It’s always been a sideline to writing even when it was a full-time gig. I’m a self-taught coder and not entirely comfortable talking about highly technical issues or methodologies of developers. I defer to the experts in those cases. But I understand a thing or two about systems because I’ve been building them (as we all have) for almost 16 years.

When I say “systems,” I don’t mean physical ones. I mean the systems that evolve from our continual practice of a process (or non-process). You build something (like a website) that lives without your intervention and it’s a little system. You build a few hundred of them and you’ve built yourself a system of building them.

God knows, I’ve contributed to many bad systems and probably accidentally created a few myself. Because like most people, I didn’t look at what I was doing as building a system. First, as a freelancer, each website I built was its own entity for a different client but eventually bad habits on one build leaked into another and another. Next, as I became more experienced and worked for larger clients, I realized that 1) clients also don’t think of the systems they’re creating when they commission that website and 2) freelancers rarely have a say in the system to which they’re contributing. Clients make decisions by which freelancers must abide.

Such is the current predicament. A client needed some work on a site (that I didn’t build) that one of their clients couldn’t update. I could have taken it on and just slogged through the work to “repair” this website by undoing some dumb decisions and making some dumb decisions of my own.

For whatever reason I decided this was the project on which to push this client to do better work. I returned with a higher-priced proposal than I’d ever given them, suggesting rebuilding the site from scratch. This met with complete silence. Literally. I didn’t receive a response at all, negative or positive. Nor did I receive a response to a follow up email wherein I explained that hiring a “$50 an hour developer to fix the problems created by a $50 an hour developer isn’t going to improve your situation.”

The client has done what is perfectly within their prerogative to do: choose a cheaper option. It’s painfully clear though that when presented the “good, cheap, fast: pick two” menu, they continue to choose cheap and fast. That’s the other reason why I went with the nuclear option and priced a rebuild so high: I’m tired of being one of their cheap and fast options. I’d rather be the good option.

Once something works for you, even if it was badly done, you’re likely to do it again. If that something makes money for you over and over again, you’re not going to think so much about the Jenga game you’re playing with your backend. Quite honestly, this is the situation I’ve seen at nearly every agency I’ve observed (for this client is an agency: they sell websites, do design in-house, and hire contractors to build them). These agencies rely on “cheap and fast” so much that they pile up problems in the background.

Part of the agency system is relying on churn: either their clients will leave or they will need a new design before the site fails. But I’m a believer in doing your work simply and openly so the client can take it with her or at least the next developer can understand innately how to keep it running. Part of doing good work is iterating and improving, not starting over again and again.

There are larger problems. When “cheap and fast” is your repeated choice, your system becomes cheap and fast. When you don’t build an infrastructure (I’m thinking of the agency where I worked with no shared backup drive or staging servers), you create an environment where there is no option but to do bad work. In these environments, you’re constantly throwing good money after bad. You hire a contractor to build something quickly and don’t pay him enough to be serious, so it isn’t thoughtfully built, so it breaks, so you pay another person to try to fix it quickly, you don’t pay them enough and they don’t have time to assess the work before starting, so it breaks, so you pay another… you see the cycle.

In these instances, these agencies and clients are creating an unstable situation that is hazardous for their clients and end users. At one agency, we hosted 300 or so sites on our Linux server for which I was the (completely out-of-his-depth) administrator. Unbeknowst to me, the boss had given a client control panel access to our server on which to run some scripts. You can see where this is going. That eventually brought the whole system down. So dozens of our clients lost access to their sites for several days.

In that case, when the boss had reluctantly allowed us to start building websites in WordPress, he never fully supported the endeavour (e.g. never hired a real sysadmin despite my repeated concerns, compromised our security by giving credentials to a 3rd party), so it led to a teetering pile of risks. Eventually, the agency paid for that bad system not only with the meltdown but by having to hire a real sysadmin and expand the team. (Not that you hire those people before you have the business but ideally, you don’t knock 300 sites offline before you realize you need them.)

It is at this point that a warning is appropriate: if you find yourself stuck in one of these bad systems, you should be aware of one of the moves the boss will do. As part of their compliance with the bad system which they created by their own unconcern and negligence and then refused to correct, bosses will blame the person who works in that system. “The Analytics broke after you worked on them!” when you just made the update you were told to. Once that happens — when you get blamed for something in which you had no choice but to comply — you know you’re in a bad system.

I write a lot about work (demonstrated most by my latest book and its website) and while it seems like there is a bottomless well of griefing I could do about bad jobs and the bad systems in them, I am sincerely looking for solutions to these problems. I want to present options. Usually this is just a different way of looking at the problem. I believe that by dismissing the usual criticism of an issue, we free our minds to examine it better. What I mean is: take the gut reaction to an issue like the bad system and simply throw away your immediate response.

“This part of the site broke.” “That’s because the developer who built it sucks.” “Let’s just fix it.” Don’t do that. Take the time to look at the underlying problems and address them.

Remember: good systems perpetuate themselves in the same way that bad ones do. So make one good system. Then make another. You don’t have to look at your entire business and tear it apart (though, you know, that might be a good idea). You just have to start somewhere.

If it’s WordPress in which you work and you always set up sites in the same way, go back to zero and see if you’re following the current best practices. Just set up one site the right way. if you’re repairing an old site, it’s an opportunity to set up a completely new one and do it right.

Once you have a couple of pieces of a good systems in place, implement some accountability. I worked with one of those old-school ninja teams of consultants many years ago: it was a bunch of middle-aged dudes who travelled around the country to fix corporate problems. They had a fax machine and printer that could run off their car battery. That was mobile at the time.

They told me that when they started a job, after they determined the situation and the needs of the company, they created processes. Processes give us an object to examine. When we have a process in place and something goes wrong, we don’t blame the person. We blame the process. We take it apart and examine it. We see where it failed and we address that failure. Then we document it and put the new process in place.

At the time, I thought it was quaint. But that wisdom has held up over time (even if I don’t see many people following it). If your websites keep breaking and you keep paying cheap developers to fix them quickly, examine your process before you do it again. We’re building systems all the time. When they go bad, we rarely take the opportunity to open them up and correct the issues. But that’s what has to happen to improve them.


My latest book, good.simple.open, is about doing better work by focusing on simplicity and openness.

Jenga image by Ashley MacKinnon