A few weeks ago a software developer came to me with a problemwith some code. My suggestion was to do X. The developer's responsewas, “No one ever taught me how to do that.”  To put thisin context my suggestion was along the lines of: “What do the logsindicate or have you looked at a stack trace?” I was not asking thedeveloper to do anything outside of the realm of the normalSDLC.

Skip forward a few days when I provided a developer with ascript he could use to do some database manipulation. It was SQL101 but rather than having the developer look it up I figured itwould be quicker to just type it out and send it over. I added thecaveat that he would need to modify the code to work in theparticular environment in which he was working.

An hour later he was hanging over my desk complaining that myscript didn't work. So I walked away from what I was doing to seewhat the problem was. Sure enough a reference in the script to anobject on the file system was never modified to fit theenvironment. I was embarrassed for the developer when I had topoint out that “h:mojobob” did not exist in their developmentenvironment. But he wasn't embarrassed at all. It was my faultbecause I didn't tailor the script for his specific use.

These are not unusual occurrences. We routinely graduatecomputer science majors who are unable to write a line of code. Iguess there is no harm in that but why do we then allow them towork as developers? There was a time that being a softwaredeveloper had a certain panache— it described an individual who hadan inquisitive mind and an ability to think critically. Not so muchanymore. Copying a jQuery script from a blog now seems to qualifyyou as a software developer.

There are still savvy, creative developers out there, but theyend up being drowned in a sea of mediocrity. And they end up doingall the real work while everyone is tossing around banalities like“agile development” or “root-cause analysis” while sitting aroundin daily standup meetings hoping they don't have to do any realwork. How did we get to this state? There are a number of reasons:a demand for developers that has outstripped supply, shoddyeducational systems, generational changes in work ethic, etc. Butthe primary factor is the increasing complexity of data systems andthe higher levels of abstraction that result from thatcomplexity.

I was a Navy officer during the Vietnam War. I was assigned to aship as operations officer. It was a choice assignment. I wascoming off a destroyer where I was an engineering officer. Mybattle station was in the forward fire room standing in front of apair of boilers filled with 900 degree steam at 600 PSI. Live steamis invisible and will cut a man in half. Not a great place to beshould something go wrong.

Electronics had been a childhood hobby of mineand I had built a slew of radios, amplifiers and other devices fromelectronic parts I salvaged from old televisions. I even built abasic AND/NAND gate machine. Back then electronic chassis were handwired so an enterprising young person could build a huge inventoryof electronic parts from a half-dozen old TV's, which the repairshops would give to me for free.

As operations officer I was in charge of the radio room (knownas the radio shack in those pre-Tandy days). One day it wasreported that one of our long-wave receivers was down and that myteam was going to repair it. I eagerly joined the electronictechnician figuring I could help him diagnose the problem. I lookedaround the space we were in and noticed no multi-meter, nooscilloscope, no schematic diagram—none of the basic tools I woulduse to start trouble shooting.

I asked about this and the sailor gave me a big grin and said,“We don't do any of that. We just swap parts in and out until itstarts working.” So much for all of my knowledge ofelectronics.

Transistors and circuit boards made it extremely difficult forindividuals to get into the heart of electronic machines andmanipulate what happened inside. Complex machines must beabstracted to be properly maintained. Testing and replacingindividual components like diodes or resistors is not scalable.Replacing components or modules is. 

It was inevitable that components became bigger and morecomplex. As they became more complex, efficiencies of design andscale came into play. A pluggable power supply or pre-amp becamemodules that could be used across a wide range of electronicdevices. The modularization of electronics allowed for massproduced, less expensive machines. It also abstracted understandingof how the machine worked. The preamplifier becomes a black boxwith x inputs and y outputs and that is all an “engineer” needs toknow to use it to build another machine. And that reminds me of thecurrent state of software development.

By the 1970's mainframe development had become fairly stagnantand unexciting using high-level languages like COBOL, FORTRAN, RPG,etc. These were and are totally cookbook languages. Not a lot ofcreative thought needed to write a COBOL program. If you needed todo anything different with a mainframe you needed to drop down toassembler, which was fun and challenging but required thousands oflines of code to do anything. Mainframes were not something brightyoung engineers and scientists were drawn to. Minicomputers,microcomputers and personal computers were compelling and providedthe genesis for the current wave of software development.

First and second generation interactive computers opened upcomputer science and programming to a lot of frustrated geeks whodisliked the inability to get inside the machine and hated theonerous overtones of big business and big government thatmainframes had come to signify. It was no accident that TracyKidder's Pulitzer Prize-winning book (1981) about the creation of aminicomputer was titled The Soul of a New Machine.  Smartyoung people wanted to control machines from the inside…not justinteract with them from the outside.

Thirty-one years after The Soul of a NewMachine, software development is at that state of abstraction thatelectronics engineering has been for years. I don't see a lot ofsoftware developers creating code that writes directly to thekernel of the operating system. Software development for early PC'swas pretty much written to bare metal, the processor, and a verylight operating system.

Windows 3.1 and Charles Petzold provided a layer of abstractionon top of that, which still felt close to the machine. Using C youcould still access machine code if you needed to do something moreefficient than Windows was able. Open-source Linux offered theability to control how your software worked with bare metal, butfor the most part Linux users just operate with someone else'scode—sometimes free, but more often commercial.

After that we layered application platforms on top of theoperating system and software developers were not even aware of theOS except as something that must be configured correctly.Application platforms come with cookbooks that form the basis for“programming” on the platform. Programmers are so stuck in theirparticular niche that thinking outside the box is not only a stupidcliché, it isn't even possible.

I was peripherally involved with a project involving mobileaccess to a web (browser-based) application. The code was allwritten in C# using .NET and various Microsoft API's as well asJava running on Linux. Everything worthwhile was exposed as aRESTful web service. The folks doing the mobile application werewriting their application using native iOS code. They did notunderstand how to consume the RESTful services nor were they ableto consume any of the security objects exposed for theapplication.

Their solution was that “someone else” needed to writemiddleware that did all the heavy lifting. The middleware wassupposed to provide the mobile platform with an exposed API thatwas essentially pluggable from iOS. Unbelievable.  Theywrite mobile code but they are unable to connect to any existingsystems. I guess that might work if your goal in life is to writeAngry Bird clones, but that just won't work for end-to-end businesssystems.

We have created a generation of software developers that are notmuch different than burger flippers or cooks at a chain restaurant.They follow precise recipes created by others to “write” code.Suppose they need to write a pluggable routine to modify a customerprofile. They already have an example of how to create a new modulefor the CRM system. They do an Internet search for code to modify acustomer profile for that CRM system. Then they start trial anderror. They keep changing this and that until they get it to work.The Internet is their friend as they search for each new errormessage to find an answer to their problem.

Once they succeed the code is locked andloaded. The developer fears any modifications to the code becausethey don't really understand what their code is doing. They arewriting to an API for the CRM system which is a black box usingcode they didn't create. This is the inevitable result of black-boxcoding. No one except for the product team knows what is in thatbox and getting to that product team is harder than getting a rideon the Concorde.

I once had a significant problem with code I had written to aMicrosoft application. After weeks of jumping from one level ofpremier support to another and calls from my CIO I finally wasgranted a call from the lead on the product team. When I posited myquestion he laughed and said, “Yeah, that won't work, but thiswill.” It took him less than five minutes to help me. No one elseon my long path to that call had real knowledge of what was in thatblack box. We are becoming very close to the state where no onewill know how to build the machines we depend on.

One more thing about that CRM problem above: It would have beenmuch easier to create a small application and write directly to thedatabase in a non-invasive way. But that would have violated thelicensing agreement with the CRM vendor, which would mean nosupport. That leaves us stuck with using their black box API orengaging them to do a product modification or add-on.

Of course, there are other reasons why software development isin such a sad state. Demand truly has outpaced supply and manydeveloper positions are filled with people who just don't have theskill set or mental capacity to do it. And it isn't as exciting aprofession as it was 20 years ago. The coding part of softwaredevelopment is not all that thrilling. The fun part is in designingthe system itself and creating the algorithms for the code.Unfortunately when all a developer can do is code against anexisting API there isn't a lot of room to incorporate clever,creative algorithms.

That being said, we still haven't explained why there are someexceptional developers who always amaze and always perform. I thinkthe answer is that there have always been killer coders out there.They are just harder to find among all the rabble.

Please address comments, complaints, and suggestions to theauthor at [email protected].