The executive meeting has concluded. Your boss calls and informsyou that the decision is made. We are going with the new Acmeratings engine. Not only that but we are building an entire newsuite of applications around it—from policy management to CRM toreporting. The timeline is 24 months, everything must be builtusing SOA, and we'll be using an agilemethodology.

You didn't get a lot of sleep that night. Agile—yikes—betterstart interviewing for SCRUM masters and hit the ground running. Aproject like this could mean the difference between finally buyingthat nice little beach condo and dumping your miserable two weekAugust time share in Key West. You have got to move fast. Two yearswill be over before you know it. Agile means fast. Maybe we canstart our first sprint next month.

Agile Does Not Equal Fast

I used to run marathons. I keep thinking I am going to runanother one but I work too much to train enough, plus I am old andoverweight. I'm not sure which of those is the real problem. Thereis one thing I learned when running 26 odd miles on multipleoccasions. Going out fast is not a good idea. It is fun to sprintout with the elite runners but running a mile or two at asix-minute pace when your target is something like 7:30 per mile isa very bad decision. You don't really understand that it was a baddecision until about mile 20 and by then it is too late. The samething applies to software development—lay down a good solidfoundation and save your sprints for the end of the race.

Larger Architecture Equals Fewer Architects

Large projects requiring multiple software systems and multipletechnologies require careful design and planning before one line ofcode is written or one subsystem is spec'd out. Senior IT thoughtleaders—you may call them system architects or computer scientistsor whatever—need to whiteboard and design the systems and theirinteractions before the level of effort can even be scoped.

Since this project will be using SOA the design can bevirtualized, with service or API layers encapsulating the actualsoftware components. End points like UX/UI and primary data storageare components that only touch the system via service layers.Business logic and rules are built into the service layer wheneverpossible, although some will reside in key subsystems. This is thefoundation upon which the entire project will succeed or fail andit is in the hands of a small number of individuals.  Youare paying a small group of people to be smart and this is whenthey earn their money.

While the foundational work is being accomplished you can startbuilding out the rest of your team. Your BA's, PM's, developers,development leads, QA teams, SME's and a host of infrastructureteams. Oh yeah, you can start interviewing scrum masters, too,although the agile part is still months away. Two of these groupsof project members—the BA's and Developers are notoriouslydifficult to work with and cause unspeakable pain and suffering ona large project if they don't understand their role.

Business Analysts

A long time ago in a galaxy far away there was science calledsystems analysis. The premise of systems analysis was that areasonably intelligent person who understood information systemscould work side by side with a business team and gain anunderstanding of the required business process. Armed with thatknowledge they could then map that process onto softwaresystems.

Somewhere in the last 30 years that methodology has beenreplaced with BA's interviewing business owners and gatheringrequirements. Notice I said business owners—BA's tend to gathertheir requirements from managers, not the people who do the work.Then they interview the manager's managers and since that group iseven further removed from the actual work they are more concernedabout reporting on the amount of work accomplished than the workitself. And the madness continues.

Too often BA's simply transcribe wish lists. To wit: "What weneed is the ability to see all the social media interactions acustomer has had in the last 90 days when they call to report aclaim." That may be a valuable chunk of information but it doesn'tsound like a core business requirement.

BA's must have the ability to understand the difference betweenreal business requirements and nice-to-haves and fantasy worldnice-to-haves. BA's do not need to be current employees. In factyou will probably be better served by using contract BA's with therequisite skill sets but without pre-conceived ideas or prejudicesabout the business. And never use a BA who is already part of thebusiness unit being analyzed. They already "know" what is needed,which pretty much precludes the possibility of innovative thinkingand discovery.

Software Developers

Now here's a tough group to deal with. I was one myself andunderstand why they can be so difficult. A common practice in largeprojects is to bring the software developers into the process assoon as the first rounds of business requirements are complete. Bigmistake. Software developers want to help people; they want tobuild the cool things that their customers want.

"So you want the system to do a speech to text conversion on theconversation with the CSR then email that to the customer when thecall is complete? Sure, we can do that."

Suddenly the fantasy world requirement is in scope. The work ofthe smart guys who did the initial high level design is notcomplete when the high level design is. That group—or theirequivalent at the system or subsystem level—should take the firstpass at mapping the business requirements to the appropriatecomponents.

Software developers also tend to overthink system design. Edgecases are important but there is probably no need to code for thoseonce-in-a-decade edge cases. I recently observed a discussion abouttransactions that occur in that undefined period when the eastcoast office has already flipped to daylight savings time and themidwest office won't for another hour. It is a non-issue becausethe data centers use a common time and any client side differencescan be compensated for. Events happen at a specific point in timeand that is not affected by location of the user. That is why themilitary does things in Zulu (Greenwich) time on a 24-hour clock.Ambiguity is a bad thing for military (and data processing)operations.

System Architecture andDesign

Back to system design. Once the high level design is solid andapproved, the lead team need to decide what system is going to liebehind each of the abstracted service layers. Build or buy isalways a tough call. If you can find commercial software that meets90 percent of your requirements then go with that and forget aboutthat missing 10 percent.

Most vendors claim that their product is accessible through afull featured API and that it is infinitely configurable to meetany of your requirements. Don't believe it unless you see it. Moreoften than not the API is limited to whatever they needed to makethe product function in the first place. The database schema ishidden and doesn't matter anyway because your license forbids youto read or write to it anyway. You have no access to source code.You are provided packages that can be deployed to your servers—andif it doesn't work it is always your problem.

That may sound a little negative and it is. If commercialsoftware fits the bill then go for it, but don't force the vendorto make a lot of substantial changes to their product. System downor degraded situations are when you discover the true worth of avendor and their product. When I am on a call at 4:00 a.m. with myCIO listening in I don't want the vendor to tell me their leaddeveloper is on a six month sabbatical in Uzbekistan. And I alsodon't want the vendor telling me that they can't really supporttheir product with this or that particular release of whatever.

I like throats to grab and I prefer to grab throats that gettheir check from the same place I do. Black box software systemsare not supportable. Kind of like "building" a computer when whatreally happened was simpler than assembling an Ikea bookcase. Andwhich requires zero knowledge of how a computer functions. All thatbeing said I have had many excellent experiences with software andsystem vendors. They are not all scary. But you need to do your duediligence before making a commitment to build enterprise systems ontheir product.

Roll Your Own

Inevitably many of the functional components of a large softwaresystem are going to be home grown. If nothing else you need to ownthe service layers with all the business logic and probably theUX/UI. Let's assume after much iteration and discussion thebusiness requirements are final and let's assume that includes allthe business and system use cases.  You are now gettingvery close to getting some value out of those scrum masters, butyou aren't quite there yet.

The single most critical documents you will create are yoursystem design specifications. These start out as high levelrepresentations of the interactions between one system and othersystems – passing through the virtual service layers. The teamsthat create these documents are small. Bite off a small chunk offunctionality and get the team leads for the various subsystemsinvolved and create process flows through each system. Then takeeach group of endpoint in the process flow and get those two orthree people to design that sub process.

Large projects are made simple through decomposition. It soundssimple, but very few organizations have the patience and theconviction to make the process work. Deadlines are more compellingthan process. Once you subvert process you are guaranteeing acertain degree of failure. Properly built design specifications canbe turned over to any set of competent developers with anexpectation of a reasonable degree of success.

Now is the time to build out virtual services layers (I meanreally virtual; not abstracted) that will return expected data whenconsumed by other components. We can get a little agile now. Thevarious subsystems can be built in parallel with straw man"virtual" services standing in for the real services or API's whilewe wait for them to be built. Fire up those sprints and beagile.

Basics

Large scale software development is not difficult. It is in facteasier than managing smaller projects since everything can be soeasily siloed. It does require strong leadership – you need anoverall program manager and program managers for each sub system.You need strong well qualified technical leaders who are capable ofdesigning enterprise software systems. You need commitment fromsenior management to support the program management and theapproved methodologies. You need to manage people's time well.Twenty people in a room designing system interactions is a waste ofproject resources. Small groups (5-6 or fewer) can make gooddecisions. Anything larger than that and you end up designing acamel and you will never get that beach condo with a camel.