IT Development Teams Take Two Steps Forward

A decade after its introduction, agile development has established a strong following. Forrester, noting that agile has “rapidly joined the mainstream of development approaches,” found it to be a primary method in use at 35 percent of organizations in a 2010 study. Novarica, also in a 2010 report, reported that the vast majority of insurers were using agile in at least some of their development, primarily around new applications.

Yet the question on the minds of many insurers is whether—or when—agile is right for them.

Agile's purported advantage lies in the literal meaning of its name. Agility is derived through an iterative development process, a continuous feedback loop that connects business and IT at every stage, and incremental deliverables.

“If I'm a customer and I want certain features to be done on my product on day one, and then on day three I read a blog posting that makes me change my mind by day four, that's what software development teams need to be able to deal with. Agile allows you to do that by providing the framework to help software developers respond to that rapid pace of change,” says Michele Sliger, owner of Sliger Consulting Inc.

“Agile allows business users opportunities to interact with the evolving system,” says George Grieve, CEO of CastleBay Consulting. “By breaking down the scope of a project into 30-day deliverables, users get early and frequent opportunities to verify that what they are getting is what they asked for.”

Grieve claims he has never been involved in an agile project that has failed. “I've seen projects where the budget and timeline certainly came under stress, primarily because the company didn't have strong enough change-control to separate simple changes from truly new requirements. However, every agile project has delivered successful results that have been well received by business,” he says.

BECOMING AGILE

Agile proponents contrast the approach with “defined” development methods, the best-known of which is waterfall. Waterfall, where one stage of development has to be completed before moving on to the next, offers a sequential and structured approach that has been used successfully on countless IT projects over the past 40-plus years; however, proponents of agile point to its waterfall's limitations.

“In waterfall, the focus tends to shift away from getting working software to the company to making sure everything is completed on a checklist before moving forward to the next stage,” Sliger says.

“As a result, you can spend months in the design phase before anyone even starts coding,” she explains. “Ultimately, because of all the bureaucracy around waterfall, the development team is often given a document that tells them what to do but that they've had no input into. As a result, they have no understanding of what customer needs are. Many no longer even see or talk to a customer directly.”

The sequential nature of defined development assumes that the business side can completely articulate its needs, which may not be the case.

“In the past we would spend months if not a year building specs, then deliver something that didn't actually meet users' needs,” says Gary Smith, manager of application development for Michigan Millers.

Tom Mellor, who began his career at State Farm in the claims department in 1992 before migrating to the development world in the early 2000s, saw this limitation first-hand.

“It was often futile for the business to try to develop complete requirements upfront, and as a result there were many problems that resulted from people not getting what they needed or wanted. I knew there had to be a better way than to think we could look into a crystal ball in new product development,” he says.

Waterfall development also assumes that business needs, even if well articulated, will not change. “When I started working with project development, I was amazed with how unpredictable the work was, yet we were expected to be predictable,” says Mellor.

“From the business point of view, change is happening all the time that impacts the scope of the technology they need,” says Grieve.

Among the different flavors of agile, the most popular is Scrum. Mellor, who was instrumental in introducing agile to State Farm in 2004, is a certified Scrum trainer and agile coach at the insurer, serving as project manager in business application development areas.

Seven years after its introduction, agile has had a marked impact on State Farm's development environment, including in the layout of its headquarters in Bloomington, Ill. In an initiative dubbed Systems@Work, three floors utilized by its development teams were converted from cubes and walled offices into open spaces and office “hotels” where no employee—even a manager—has a designated workspace.

“My office is a backpack and I have a locker I put stuff in. I go where I'm needed. There is open seating and honeycomb work areas. It works well and is very collaborative,” Mellor says.

AGILE SUCCESSES

State Farm's venture into the world of agile began, in the words of Mellor, “surreptitiously.” A business unit saw an opportunity to get a product to market to take advantage of new health savings account legislation, but needed a system up and running in 60 days in order to do so.

“I had explored a bit about Scrum and believed it would work. I promised the business and technical sponsors we could get the project done, but it would have to be run under the radar,” Mellor says.

The project, requiring about 2,000 development hours, came in on time and on budget. Based on that result, State Farm used Scrum successfully in a 25,000-hour project later that year.

“After we had completed the core work on that [second] project, we still had money and people left. We asked [the business] if they needed anything else delivered and they were flabbergasted. After that, word got out. There was a lot of buzz going on,” Mellor says. That led to additional Scrum projects, the certification of Scrum masters, and experimental projects with other flavors of agile, such as Extreme Programming (XP).

State Farm's start-small approach is one that Grieve recommends. “From a practical point of view, the first couple of projects a carrier does should be small to control the sheer scope and risk and to allow time to learn the methodology as they go along. The methodology isn't flawless, and there are challenges that arise because of the continuous feedback loop of agile,” he says.

“The difference of addressing those challenges with Scrum compared to traditional waterfall development is that everyone is aware of problems, such as when projects are running late. We also used Scrum philosophy and framework to create solutions to time overruns,” Mellor says.

“What caught the eyes of a lot of people was that we were delivering along the way, we recognized early on when problems were arising, and we could respond with simple, interim solutions,” he says. “That wouldn't have happened naturally in waterfall, because in waterfall there are no provisions for that kind of adaptation.”

Yet even with a high level of project success and the environmental changes that occurred at State Farm through the Systems@Work initiative, the insurer's development teams still use waterfall for the majority of projects.

Out of over 700 projects in progress in 2010, fewer than 100 involved agile methodology. “We use agile primarily for new product development and applications where there is a lot of uncertainty around requirements,” Mellor says.

State Farm encourages project teams to select a development approach based on three factors: the relative certainty of requirements, the types of technology platforms involved, and the preferences of the customer. Projects with a high degree of certainty around requirements or that involve legacy or exclusively infrastructure development are more likely to be addressed with defined development approaches such as waterfall.

The third factor is most important. “If the customer wants to inspect a product as it is being developed, including delivering it incrementally, then Scrum is a great fit. But if they don't want it delivered that way, or don't want to be involved in the development process, its usefulness becomes compromised and we likely won't use it,” Mellor says.

Whereas State Farm started small, Michigan Millers took on the development of a new policy administration system, dubbed Matrix, for its first venture into agile development. Michigan Millers was committed to developing the system, incorporating a multi-variant rating engine, in house rather than deploying an off-the-shelf platform.

“We looked at buying products, then we looked at outsourcing the development. Ultimately, we believed we could provide a better product that we could obtain elsewhere by doing it ourselves,” says Jim Wieber, Scrum master for the Matrix project.

“We also showed that time frame and cost were in favor of in-house development,” says Smith. “We calculated we could get the development completed in half the time it would take if we went outside.”

Michigan Millers believed that agile development was essential to meet that goal. “Our old methodologies weren't going to meet our timeframe, particularly because we had always had trouble nailing down business requirements. In the past, we'd get bogged down in requirements, and sometimes we didn't deliver because we'd work too hard on minute details that weren't important. We needed a better way,” says Wieber.

The move to agile was driven by the development team. “They presented a proof of concept to our senior management, explaining the concepts and how we wanted to run it, and the need to partner with the business unit to deliver,” Smith says.

Although Grieve wouldn't recommend starting the journey to agile with such a large project, he says that core administration systems are a perfect fit for the development methodology. “In any environment there are variables you can't control—regulatory developments, changes in business opportunities, changes in the market. During a lengthy core system project, many things happen in the outside world which impact the project, and which the project doesn't have control over. Agile responds best to that,” he says.

Michigan Millers created development teams and broke down physical cube walls. They also leveraged the expertise of vendors for other platforms that were using agile in their own development, such as in StoneRiver's billing system. “We noticed that after StoneRiver began utilizing agile development for code releases, we were able to accept the releases more quickly,” Smith says.

Most important, the project team got the necessary resources from the business side. “Our personal lines unit gave up one of their best people to be the product owner. That was a huge commitment, but it was essential,” Smith says.

“There was a steep learning curve for both sides at the beginning, but the project team immediately saw the benefit. The business side saw that in every sprint we were developing what we said and that it met their needs,” says Wieber. Those sprints started at four weeks, and evolved to two as the launch date neared.

Smith explains that Scrum put project prioritization in the hands of business, where it belonged.

“The business knew exactly what our velocity was—how much work we could do. So if they knew we had the resources to do 100 story points but they wanted 150, they could either reduce the number of points or change the delivery date,” he says.

The policy administration system was rolled out to the personal auto line in March, 2010. “We hit the target for time, budget, feature set, and adoption rate with our independent agents,” Smith says. “It far exceeded everyone's expectations for a best-case scenario.”

AGILE CHALLENGES

Despite the successes enjoyed by agile proponents, there are challenges to its adoption. “It's not problem-free,” Grieve cautions. “Not only does agile break down the rigid development structure in terms of sequencing of tasks, but it breaks down the typical project management hierarchy. That's the biggest change companies have to be prepared to deal with.”

Sliger points out, “Often times the adoption of agile triggers a cultural change. Some companies have a 'values mismatch' and will fail at utilizing it, while others actually embrace it the change.”

The most important point is the original team you put together, explains Smith. “They are going to lay the foundation for everything else going forward,” he says. “You need to be sure they have the personal commitment and buy-in to their Scrum team. That has been hard to do—they have to realize that they succeed as a team and not as an individual. That's not something you can teach people, so you have to choose wisely.”

Agile requires a “flat” structure where project teams operate with a higher degree of autonomy than in directive-driven waterfall development.

“Ideally with agile development, you want autonomous teams that can make decisions about the best course for making the product. But sometimes, people are fearful of pushing that authority down,” Mellor explains.

He stresses that top-level support is needed for the cultural change—an observation that is unexpected given the under-the-radar manner in which agile took hold at State Farm.

“You can start [agile] at a grass roots level, but eventually you'll run into too many impediments as projects grow in scope,” Mellor says. “You need full commitment and resources.”

“You have to have a champion or multiple champions, and it has to be more than just a checkbook commitment,” Sliger stresses. “You have to be willing to remove the organizational roadblocks the team uncovers.”

Failing to provide high-level support can lead to “mini dictatorships” on one hand, or a directionless, hands-off approach on the other.

Sliger adds that insurers need to understand that agile is more than a checklist of steps to follow. “It's easy to focus too much on the set of practices rather than the philosophy and mindset that are needed with agile,” she says. “I've seen people try to take bits and pieces of the practice like it's a menu. You will get some of the benefits, but you don't understand the principles driving the value systems that these practices help to bring about. Without understanding that, agile is not going to work as well for you.”

“A company either does or does not have an agile culture,” Mellor says. “It does not 'do' agile.”

While the structure of projects is less rigid and daily stand-up Scrums add informality to project management in agile, there is a heighted focus on execution due to the increased number of iterative deliverables.

“Some of the developers thought Scrum was essentially 'cowboy coding,' where they documented a little bit and did what they wanted. But they soon realized that Scrum actually introduced more engineering processes and discipline than we had before,” Smith says. “They also realized that we are now expected to be partners with the business in an iterative and collaborative process.”

Managing that collaboration can be a challenge for companies and has given rise to a new breed of project management tools targeted toward agile.

Forrester points out that agile collaboration eventually requires automation because sharing status is time-consuming, particularly when a team is dispersed. Agile practices around testing, architecture, and build also require automation to be most effective.

Michigan Millers uses ScrumWorks Pro by CollabNet for project workflow, and the open source CruiseControl framework for a continuous build process and automated code deployment across different environments. “ScrumWorks was installed at the start of our first agile project,” Smith says. “We use the Web report feature to publish reports to a project status page. From there, everyone in the company can see the current sprint burndown chart, product backlog, velocity history, and more.”

CHOOSING TO BE AGILE

Whether agile is the right choice for a company or particular project should be driven by business needs. “If waterfall is working for you, don't change it, because it doesn't make any logical sense to do so,” Sliger says.

Defined development may well be the right choice in projects where requirements are stable or where speed-to-market is not critical. Waterfall's sequential process emphasizes requirements and design, and brings structure to the project. Stages have defined beginning and end points, giving developers clear indication of progress—and a comfort level associated with familiarity of process. The benefits of agile development are also more difficult to derive when the technology platform itself is not agile, such as with inflexible and legacy platforms that lack configurable capabilities.

Nevertheless, agile has established its place in the development world. “Agile will have strong staying power for many reasons, the most important of which is its proven successes,” Sliger says. “Even though agile is a modern philosophy, it is really built on ideas of iterative development that make sense to both business and IT and are certainly not new in the business world. I fully expect the movement to agile will continue to gain strength.”

“Your attitude toward agile should be 'Why shouldn't we do this?' However, there are a lot of insurance companies that ask 'Why should we do this?' instead,” Mellor says.

“From a competitive standpoint, there's nothing I'd like to see more than other insurers sticking with the old ways of development,” Mellor adds. “If you're not focused on continuous improvement and adapting your development organization to a cultural model of efficiency and production, it does nothing but benefit us as your competitor.”