After sitting in on usability testing conducted by the IT teamon website design, JoAnna Carey, marketing development manager forForemost Insurance Group, came away wanting more information thanthe traditional testing sessions could provide. To determinewhether its independent agency force was using the carrier's portalefficiently, Foremost would invited agents to look at sketches ofwhat the site might look like and asked for feedback on the userexperience.

"I kept hearing questions about how to quantify thisinformation, but that was not something we were able to do," shesays. "I began looking for solutions with electronic testingcapabilities. I had a specific idea in mind to show an image of thewebsite and provide a picture for agents that would ask them wherethey would click in a particular scenario."

Carey knew about heat map technology where testers would show animage to the audience, the audience would then click on the image,and the testers could determine what caught the user'sattention.

"That would give us a general idea, but we still wouldn't beable to quantify the responses," says Carey. "We were doingone-on-one interviews with agents, but there was always thequestion of how many agents made that response. Was it thatperson's opinion or anecdotal information?"

What Carey wanted was to capture a larger group and quantify theinformation.

"If we survey a thousand agents, how many would click on theexact spot we wanted them to click?" she asks.

Carey came across the software solution provider Qualtrics andits Research Suite solution, which would allow Foremost to post animage for the audience to view.

"We could show agents a static image—one frame of this newwebsite—and ask them a question such as: If you wanted to make apayment, where would you click? We would see where the agent wouldclick and out of a thousand agents we could learn how many wouldclick exactly where we needed them to click," she says. "We couldsee where they were clicking that would take them to the wrongplace and with a visual picture we could show the web design teamto find a way to address any problems."

Foremost Insurance is a business unit of Farmers Insurance, butunlike Farmers, Foremost works with independent agents. Farmersreported approximately $19.1 billion in gross written premium in2012 with Foremost accounting for about 10 percent—or $1.9 billionin GWP—of the enterprise.  Foremost offers specializedprograms for auto, manufactured home, specialty homeowners,landlord or rental property, seasonal or vacant property, motorhome, travel trailer, motorcycle, off-road vehicle, snowmobile,boat, personal watercraft, collectible auto and flood. The currentnumber of independent agent active locations for Foremost is85,695.

Usability testing is both an art and a science, according toCarey, and Foremost uses a blended approach, but the Qualtricssolution gave the insurer a way to examine at the user interface(UI) and see where the agents were falling.

"This allowed us to test a specific page that we were using andit was just one specific snapshot to see if we were moving in theright direction," she says. "We use the comments we receive fromour agents as well when they call the help desk about the websiteor other issues."

The survey, conducted in 2012, showed high accuracy in thescenario Foremost presented, according to Carey.

"The UI offered a different look and feel than they were usedto," she says. 'We wanted to know if they could find the things wewere trying to make very obvious. In most cases they were. Theywere clicking in areas that would take them in the right directionso they could find what they were looking for."

Carey did not receive much feedback from the agents that tookpart in the testing, but she reported receiving one email from anagent who expressed thanks for being included in the test. Thatagent had done usability testing before, but not in an electronicmanner.

"I think it was a positive experience for the agents becausethey didn't have to sit with us for an hour and talk through theinformation," says Carey. "They were able to see it and provide ussome direction in just the 10 minutes that it took to complete thesurvey."

As for the future, Carey would like to buildin follow-up questions—which the software allows—so if an agentclicks an area during subsequent testing a follow-up open-endquestion will allow the design team to determine what the users areexpecting.

That would allow us to get some of the qualitative informationwithout taking up too much of the agent's time," says Carey. "Therewill always be the need for qualitative research because we have tohear the agents in their own words, but this gives us a morespecific number, so we can use that to guide us in our futurediscussions to pinpoint the areas where we need to have morein-depth interviews with our agents."

What pleased Carey was there was no large learning curve withthe Qualtrics software.

"I watched a one-hour demo and would have been able to get offthe demo and program it," she says. "There was little to notraining needed. Someone on our team designed the survey and welooked at the back end of the reports to determine what more weneeded. It was point and click. Any questions we had, the supportteam was there."

As for the drop rate on surveys, Foremost tracked how manyclicked in to try and take the survey and how many dropped out andif they dropped out where were they in the process.

"We found that among those that clicked in there was probably acuriosity factor so if they dropped out it was at the firstscreen," says Carey. "These are independent agents that work withour website and our competitor's websites, so they are comparing usto the other sites they are on. That's why it was nice to hear fromthem to learn how we compare."

Foremost wanted to keep the survey short to show the insurer wasrespectful of the agents' time. The test site would update everytime an agent responded and the heat map was building before theireyes.

"As fast as we gathered enough information, we were able toshare it with our design team along with the percentages they wererequesting," says Carey.

In the short term, Carey believes this was a big win for theresearch group because it created a way to solve the businessneed.

"There is something exciting when the technology catches up tothe business needs and when you can provide something that isexactly what was requested," she says. "Plus, it gave immediateinformation to the design team to help them think through some ofthe questions they had."

In the long term, Carey believes there are many other areasForemost can use the technology.

"We are thinking of ways to use this tool creatively in otherbusiness situations," she says. "It's opened our perspective toknow that the solution may not present itself in exactly the wayyou think it will. We have to be open-minded enough in our thinkingto find long-term solutions when we are using tools like this."