Measuring for Meaning

My suspicions about quantitative measures were confirmed during a series of research classes I completed as part of my doctoral program at Drexel University. The first course in the series focused on quantitative research methods, in which we learned in nauseating detail the importance of controlling variance in our experiments and eliminating bias in our collection of data. Although the course was difficult, especially for those of us who struggle with statistics, most of the students felt comfortable with the underpinning philosophy of quantitative research: Real data is objective and can be analyzed to draw conclusions that are justifiable.

This is very similar to mindset in claims where file audits, cycle time metrics, and customer surveys are used to draw accurate conclusions about the business. However, anyone who has supervised an employee struggling to achieve these measures realizes that they can leave huge gaps in the landscape of meaningful evaluation.

The problem isn't with quantitative analyses per se, because without goals and objective measurements employees flounder and supervisors struggle. The problem is that quantitative measurements have come to be seen as synonymous with comprehensive evaluations. Although most supervisors realize that cycle time and average paid metrics don't tell the whole story, they are often content with using these numbers as an exclusive summary of an employee's performance.

Responding to performance issues with only a graph and a gut feeling does a disservice to employees as well as the company. In order to create meaningful and effective interventions that result in sustained changes in attitude and behavior, supervisors will need to master qualitative performance analyses.

Importance of Qualitative Measurements

As I mentioned earlier, my first research class focused on quantitative measurements, which allowed the professor to ease us into the practice of research design, but all bets were off for the second course. The professor opened class by asking us what we thought was lacking in quantitative research analysis.

You could see the eyes rolling back into each of our heads as we struggled to recall the notes and discussions stored in the darkness of our grey matter. It all clicked at once for us and collectively we shot out answers describing the barriers to good quantitative designs such as, sample bias, statistical insignificance, and confounding variables. We were on a roll; a sign of relief spread over us as we realized we had retained something from the last class. That is, until the professor interrupted our brainstorming session with the slight shaking of her head.

"You are listing the barriers to a good quantitative design," she reproved. "I am asking what a well designed quantitative study lacks." It took a moment to sink in, but thankfully it hit me first.

"It lacks meaning," I said. She nodded so I continued, "Even the best designed quantitative study doesn't tell you what the results mean. You cannot truly understand individual instances through statistical analyses." I noticed out of the corner of my eye that one of my classmates was shaking his head and clearing his throat in preparation to disagree.

Just as the last syllables of my sentence passed over my lips, he blurted out, "The numbers don't lie." The professor remained silent and as ranted, "You cannot argue with the numbers; that is the whole point of a properly designed experiment."

"But what do the numbers tell us?" I asked contritely.

"Whether or not the experiment made a positive impact on the group," he lectured.

I pressed him further. "But do we know why or even how it made that impact?"

After a few moments of silence, he reluctantly conceded, "Of course, I would have to investigate that a little further." At this point, the professor reentered the conversation, citing this as an example of the limitations of quantitative analysis. She then turned to my contrarian counterpart and inquired as to the methods he would use to investigate it further.

"I would just talk with the people who didn't do well and ask them why they didn't," he said. She reminded him that this assumed the people would even know why they did poorly; to be informative, he would have to use a structured approach to his investigation. She took this opportunity to introduce three useful qualitative measurement tools: in-depth interviews, participant observations, and interaction analyses.

In-Depth Interviews

Interviewing is an art, as anyone who has been responsible for filling a difficult position will tell you. Getting past the smoke and the personal bias to the real person in less than two hours is a skill that takes honing.

However, the smoke and bias continue, becoming even thicker with familiarity. Opinions solidify and patterns form that inhibits a clear perception of the individual's performance. Curiously, in-depth interviewing actually involves asking fewer questions overall, using open-ended inquires as in recorded statements that are tailored to the individual rather than the position. Despite the term, "interview," these conversations should occur in a natural setting, not in a cold conference room. When done correctly with a short list of targeted questions that can be communicated in a natural setting that allows the person to work through their responses while you listen without judging, in-depth interviews can help you build a sustainable rapport with your employees, who can then partner with you in recognizing their performance challenges.

Activity tool: Select an employee from your team with whom you are the most familiar. Create three questions about an area of professional experience you know they have but have yet to explain to you in much detail. Respond only when prompted and always with phrases that encourage them to continue with their descriptions. See what new information you can find out about this person and then use it to give them some new opportunities.

Participant Observations

Observations come naturally to almost everyone and yet because they are so natural they are so difficult to make in an intentional and systematic way. Observation, in this sense, should not be thought of as passive, fly-on-the-wall surveillance but rather as an activity requiring supervisors to step outside of their offices and into their employees' shoes like anthropologists who study remote tribes by participating in their rites and ceremonies.

All too often, supervisors who have come up through the ranks rely entirely on their past experiences as a way to gauge their reps' performances. Participant observation allows supervisors to step out of their roles and understand the position from a fresh perspective because processes change and systems evolve. Simply relying on an idealized version of the past is no way to manage people in the present.

Activity tool: Participate in a task that everyone on your team complains about, such as ordering a police report or finding information in the policy management system. Try to see the process from their perspectives by paying particular attention to your feelings and frustrations during the task.

Interaction Analyses

In order to triangulate their findings from in-depth interviews and participant observations, researchers often use two types of interaction analyses: kinesics and proxemics. Both are ideal for uncovering the subtle messages embedded in non-verbal communications. Kinesics studies gestures and expressions, while proxemics involves the study of space, such as the distance established between parties in conversation and even the arrangement of furniture.

As with in-depth interviews, claim supervisors have a unique advantage with interaction analyses because of their experience with investigating claims. In their training, many claim adjusters learn how to identity slight changes in tone, volume, and rate of speech, as well as gestures and expressions, to anticipate objections and avoid conflict.

These skills can be used in personnel management. However, this technique in particular is susceptible to abuse by untrained and malevolent supervisors who seek only to validate lingering biases. Despite this, interaction analyses have the potential to provide unique insights into individual nuances in behavior and task completion in an unobtrusive way, which avoids unnecessary stress on employees.

Activity tool: Select an employee and observe the items in and around his work area. Reflect on how this use of space can give you insight into his personality. Remember, observe for understanding, not judgment.

The Bottom Line

Healthy organizations, like healthy organisms, are able to adjust and adapt to environmental changes. Changes like new products, processes, and senior management, as well as mergers and acquisitions, are all examples of second-order change—change that is radical and can sometimes be catastrophic if initially mismanaged.

Without a keen seen of qualitative measurements, supervisors will be unable to guide their teams through radical shifts in the business because all change happens at an individual level. By complementing an exhaustive knowledge of insurance principles and claim processes with an authentic sensitivity to the verbal, non-verbal, and environmental messages contained in every job interaction, supervisors and managers will be able to measure for meaning and thereby guide their people toward the right outcomes more efficiently and ethically.