Embracing Skepticism

Understanding and interpreting data can be a challenging task, especially when the stakes are high. Imagine being in a situation where the analysis of customer satisfaction scores could influence an entire company's performance evaluations and bonuses. This is a story of how nurturing a skeptical mindset can reveal hidden truths and lead to more accurate conclusions.

The Setup

You find yourself on a software training assignment with an electric utility company. Your role is to help analysts calculate customer satisfaction scores more efficiently using software. Each month, the company surveys customers and uses a complicated weighted average to evaluate its performance. These scores are critical—they determine whether employees receive a bonus or undergo remedial training.

From the outset, you sense that something isn't quite right with the analysis. Despite not being a management consultant and lacking significant clout, your intuition pushes you to dig deeper. You request 36 months of satisfaction scores and plot them on a graph with the analysts. This exercise, initially meant to teach software skills, soon reveals inconsistencies.

The Skeptical Mindset

Here’s where your skeptical mindset comes into play. Questioning the formula used for the scores, despite it not being your job, takes bravery. The formula is a big deal—two weeks' pay hangs in the balance. But you press on, driven by curiosity and a desire to uncover the truth.

Analyzing the data over a longer period, two significant patterns emerge:

• The scores appear to bounce within a narrow band, never straying outside a certain range.
• The lowest scores consistently occur during the same two months each year—early spring and early fall.

You question the team about the seasonality of the satisfaction scores. While seasonality is expected in utility usage, it hadn’t been considered for customer satisfaction. This insight leads to a deeper investigation.

One time period with low scores across all three years is October or November. This isn’t a particularly hot or cold time, but it’s about 90 days after the peak summer demand. Revisiting the raw data, you notice a significant spike in low scores roughly three months after high-demand months. This period coincides with increased shut-offs for nonpayment.

The Human Element

Imagine sitting in a cold, dark living room after your power has been shut off, and receiving a survey call about your satisfaction with the electric utility. Naturally, your response would be extremely negative. This human element changes everything and offers a new explanation for the periodic dips in the weighted average scores.

Skepticism isn't about negativity; it's about curiosity and a willingness to explore beyond the surface. By questioning established formulas and considering broader contexts, you can reveal insights that might otherwise remain hidden. This mindset not only improves data analysis but also leads to more informed and effective decision-making.