12 Analytics Lessons from It’s Always Sunny in Philadelphia

It may seem like a stretch to pick up anything of cognitive value from a show centered on selfish incompetence, but who doesn’t love a bit if irony? There is a single Sunny episode that truly embodies the most essential tips to solving mysteries, including those found in web analytics.

Obviously, I’m talking about Who Pooped the Bed?

1. Make Sure You Know It’s Actually a Problem

What is normal? What should be reasonably expected 1? The answers to these questions vary wildly from site to site. The more you can anticipate shifts from seasonality, day-of-the-week variability, and the everyday ebbs and flows that keep performance from staying flat, the less time you’ll waste chasing after a problem that doesn’t actually exist.

2. Be Equally Concerned About Unusually High Performance

We rarely hesitate to dig into why performance took a turn for the worst, but what about when stats look eerily positive? We tend to get a little lazy when things are going unexpectedly well and assume it’s because we are awesome. While this can certainly be true, what happens when it’s not?

What goes up often comes down. Failing to understand why performance went up all of a sudden may lead to you reactively explaining the regression towards the mean. Trust me. It’s much better if you can get out in front of it to set expectations.

3. Keep a Log of Findings and What’s Been Ruled Out

Have you ever buried yourself in data, stepped away for a while and returned only to lose your place? Heck. I’ve lost track of my progress without even going anywhere 2.

Having to retrace your steps can be frustrating and a huge time suck. Whether it’s just a notepad or something more complex, keep track of your progress, especially if you’re doing it in phases.

4. Give Yourself Enough Time

When I first started in the agency world, I would do one report at a time. That approach certainly had some appeal. I was able to completely focus on one client and didn’t risk confusing details from one report with another. However, it always seemed like I inadvertently saved the toughest for last. Of course, that’s also when a deadline was breathing down my neck, giving me little time to troubleshoot and develop insights.

This is why I no longer adopt that process. Instead of doing one report entirely before starting on the next one, my advice is to pull data for every report at the beginning 3. From there, you can quickly skim each one to get an idea of which might be most problematic. That way you can allocate an appropriate amount of time for it at the start.

5. Don’t Limit Yourself to Quarters or Months

Data fluctuations don’t care about calendars. Just because a problem was presented to you as this month versus last month doesn’t mean performance magically shifted on the first. So, before you start comparing date ranges to dig into the issue, get a daily trend of the metric in question to get a more accurate picture of when the zig started to zag.

6. It’s Not Always Just One Thing

This is one of the most exhausting truths with web analytics. It can be so satisfying to find the culprit and equally deflating to later realize it doesn’t explain the entirety of the problem. Sometimes it can take three to four explanations before you can allocate for all of the shift. Here’s what I mean.

Let’s say you received 10,000 visits last month and 7,500 visits this month, which is a 25% decrease. If you were already expecting a 10% drop, or 1,000 visits, due to seasonality, then you should really be trying to find the remaining 1,500 visits.

Once you find a new factor, quantify it and subtract it from the remainder until you get (close enough) to zero.

7. Use a Control

A true control is pretty rare to find when looking through historical analytics data. However, the concept of isolating a variable by comparing it something that should be immune to it can be a big help in these situations. The most common applications I’ve seen are among traffic sources, geographic locations, devices and content groups.

As an example, let’s say your hunch is you’ve been hit by a Google penalty. This would first assume the problem is unique to organic search. A helpful way to validate that assumption is to compare organic search trends to another channel 4. If you truly have a problem specific to organic search, the other channels shouldn’t follow suit. Or you can get even more granular by comparing Google to all other organic search traffic.

8. Use Data to Find the Truth, Not Support What You Hope It Is

I’m not here to give you a morality lesson 5. It’s not my place to dissuade you from spinning the results. However, before you decide how you want to position your findings, shouldn’t you at least know the truth yourself?

9. Sometimes the Data Lies

What you see is sometimes not what you get. Issues like tracking hiccups, dark traffic, and broken filters add an extra layer of complexity to problems complex enough on their own. The sooner you identify these issues, the sooner you can understand and eventually solve the problem.

Making sure you have a good understanding of how analytics platforms categorize traffic and commonly associated problems is a good place to start.

10. Get a second opinion

The deeper you get into an issue, the easier it is to get a little turned around and oblivious to something right in front of you. A second set of eyes can go a long way in preventing this. Don’t worry about their lack of knowledge around your particular website or situation; coming in blind can be an advantage.

11. Be Your Own Devil’s Advocate

As soon as a plausible idea creeps into your head, it’s actually really easy for it to cloud your judgement. The more you’re confident in your conclusion, the more you’ll naturally downplay any signs of it being a non-sequitur. This is essentially choice-supportive bias.

How do you avoid it? As you become increasingly sure of your hypothesis, ask yourself, “If I’m wrong, how could I tell?” Literally try to prove yourself wrong.

12. The simplest answer is often the best

Enough said.

Conclusion

This stuff can get complicated and unfortunately there are no fool proof, step-by-step guides (although I think this traffic decision tree can be helpful) that uncover every answer. Whether you’re getting ready to start your next report or another episode of Sunny, I hope this gave you some new perspective to take along with you.