This past weekend a number of us were focused on a really important annual prime time television event (the Puppy Bowl, of course). Turns out other people out there were watching some other sporting event, which leads to the rest of this story.
At Wanelo we care a lot about the positive experience of our users, and central to a great user experience is a speedy site to use. While we track a plethora of metrics and use a number of alerting tools to ensure that we know when anything untoward is happening, lately we've grown enamored of Circonus and its ability to alert on derivatives -- the sort of thing that is extremely difficult if not impossible to do in Nagios.
Super Bowl Sunday afternoon, we were unpleasantly surprised to receive an alert that the rate of product saves had suddenly dropped. Usually this foretells some disaster in the infrastructure, as the ability to post, save and resave products is one of the most critical aspects of Wanelo. Here's what we were seeing:
That big dip in the middle was the cause of our alerts.
Checking the site and our other monitors, everything seemed to be working correctly. Yesterday morning we realized that this timing aligns perfectly with events at the Super Bowl. Our alerts went off the when the halftime show began, followed by a steep incline when the power went out at the stadium and product saving ramped back up.
-Eric