The Super Bowl was 99.999% perfect

With a viewership over 100 million people, the average Superbowl lasts 3 hours 44 minutes,with an Average Minute Audience (AMA) of 2.02 million, peaking at 3.1 million concurrent video streams.

Undoubtedly, the engineers used cutting edge technology and did absolutely everything in their power to prepare the broadcast on the multiple platforms that consumers demand today. Here are some more numbers from NBC to show the scale of the event:

  • 264,000 Feet of camera and microphone cable
  • 130 Microphones
  • 106 Cameras
  • 100 + Hours of coverage from NBC Sports Boston and NBC Sports Philadelphia
  • 50 Miles of camera and microphone cable
  • 14 Mobile units
  • 10+ NBCUniversal shows onsite or reporting from Minneapolis
  • 7 UHD 4K cameras
  • 2 SkyCams (including “High Sky”)
  • 1 ND1 – NBC Sports’ state-of-the-art, remote production unit
  • 9 streaming outlets — more than any other Super Bowl — NBC Sports app, NBCSports.com, NBC.com, TV Everywhere, En Vivo app, NFL.com, NFL Mobile app from Verizon, Yahoo Sports app, go90.

All of this is best in class technology, assembled by the smartest people in the video streaming industry, armed with huge budgets. So, it must surely turn out to be perfect. Except that it wasn’t.

The term “Five Nines” refers to 99.999%, meaning high availability, “a characteristic of a system, which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period.

According to Ericsson CTO Ulf Ewaldsson, “building individual network elements to certain standards and having them certified as “five nines” (99.999%) telco-grade reliable isn’t enough any more. Instead, that level of reliability needs to exist across an entire network and it’s up to the industry to deliver those capabilities and that mentality: “It’s all about end-to-end performance now,”

If each of these 9 streaming outlets relies on 25 vendors and each of those vendors uses “telco grade” technology with high availability, with a service level agreement (SLA) of 99.999% uptime, it is simple math that 99.999% to the power of 223 equals 99.777%.

And indeed an an uptime of 99.777% is equivalent to downtime of 30-seconds during a 224 minute game, not to mention the 10 hours of pre-game and post-game footage.

A 30 second ad spot was a black screen. NBC explained that equipment failed, according to a tweet:

Untitled1That explanation wasn’t satisfactory to the Baltimore Sun, who were left with questions, “Communications technology isn’t some vast mystery beyond the understanding of peasants like us. The ‘equipment failure’ talk was OK in the 1950s when TV was new, but not today. Not even close…

“What actually happened?

“Did someone mess up?

“Who and how?

“Did a piece of equipment fail?

“If so, what equipment and how? We’d like to know…” (Baltimore Sun).

NBC also said the failure didn’t cost them anything (Marketwatch). However CNN notes, “By saying no ‘commercial time’ was missed, NBC might be indicating that network promos were supposed to air during the blackout time.” Some on Twitter captured the video and suggest it was a slot meant for local ads (SI):

 

Untitled2This was only the first of several issues. By now you’ve probably heard that there were other issues with the broadcast:

  • Two virtual MVPDs – Hulu and PlayStation Vue – faced some technical snafus during the game … latency that puts viewers well behind the regular broadcast…”
  • Users of the NBC app or NBC’s livestream on its website also experienced other difficulties tonight, such as seeing several commercial breaks where the screen would reach a “Coverage will resume momentarily” message. (Verge) Unlike the over-the-air broadcast, the app/livestream was managed by “Playmaker Media, a technology service that provides “end-to-end support” for partners that are looking to deliver live and on-demand streams,” who also will be delivering the Olympics. (Multichannel)
  • Viewers couldn’t hear Justin Timberlake and complained about audio levels during the halftime show.
  • Other fans said they missed the end of the show because their DVRs did not capture the full episode thanks to the Super Bowl running over. (Fox News)
  • After the Super Bowl ended. The most-anticipated “This Is Us” episode yet aired after the Super Bowl and did not start until about 10:45 p.m. (Fox)

That is not to say that the technical staff and engineers didn’t do a wonderful job. They did an amazing job! However due to the sheer scale, and unmanaged nature of streaming over IP, there are bound to be technical issues even when you’ve guaranteed high availability of each individual component.

End-to-end is the exact methodology that Witbe uses to give operators control back. Witbe’s automated robots proactively monitor from edge to core, to provide meaningful data for quick and informed root cause analysis, and fast resolution times.