Yesterday – the 29th of April 2021 – the NFL Draft started; a three-day event watched on average by 8.4 million viewers in 2020. Because some of our customers and collaborators are fans of the NFL, we decided to use just a few of the robots of our Witbe Cloud OnDemand platform to monitor the video quality of this live event and write this article. During this live event, we monitored the ESPN app on four devices – Apple TV, Roku, Xbox One, and Samsung Smart TV – as well as the ABC app on Apple TV and Roku. In each instance, we’ve measured the Video and Audio quality of the live, which showed us some pretty interesting things. The videos that you will see in this article have all been recorded by our robots during the runs.
Overall, we’ve seen a pretty good quality of experience across the different applications and devices. The Disney operations team did a good job at managing this big event. However, there have been a few interesting issues that arose during our automated monitoring.
Issues on Roku: Ad insertion error
On Roku, we’ve seen a problem that can be very hard to catch without the right monitoring tools. We opened the ABC app, navigated to the Live, clicked watch, and then waited to watch an ad before accessing our content, and waited… and waited. As you’re going to see in the following video, when we launched the live, the app had a problem and did not properly insert the ad. For 45 seconds, we had the same screen looping and not even a single sound to go along with it. From our measurements, this was the most impactful issue.
Issues on Apple TV: App crashing and two audio sources playing
When monitoring the event on Apple TV, we saw two issues that can be pretty detrimental to the user’s experience. The first one happened when trying to watch the event on the ABC application. Our robot tried opening the app to access the live stream but was instead greeted by a screen you want to avoid your customer seeing: “There appears to be a problem.” Thankfully, It was a one-off and the next run was working correctly.
On the ESPN app, we haven’t seen a crash like that, but we have encountered an issue with the audio. At some point, when opening the application, we could hear two different audios playing at the same time and overlapping each other. Sound On!
Issues on Xbox One: Strange behavior on the thumbnail videos playback
The Xbox One generally had a good experience. But at some point, we did notice that there was a strange behavior happening with the thumbnails video. When you first go to the screen with all the featured live, you can see that all of them are playing. Hovering over one of them allows you to get a preview of what is playing with the sound. However, after a little while, all the videos stop playing, and hovering over one of them does not play the video or the sound anymore. Check out this recording of the run where we were able to catch this strange behavior:
How to go further with test automation?
While the overall experience is generally good across these devices and applications, we were able to see a few issues that can impact the quality of experience of end-users. Because of how infrequently those errors appear, it is important to put in place both an automated QA and a live monitoring in order to really see how it works.
Luckily, you just have to send us a message via the live chat in the bottom corner or through this contact form in order to get access to our Witbe Cloud OnDemand platform. Like that, you’ll be set to ensure the best QoE for your next special events!