Operators promise the highest video quality even during peak times
5G has never been more upon us following Tuesday’s announcement by Apple and Verizon! The new technology standard for broadband cellular networks has already been deployed by mobile telecommunications companies in more than 378 cities worldwide. But what exactly does that bring to the users? Or what exactly does this mean for its users?
Mobile operators and MSOs make many promises when it comes to the new 5G networks. They claim that users will be able to benefit from a better overall experience. This includes the capacity to reliably stream videos in the highest resolution, even during peak times.
Thanks to our Quality of Experience monitoring robots, we can verify if those promises hold true. So that’s exactly what we decided to do!
Real “in-app” Video Quality improvement lower than expected
The first thing we saw is that the 5G network is boasting raw download speeds of almost four times its 4G counterparts. But to truly know the quality of a service, we need to look at it in conditions closest to the end-users’. In order to do that, we monitored how a device was playing the same YouTube video on a Samsung S20 5G connected in 5G and in 4G.
As typical technology fanboys, we were really expecting Youtube to stick to the highest “almost 4K” 1440p profile on the latest Samsung S20 5G phone.
We were surprised to see that the video streamed at 480p resolution during peak times for the whole duration of the analysis, even on 5G. It was exactly the same pattern on 4G and 5G with no visible quality gain. It appears the Youtube app does not automatically select the highest profile in 5G.
But can it be played properly if we force it in 1440p? Spoiler alert: It can!
We measured perfect video playback session (0 rebuffering measured with VQ-ID) in 5G and … also in 4G.
And because it would have been underwhelming to conclude that 5G didn’t offer any improvement compared to 4G, we did find a KPI which performs better: Initial buffering time when 1440p profile is forced.
Our robots have been able to play their 4K contents half a second faster and with more regularity in 5G (in 4G it exceeded 5s during 7% of the runs). One could have expected a bigger improvement reflecting the 4x improvements in raw download speed. But as Apple said during their October events, it is important to optimize all layers: hardware, driver, OS, softwares, servers, etc. That’s why it is always very important to rely on true user experience with real devices and not raw download speed only.
5G deployment is still at its debut and the good news from our findings is that it was never less performant than 4G which can sometimes occur when the technology is new.
Let’s measure how it evolves over time, and how each operator performs on their own network.
Methodology and set-up
To execute this 5G benchmarking, we performed a series of tests on a Samsung Galaxy S20 device connected to our Witbe “Video & Media” robot hosted in one of our offices. Our robot performed a series of tests with the device connected to the 5G network of a European operator. We compared it to the same device at the same time with the only difference being that it was connected to the same operator's 4G network. The mobile phones were set in a real-world outdoor situation to have good network coverage. Next, we verified raw download speed to make sure the 5G provisioning was fully effective.
With this methodology, we measured and verified the following KPIs:
- Video initial buffering time: The time between clicking on the play button and the video starting to play. When a pre-roll ad was detected, the initial buffering time is not taken into account as to not skew the results.
- Witbe VQ-MOS Score: Witbe Video Quality Mean Opinion Score is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of displayed videos. It works without referential or previous knowledge of the video, and in real-time. It is based on three main artifacts: jerkiness/rebuffering, blurriness, and blockiness/pixelation.
- Witbe VQ-ID: Witbe’s Video Quality Incident Detection measures all quality-related defects during the video playback and scores the percentage of clean video. Focus is done on rebuffering events, drops in the audio of video quality including pixelation/macroblocks or excessive compression/blur. Like Witbe VQ-MOS, Witbe VQ-ID is non-reference based and works in real-time, and on the decoded displayed streams.
- Quality profile selection: The quality profile that was automatically selected by the player after 1mn of uninterrupted video playback.
The KPIs were measured automatically every 5 minutes over a Two-Day period on Wednesday, September 20th, for a total of more than 400 viewing sessions. Every session included a 1-minute Witbe VQ-MOS and Witbe VQ-ID analysis, representing a staggering 200k+ metrics measured!
All graphs and other visual representations of KPIs in this QoE Snapshot were generated by Witbe Datalab, our restitution interface for fault finding, root cause analysis, and advanced analytics.
What next?
You can’t improve what you don’t measure, and the best way to see for yourself is to request a demo and understand how your 5G network performs on over any kind of real-life application.
When an issue is detected, a great way to find the root cause and pinpoint bottlenecks in the network is by using Witbe Smartping.
Because 5G networks should also bring better performance for cloud gaming and edge computing, let’s measure how the users experience them with our robotic player showcased in our Google Stadia snapshot.
As a service provider, our technology allows you to monitor the newest immersive experiences that will emerge thanks to 5G. The best part is, because we test and use real devices, Witbe robots are already compatible with your network and are ready to ensure that the quality of experience of your 5G is top-notch.
Your customers will thank you!