At Witbe, we care about video quality; we have for the last 10 years. We have worked on the Witbe Video MOS Algorithm tirelessly, to make it more accurate at reflecting real human’s perception of a video stream, on any device and through any network. With their article “Toward A Practical Perceptual Video Quality Metric“, Netflix clearly raised the general awareness on the importance of automated methods for measuring the perceived quality of a video. So, Netflix, thank you. And welcome to the party.
10 years ago, we started to believe that we could teach our Robots to experience a video like a human would; and many believed that it could not be done then. They were used to PSNR and other usual mathematical analysis for signal degradation measurements, which often require the original video stream. At Witbe, we wanted to go one step further and create an algorithm that would work like a “Golden Eye”: watch a video holistically, in real time, without the original stream, and instantly evaluate its degree of degradation.
It was not a surprise to see converge the approaches we took to solve this complex computer vision problem. First, create a data set that matters and set up a proper testing environment, following BT-500 recommendations. Then, identify and compute key performance indicators that are relevant. Finally, use some machine learning techniques to translate these independent metrics into a systemic one that makes sense for a real human. For the Witbe Video Mean Opinion Score (WMOS), we chose to focus on three main artifacts: blockiness, blurriness and jerkiness. We believed it was important to measure not only compression and upscaling degradations, but also transport artifacts, because not everybody works in TCP, on lossless transmissions networks.
And yet, Netflix’ and Witbe’s algorithms do differ in results and philosophy. Netflix VMAF was designed to solve one problem: the encoding. At Witbe, we have a different objective. To reach the ultimate viewer’s experience, the problem needs to be looked at on a systemic level. By having a broader scope of measured artifacts, the Witbe Video MOS not only detects encoding issues, but also transport ones. And since it does not require the original stream, it works just as well on any kind of content: live or on-demand. It provides a powerful way to measure the quality of any video stream, as it could be perceived by real end-users.
For 10 years, we have been working on the Witbe Video MOS that is now used by hundreds of our customers worldwide. Our experience—and the new machine learning techniques that appear every day—helps us keep up with the new technologies that constantly emerge (H264, H265, IPTV, OTT, etc). 10 years later, here we are. We taught our Robots to measure these complex artifacts and understand how they could impact the perceived quality by real humans.
So thank you Netflix for joining forces in showing the way and for demonstrating we chose the right approach. Thank you for helping us promote the importance of measuring the true quality, the one that is perceived: the Quality of Experience (QoE).
As we say here:
It’s all about User Experience!