Witbe Snapshot - A deep dive into the video conferencing world
This article is a deep dive into a ranking done in partnership with the Journal du Net in which we compared 4 major video conferencing platforms: Microsoft Teams, Google Meet, Cisco Webex, and Zoom. You can find the ranking on their website.
Introduction
After first partnering with Le Journal du Net to compare french internet service providers at the beginning of the millennium, we are happy to partner with them again to compare the availability and performance of 4 major video conferencing systems - Google Meet, Cisco Webex, Zoom and Microsoft Teams - a great addition to the feature comparison that they first did last month. After measuring the availability, performance and quality of each service, we were able to compile an overall ranking to determine which one is the best video conferencing application - today. The full ranking, general and per category, can be found on the JDN website. At the end of this article, you will be able to read about the methodology that we performed to establish this ranking.
Deep dive #1: How to explain the availability errors occurring on Teams
One of the most surprising results of our analysis was the poor availability of Teams. 19% of the sessions we tested saw a guest unable to join the call. As the Witbe robots record the Audio and Video of each tests, it is easy to see the error. Here is an example of a session in which the problem occurred:
On this video, we can see that the application on the Guest side waits for the video for 30 seconds before failing.In addition to the video and the audio, Witbe robots also record the details of all the http requests between the application – Teams running on Chrome – and the server, hosted in Microsoft’s cloud. These interactions are aggregated in a HTTP Archive Format (HAR) file that we can directly inspect in Witbe’s restitution interfaces.To understand what the problem is, we can compare the HAR file of a session in which the service was available with the HAR file from a session in which the service failed. Here you can see the extract of the HAR file of a test that was successful, followed by the extract of the HAR file of a failed test.
Successful test
Failed test
In these extracts, we highlighted 2 things. First is that the GET command on the URL « user/[id]/endpoint/[id]/events/poll” took 40.02s when the session fails, whereas it only takes 125ms when the sessions succeed. Secondly, when the connection fails, the POST command on the URL “controlVideoStreaming” is never sent by the application.Thanks to Witbe’s restitution interfaces, we can confirm that all the 53 errors that occurred concerning the availability of Teams were linked to the same problem of communication between the application and the server. We hope that this application problem will be corrected in future updates. If Teams does fix this issue, it will surely impact their position in the global ranking. As such, we will be following their progress with great interest and very closely.
Deep dive #2: Google Meet and its record establishment time
We were also surprised by how fast Google Meet’s connection time was. On average, the call was created in 530ms whereas Microsoft Teams – the runner-up – took 1.8s.In order to understand this impressive result, we wanted to analyze the network infrastructure used by Google for its video conferencing service. We used Witbe Smartping to map the complete network infrastructure used by companies for their video conferencing services.
Witbe Smartping is a non-intrusive algorithm that can map a delivery chain and pinpoint bottleneck, even in unmanaged networks. This patented technology allows to visualize the network routes taken for services delivery, allowing easier troubleshooting and enforcement of Service Level Agreements with third parties.
To better understand the investments made by Google, let’s start by looking at the infrastructure that Cisco did for Webex:
When we first look at a network infrastructure, the first KPI to check is the RTT Delay which is the time it takes for the application to obtain a first response from the server. This is purely related to the network and represents the latency that can exists in it.For Webex, we can see that there is a very good RTT Delay of 15ms, on average. This reflects of a properly sized network for this service. Another important aspect is that this KPI is very stable over time, varying between 15 and 17ms. A bigger variation may introduce problems with the application.The Smartping graph detail maps the network used by the service between the robot (on the left) and the Webex servers (on the right). When we look at this mapping, we discover a relatively simple infrastructure, containing only one Webex server.Let’s now look at the network infrastructure of Google:
Not only do we see an RTT Delay extremely low and stable that never exceeds 4ms, we also discerned a network much more developed with eight servers available in redundancy to distribute Google’s video conferencing system. This exceptional infrastructure is one of the reasons that explains why Google Meet’s establishment time is so low and demonstrates the power of Google.Sadly, this impressing feat is tarnished by the establishment time of 5.6s for the guests – a KPI that we believe to be very important for real consumers. However, since it looks like the infrastructure is not the cause of the problem, we hope to see this KPI evolves in the future. Google may rank better by, for example, optimizing portions of their application.
Deep dive #3: How to measure the video quality of a video conference
The final aspect that we wanted to dive into for this snapshot is dedicated to the analysis of the video quality, made by our famous VQ-MOS algorithm.
Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential or previous knowledge of the video, and in real-time. It is based on three main artifacts: jerkiness, blurriness, and blockiness.
The first indicator that influences the quality of the video is the blurriness, representing mainly the quality of the encoding and the bitrate used to stream the video (the higher the bitrate, the lower the amount of blur in a video). The more a video is encoded in an aggressive way to be lighter on the network, the more it risks being blurry.When we look at the percentage of blurriness on Webex, only 9%, we could say that they probably have a very good encoder or that they choose to stream the content in the higher resolution. And therefore, that they have the best experience in terms of video quality.But the blurriness is not the only parameter to consider when we transfer a video over the internet. We also need to consider the way the video is transported throughout the network, as it can induce blockiness or rebuffering. Indeed, when packets are lost in the network, blocks can appear on the screen.As such, when we look at the percentage of blockiness on Webex, 42%, more than the 3 other platforms, we realize that the quality/network ration Cisco used for their video conferencing system seems not properly sized. It looks like they select a bitrate that is too high for their infrastructure to handle.As we saw previously, this is another indicator that the network infrastructure put in place by Cisco does not compare to Google’s: Webex is much more prone to blockiness than Meet.Let’s also note that we did not measure any blockiness on Zoom. This is not necessarily due to a perfect infrastructure, but most likely due to new techniques that enable them to remove the blocks during the transport of the video. This can be done by having the application perform new advanced image processing techniques. The use of these new techniques works well with the innovative image that Zoom has, which is also the only platform that forced us to use their application instead of the Web service.
Methodology
For this ranking, we used 2 Witbe robots:
- 1 at Nanterre, hosted in our offices
- 1 in Paris, hosted on our cloud infrastructure
Both robots were connected to the Internet through a standard consumer connection to be as close as possible to the real users’ experience. A speed test measured a download speed of around 100Mo/s on both robots.With this setup, here are the video conferencing platforms we measured:
- Google Meet
- Microsoft Teams
- Cisco Webex
- Zoom
To measure these platforms, one robot acted as the organizer, while the other acted as the guest. The organizer robot would connect to the service, start the call, and accept or invite when possible the guest user. It would then play a 1-minute-long video of a classic group meeting. The video would be pre-recorded to maintain the same conditions throughout every test as close to the real environment as possible. The guest robot would connect to the service, join the call, and watch the video streamed by the first robot.We measured the following KPIs:
- Availability for the organizer: measures if the organizer can start the call
- Availability for the guest: measures if the guest can join the call
- Establishment time for the organizer: measures the time it takes for the organizer robot to start the call
- Establishment time for the guest: measures the time it takes for the guest robot to join the call
- Witbe MOS Score: a score representing the video and audio quality of the stream, measured with our mean opinion score (MOS) algorithm.
- Witbe VQ-ID: measures in real-time any incident that can occur during the streaming of a video, such as loss of audio or freezes.
Tests were run during a full week, from Monday, November 2nd to Saturday, November 7th. The robots tested each platform every 15 minutes, for a total of 2400 video conferences tested.Finally, we established a scoring system to rank these 4 platforms. In order to calculate the overall score, we first established a ranking for each of the main sections:
- Best availability of the video conferencing service
- Best establishment time for the organizers
- Best establishment time for the guests
- Best video quality for the guests
These four categories best represent the expectations that a real user would have of a video conferencing service.Each platform was attributed points based on where they ranked in each categories – 100 points to the 1st, 75 points to the 2nd, 50 points to the 3rd, 25 points to the 4th. Finally, we weighed each category based on the importance of it for the real consumer:
- Best availability of the video conferencing service: x4
- Best establishment time for the organizers: x1
- Best establishment time for the guests: x2
- Best video quality for the guests: x2
This gave us a score on 100, and an overall ranking.