Troubleshoot media quality issues

The troubleshooting view in Webex Calling is a helpful feature that allows administrators to troubleshoot media quality issues in a Webex call. You can search for information related to the call, view its media statistics, identify where the issue occurred, and resolve the problem.

To determine the probable cause, it is recommended to use the troubleshooting view along with the aggregate information derived from the Webex Calling Analytics dashboard. Use the Webex Calling Analytics dashboard to view issues impacting multiple users with a common cause that is related to location, network, and more; while the Webex Calling troubleshooting view can identify issues with individual calls.

The media quality troubleshooting feature does not require any specific configuration and is available by default in Control Hub to full customer, read-only, support, partner, and partner read-only administrators.

Administrators can search using the following criteria to get a list of calls where a media session was utilized with at least one Webex Calling registered endpoint:

Email IDs
Phone numbers (exact string match)
MAC address
Call IDs

The media quality troubleshooting allows administrators to:

View the end-to-end experience of the participants of the call.
View a hop detail of the call.
View if the media traverses through the Webex Calling cloud, or directly between the users (using Interactive Connectivity Establishment (ICE).
View calls for the past 21 days.
Analyze the call quality metrics that impacted the experience of the user.
Detect if the issue is with the caller or the callee.

Media Quality Troubleshooting applies only for media sessions, and it does not show call signaling sessions. The calls using Webex Calling appear after the call ends.

The troubleshooting view helps to identify the problem area by providing all the relevant metrics and cannot necessarily provide you the root cause for a poor call. The following pointers identify various factors and determine the resolution options:

The end-to-end experience of the user.
The Hop Details view.
Send or receive metrics from the user or the media relay point.
Whether the call occurred to or from an external network or between the Webex Calling registered endpoints.

Supported call flows

Media quality reports are collected from the caller and callee endpoints and the media relay points. This allows segmentation of the media experience to narrow down and identify whether the issue occurred at the:

Caller or callee
Media path to or from the Webex Calling cloud

Call legs appear if there is a media session that is established with at least one Webex Calling registered endpoint on the call. For example, for an outbound call from a hunt group to eight agents, if only one agent answers, then there is no media experience to troubleshoot for the other seven agents.

There are five types of media experiences or paths for Webex Calling troubleshooting:

On-net Optimized – Calls within the organization where ICE is successful and the media flows directly between the users. See Webex Calling Media Optimization with Interactive Connectivity Establishment (ICE) for detailed information.
On-net Unoptimized – Calls within the organization where Interactive Connectivity Establishment (ICE) was not possible or established. In this scenario, the media flows through the Webex Calling cloud.
On-net Cloud Hosted – Calls within an organization where media is provided by a media server that is hosted in the cloud (for example, listening to voicemail, dialing into an auto-attendant).
Off-net Call to or from the Webex Calling registered endpoint
- via Verizon VoIP and/or Cloud Connected PSTN Provider - Inbound and outbound calls of an organization where the other party is on the PSTN network. The media is relayed through a cloud-connected PSTN provider (CCPP), over a high-quality interconnect.
- via Local Gateway - Inbound and outbound calls of an organization where the other party’s media is through the enterprise. Behind the local gateway, the media session can be from an enterprise-hosted user (for example, registered to call control in the enterprise) or from PSTN, where PSTN is provided by the enterprise.

If there are 1 or 2 Webex Calling registered users who are involved in a point-to-point on-net call, then the troubleshooting view presents metrics for one or both parties. If the call is off-net (User 1 receives a call from a PSTN user), then the troubleshooting view presents only User1's client metrics, along with the metrics taken from the media relay point.

Most of the call scenarios in the troubleshooting view show two call legs (caller and callee); however, some of the call scenarios (such as call park or retrieve) show only one call leg. In such cases, the other call leg shows up separately in the troubleshooting view. This does not hinder troubleshooting the call and detecting where the problem occurred. However, it does require the admin to manually correlate the two call legs using a common entity such as overlapping time. Future enhancements to the troubleshooting view will eliminate the need to use manual lookups.

Access calling troubleshooting

From Control Hub, go to Monitoring and select Troubleshooting.
Select Meetings & Calls.
Search for the email ID of the user or device, phone number (exact string match), MAC address of the user or device, or the Call IDs of the call leg you’d like to view.

The listing view displays calls made using at least one Webex Calling registered endpoint and having a media session. The Quality column appears only if your organization uses Webex Calling.

The Webex Calling calls are graded for quality. However, for the Webex meetings or Call on Webex sessions, this grading does not apply. The call experience is graded as:
- Poor – indicates that either the caller or callee had a poor end-to-end experience (for example, choppy audio).
- Good – indicates the end-to-end experience for the caller and callee did not exceed thresholds.
- None – applies to meetings or Call on Webex sessions.
Click a specific call on the listing view, to inspect the Hop details. From the Hop details, you can:
- Hover on the device to view the end-to-end call experience.
- Hover on the hop between the endpoint and Webex Calling cloud to view the hop details.

‍

The end-to-end call experience is based on the media quality data that is collected from each Webex Calling registered endpoint (Webex App, Webex Calling App, Webex Calling Device such as 8865) at the end of the call. The call is graded as good if it satisfies these thresholds-Packet losses less than 5%, latency or Round Trip Time (RTT) less than 400 ms, and jitter less than 150 ms.

The quality of the hop is derived from data that are collected from the media relay point in the Webex Calling cloud. For PSTN calls through CCPP or local gateway, the data collection is from the Webex Calling cloud and not from the PSTN endpoint. A hop is graded as good if it satisfies three thresholds:

Packet loss is less than 2.5%.
Latency or RTT less than 200 ms.
Jitter less than 75 ms.

The hop metrics vary during a session depending on the sampling time and variability in the network. The values that are reported by the media relay point and the clients (end-to-end experience) may not align. However, they should be in close alignment to allow segmentation along the path.

It is recommended to use the individual call troubleshooting view with the aggregate information that is derived from Analytics.

Analyze the call quality for the different call types using the troubleshooting view.

On-net Optimized Call Flow - Calls within the organization where ICE is successful and the media relay in the cloud is removed from the path. The media flow is directly between the user's devices.

The call is graded as good as both the caller and callee have a good end-to-end experience.
The administrator can observe that media flows directly between the two users and does not travel through the Webex Calling cloud.

‍

Optimized call flows can have a poor experience if the enterprise or local network is the source of the issue since the media between the two users will traverse the local network. Latency or RTT is always lower on an optimized call but packet loss and jitter can still be a factor depending on the network between the two users.

On-net Unoptimized Call Flow - Calls within the organization where an ICE call isn’t successful and the media flows through the Webex Calling cloud.

The administrator can observe the following:

The caller’s end-to-end experience is graded as poor.
There is a problem with the caller's network hop that affects both the send and the receive stream.
The callee’s network hop does not have an issue.
The callee’s end-to-end experience is graded as poor due to the issue from the caller.

On-net Cloud Hosted Call Flow - Calls within an organization where the media is provided by a media server that is hosted in the cloud.

The call is graded as good as the caller’s end-to-end experience is within the accepted thresholds. The administrator can observe the following:

The network hop for the caller is graded as poor as some of the metrics are above the acceptable threshold.
The send stream from the voicemail is graded as good per the metrics collected from the media relay point.
The media server used to collect or deposit the voicemail does not currently report metrics. However, these servers are hosted and managed as part of the Webex Calling cloud so the quality of that link segment is internal and always high quality, with low latency.
The administrator can observe the caller’s end-to-end experience is graded as good although the hop is graded as poor. This is due to the callee's hop having a good network that compensates for the performance degradation of the caller's network.

Off-net PSTN through CCPP - Calls in and out of an organization where the other party is on the PSTN network. The media is relayed from a cloud-connected PSTN provider (CCPP).

The call is graded as poor as the callee’s end-to-end experience is not within the accepted thresholds. The administrator can observe the following:

The problem is with the caller’s PSTN hop specifically with the sending stream.
The callee’s network hop does not have an issue.
The callee’s end-to-end experience is graded as poor due to the issue from the caller.
The end-to-end experience and the receive stream metrics of the caller who is on the PSTN network are not currently available as these metrics are not transmitted to the Webex Calling cloud.

Off-net PSTN through Local Gateway - Calls in and out of an organization where the caller's media is coming from an enterprise. The media session can be from an enterprise-hosted user or from PSTN, where the PSTN is provided through the enterprise.

The call is graded as poor as the caller’s end-to-end experience is not within the accepted thresholds. The administrator can observe the following:

There is a problem with the caller’s hop to the Webex Calling cloud both on the send and receive the stream.
The caller’s end-to-end experience is graded as poor either due to the issue observed in the hop or the issues at the user’s end (devices, network, etc).
The incoming traffic to the Webex Calling cloud from the callee’s end is graded as good.
The end-to-end experience and the receive stream metrics of the callee who is called from the local gateway are not currently available as these metrics are not transmitted to the Webex Calling cloud.

Troubleshoot

The hop-by-hop view helps you to locate where the problem occurred. Now that you’ve found where the problem is, and with the metrics (jitter, packet loss, latency) you can try the following to troubleshoot the issue.

Typical possibilities for media issues are:

Network/ISP/location-specific issues - Due to firewall, network configuration, or bandwidth there is a pattern of poor experiences in a particular location or network subnet. Use the per call troubleshooting view (identify the location associated with the poor session) with the analytics view to review the aggregate patterns for the location.
User-specific issues - A user or device is connected on a poor network (for example, Wi-Fi or working from home) which means their experience is impacted by the associated network capabilities. See the Use CScan to Test Webex Calling Network Quality article to identify issues with the network.
Call type-specific issues - A user’s poor experience is because of the quality on the far end. This is typical in PSTN scenarios where the user is talking to another user on a mobile network and the session has high packet loss on the PSTN network.

Limitations

Media quality metrics are not available from the following devices:

Analog Phones
Third-party devices
IPv6 endpoints