New Cisco Quality Scores

Just when you were getting a handle on knowing what MOS (mean opinion score) or it’s cousin MLQK (MOS Listening Quality K‑factor – the last 8 seconds MOS) means and how they’re useful, Cisco is phasing them out.

But first and important (and short) note:

Please update to version 5.10 to get all the goodies mentioned below. Remember, since we’re now hosted directly on Splunkbase, updating is generally as easy a just clicking the ​“update” link!

Why are they being dropped?

Well, there are many reasons, but the two primary ones are that

• MOS is codec dependent so is hard or impossible to compare across codecs
• it also doesn’t scale properly between calls of different durations so can’t be used to reliably compare those either.

What good is a metric that you can’t use in comparisons?

What are the replacements?

All Cisco IP phones now support a few new metrics around ​“call concealment time”.

IP based conversations (via IP phones of a variety of sorts) are subject to lost packets, out of order data, and indeed all sorts of generally minor and unavoidable problems. When this happens the devices ​“fill in” the missing voice information with made up audio. These losses that are ​“filled in” are called ​“concealed” time, and the new statistics measure those losses and its effect on voice quality.

“Made-up audio” sounds like it should always be a problem, but the fact is that small amounts of missing data can be filled in very effectively and without any real degradation of the conversation. It’s the longer periods that are the problem.

So without further ado, here’s the new metrics that Cisco reports:

What Cisco Reports:

Concealed Seconds (CS)

A measurement of how many seconds showed some concealment during a call. ​“Some” in this case being at a level where it probably wasn’t hard to understand the speaker. These are useful to know are happening, but aren’t generally a problem.

Severely Concealed Seconds (SCS)

Similar to Concealed Seconds only measuring seconds where the speech was probably very hard to understand. Cisco defines this as seconds where the concealment that second was greater than 50 milliseconds or approximately 5 percent. SCS are seconds that are probably problems.

What we create from those:

SCS_​total and CS_total

These two fields add up the individual call leg’s CS and SCS numbers. Alone they’re not that useful, but the next section makes them so:

CSR_​overall and SCSR_overall

The holy grail – these are the two fields that give you a ratio of how many seconds were moderately or severely ​“concealed”. If you take the number of Concealed Seconds (CS_​total) or Severely Concealed Seconds (SCS_​total) and divide it by the duration of the call, you get a ratio or percentage.

This gives us a way to compare any codec, any call duration, any sort of devices – the scale is always the same, it’s simple to calculate, and it is a good proxy for ​“I couldn’t hear what you said there”.

What is generally considered a bad call?

As with most answers, it depends.

First, know that Cisco has decided what they feel is ​“Bad”. You don’t have to use these, but they may be a good place to start.

Second, it really does sort of depend on the length of the call. If you have a really short call – say leaving a short voicemail with a name and telephone number – it doesn’t take a lot of SCS to make a high SCSR, and that makes sense because it only takes a second or two and suddenly the call was useless. ​“This is ~~~ch at ~158~~7420, call me back!”. Maybe you’ll guess from context or from the part of the number you heard, but who knows?

If you have a longer call, well, a small amount of concealment can be dealt with easily enough. ​“Sorry, you were garbled there for a second, what was your number again?” Cisco thinks this is very important (see below), but we’re not so sure it’s this cut and dried.

So the guidelines Cisco uses are:

• For call durations under 20 seconds
  • Good is under 3% SCSR
  • Acceptable is between 3% and 7% SCSR
  • Poor is anything over 7%
• For call durations over 20 seconds
  • Good is under 20% SCSR
  • Acceptable is between 20% and 30% SCSR
  • Poor is anything over 30%.

We don’t convert to ​“percent” from ​“numbers”, so instead of 3% you’ll see ​“0.03”. We don’t have enough data to make any sort of firmer guidelines, but our guess is that length of call matters less than Cisco thinks it does – a call where 20% of it was unintelligible seems like one we would have hung up on at some point, although we’re not totally unconvinced that some difference is useful. Still, we’d love to hear what you find so let us know!

How do we see our SCS/SCSR

Easy!

Grab your field picker and search for any of the above fields. Add them in by clicking the green “->” arrow. I hope you didn’t think this was going to be hard.

I’ve found it was easiest to filter by keyword down to ​“cs”, which shows the 6 fields we are talking about. Notice I’ve already added 3 to my display, but not the second three. Click their green arrow to add those too. Also, don’t forget you can drag the Selected Fields around to reorder them!

Once you have it there, it’ll follow you from Browse Calls to General report and back again, you can sort on the columns, and also it’ll also be searchable like anything else using the ​“search filters” field in Browse Calls.

Can you help me filter on this?

Absolutely!

If you follow the above instructions and add the various CS and SCS fields to your Browse Calls, you can filter it by adding some search filters.

Filtering to just calls with some Concealed or Severely Concealed seconds

A good place to start is to add CS>0 to the search filters. This will filter it to calls that had one or more legs with a Concealed Seconds greater than zero. This is functionally equivalent to a search filter of CS_total>0.

Remember – by default the Browse Calls page only picks up the most recent 1000 call legs – it’s for ​“skimming the top” of the data, not necessarily for deep dives into history. Of course, you can change the get only … dropdown to something larger, or click » graph calls over time in the top right. More on that in a bit.

A related but more targeted search might be to add SCS_total>0 to filter it to just those calls where at least one second was Severely concealed. Fun note, for some reason I always default to typing ​“Seriously concealed”.

But you already talked about how raw numbers are nearly useless? Ratios are what we need!

Right, I did say that, didn’t I? Well, they’re not entirely useless. While they don’t make good comparisons, they DO make a reasonable litmus test for ​“Calls that had at least some problem.”

So, ratios! They’re just as easy. Let’s say we want calls where more than 5% of the call was garbled. That’s SCSR_overall>=0.05.

What if we wanted to use Cisco’s criteria?

If you’d like to use the criteria Cisco came up with, and let’s say you were trying to find all the ones with ​“Poor” call quality. That’s more than 7% for calls 20 seconds or under, and 30% for calls over 20 seconds.

(SCSR_overall>=0.07 AND duration_elapsed<=20) OR (SCSR_overall>=0.3 AND duration_elapsed>20)

There we just put a pair of ANDed together criteria – the SCSR_​overall and duration_​elapsed – in an OR with another set of them. Not too hard.

Management, of course, needs to see charts?

Going back to the original ​“find all calls that had some voice quality issues” with our criteria CS>0. If you plug that back into your search filter and run it, then you can click » graph calls over time in the top right and prepare to be amazed…

By default, that shows the distinct count of calls over time, which maybe you’ll find useful but maybe might be better as the 95^th percentile of CSR_​overall, displaying over time, and possibly with a split by of gateway.

Or, perhaps you care more about SCS than CS, and since you aren’t management you also care more about the numbers and less about the exact time of day? A few changes to options, then selecting the table, gives a list in descending order of SCSR_​overall so that you can pinpoint the troublesome gateways.

(Note that’s 95^th percentile of SCSR_​overall, over gateway with a split by of none, sorted by perc95(SCSR_overall) (descending), then switch to the Table view.

Conclusion

I hope this helps you get started in analyzing your own call quality using Cisco’s newer metrics. If you have any questions, or have a screenshot or two you’d like to share, please let us know at docs@​sideviewapps.​com!