Ingest Other CM data via syslog

Set up Syslog

Our first big task is to have syslog installed and available. This is very likely only going to be a concern if you run a single, standalone Splunk+Cisco CDR instance. If you have a bigger Splunk install than just our Cisco CDR Reporting and Analytics app, then you are likely ingesting syslog somewhere already. If you have a Splunk ​“Team,” then I’m *sure* you have syslog somewhere. Ask them about it after bringing them brownies or cake.

If you do not already have a syslog server, you will have to build one. This is a task far outside our bailiwick, but we can at least give you a few nudges in the right direction.

• If you already run a Linux server for Splunk (or where your SFTP server is), then …
  • Install syslog-ng on it (the built-in syslog is difficult and hard to configure).
  • Configure it to handle receiving remote syslog (see below).
• If you already run a Windows server for Splunk (where you’ve installed an SFTP server), then
  • Install the free version of Kiwi Syslog Server on that server.
  • Move the log file location to somewhere sane (see below).
• If you have neither (e.g. you have a bigger implementation but syslog is on a machine you aren’t allowed to play with):
  • Did I mention bribes of brownies to the syslog team? If that still doesn’t work…
  • Build a tiny VM (1−2 cores, 2 – 4 GB of RAM, 10 – 25 GB of disk) running Ubuntu or some other Linux distribution.
  • Install syslog-ng on it (the built-in syslog is difficult and hard to configure).
  • Configure it to handle receiving remote syslog (see below).

There are lots of places to go for help with this. Use the Search, Luke!

Configure Syslog

Depending on your needs, you’ll want either only one of the below two sections.

Linux with Syslog-NG

Assuming you did syslog-ng on a Linux VM, you will need to configure the syslog-ng instance to listen on the network and to create new directories for each new host it sees. To do this, you’ll want to edit syslog-ng’s configuration (locations vary, use your OS’es documentation on where it hides syslog configuration files), and add something like the below near the top of the file.

 source s_network_udp { udp( port(514)); };
 source s_network_tcp { syslog( port(514) transport("tcp")); };
 destination d_syslogs { file ("/var/log/remote/${HOST}/log.txt"); };
 log {source(s_network_udp); source(s_network_tcp); destination(d_syslogs); };

That would make directories for each host (either name or IP, depending on if it resolves) like ‘/var/log/remote/192.168.0.1/log.txt’. Restart syslog-ng after making those changes.

Note that this is a really simple, basic config. While it should work, we make no guarantees it’s appropriate for your environment. If you have problems with it, there are lots of forums and help documents available, but let us know what you find out!

Windows with Kiwi Syslog

After installing Kiwi Syslog Server, the only thing you’ll likely need to do is move the log file location.

1. Open the Kiwi Syslog Server Console.
2. Click File, then Setup.
3. Under Rules, Default, Actions, Log to File.
4. Change the path and filename of the file to something NOT in program files (perhaps C:\Logs\ or D:\Logs\), otherwise, permissions for Splunk to read the files might be a problem.

Note that these are the minimum changes to get a working Kiwi Syslog Server installation working well enough for our needs. There may be other configurations you need to do for your own environment.

Configure CUCM to output Syslog

Here’s where there are choices to make on what you want to collect. This example is going to collect all the CallManager service information, which includes device registration/​deregistration, resource allocation failures, and lots more. There are other options scattered all over the CUCM installation – finding them and interpreting them is an exercise left to the reader (although, down below there’s at least one more location mentioned).

1. Open Cisco Unified Serviceability.
2. Click Alarm, Configuration.
3. Select your CUCM Voice/​Video server and click Go.
4. Select the Service Group CM Services and click Go.
5. Select Service Cisco CallManager (Active) and click Go.
6. You may or may not need to click the ​“Apply to all nodes”, depending on your Voice infrastructure.
7. Under the section, Remote Syslogs:
   • Click in the checkbox to Enable Alarm.
   • Enter the IP Address of your Syslog system in the first available Server Name # spot.
   • Change the Alarm Event Level to Informational.
   • Do NOT click the checkbox Exclude End Point Alarms.
     
8. Click Save.

You might want to also check the other services available – I’d suggest enabling only one at a time and carefully checking the license amounts they seem to take. If you have suggestions, please send them to us!

Check Splunk license amounts

Now that we have this dumping data to syslog, let it run for an hour or two, then check the filesize of the created log.txt file and do a little math.

What you need to check is only that this amount of data isn’t likely to take you over the license. The file size isn’t exactly going to be the size ingested in Splunk, but it’s usually close enough to give you an idea if this is a good idea or not.

Also note that Splunk Enterprise lets you go over your allocated data amount up to 5 times in a rolling 30-day window (up to 3 times if you’re on a Trial Splunk license), so even if you ​“blow up your license” for a day, it’s not the end of the world. Perhaps ingesting one afternoon’s worth of data into Splunk would be useful just to see what answers you can really get out of it?

A warning though that if you have a ​“no enforcement” license, then ​“going over” isn’t really a concept; but if you consistently do it, at some point, Splunk will make you pay for that extra license.

Have Splunk ingest the data

If it’s not there already, install the UF on the syslog machine, and in the Splunk configuration tell it to read the files in a ​‘local’ input.conf file:

However if you are using kiwi syslog, then do not set sourcetype to ​“syslog”. Instead set sourcetype to ​“cucm-kiwi-syslog”

[monitor:///var/log/remote/192.168.0.*/log.txt]
 host_segment = 4
 # NOTE: if you are using kiwi syslog, then uncomment the following line and comment out the sourcetype=syslog line.
 #sourcetype = cucm_kiwi_syslog
 sourcetype = syslog
 index = cucm_syslog0

Obviously, that’s just a sample – you need to fix it up to your path!

After that, as long as your UF’s outputs are set right, you’ll have data coming in.

What kinds of data do we get?

If you followed along and used the same Alarm settings as we outline above, you’ll get a variety of messages. Here are a few samples to gaze upon – take in their beauty!

Device Unregistered

%UC_CALLMANAGER-6-DeviceUnregistered:%[Device name=CiscoUM1-VI3][Device IP address=10.10.20.18][Protocol=SCCP][Device type=80][Device description=VM][Reason Code=9][IPAddressAttributes=3][App ID=Cisco CallManager][Cluster ID=LocalCluster][Node ID=hq-cucm-pub.abc.inc]: Device unregistered

Device Registered

%UC_CALLMANAGER-6-DeviceRegistered: %[Device name=CiscoUM1-VI3][Device IP address=10.10.20.18][Protocol=SCCP][Device type=80][Performance monitor object=2][Device description=VM][Associated directory numbers=7003][IPAddressAttributes=3][App ID=Cisco CallManager][Cluster ID=LocalCluster][Node ID=hq-cucm-pub.abc.inc]: Device registered

Something broke when a device tried to register

%UC_CALLMANAGER-4-DeviceTransientConnection: %[Connecting Port=50538][Device name=][Device IP Address=10.10.20.18][Device type=255][Reason Code=6][Protocol=SCCP][IPAddressAttributes=0][App ID=Cisco CallManager][Cluster ID=LocalCluster][Node ID=hq-cucm-pub.abc.inc]: A device attempted to register but did not complete registration

How would I make a start at using those?

We have some ​“behind the scenes” extractions for those that you could apply (Contact us – we’ll probably make them more public soon, but for now we’d be happy to provide them to you!), but for just fiddling around and trying to find out what sort of useful information you can get, you could start with a search that does a bit of delimiter based extraction…

index="cucm_syslog"
| extract pairdelim="[]" kvdelim="=" auto=f

If you run that search, a lot of useful fields should pop up on the left. Cluster_​ID, Node_​ID, Device_​IP_​Address, etc… From there, dig around like any other Splunk data!

What else is available?

Cisco has a variety of places that syslog output can be enabled. We talk about a few of the locations and services that they provide syslog ability for, but there’s also Cisco Unified CM Administration, System, Enterprise Parameters, and the section Cisco Syslog Agent, which gets system information too – both OS and hardware information and alarms.

Frankly, we’re not even SURE what all is available or how you might use it. The sky really is the limit, we think, but we aren’t *nearly* as hip to what’s in CUCM as you probably are, so tell us your ideas!

And, that concludes our wonderful foray into the wide world of Syslogging from CallManager.

If you have any comments at all about the documentation, please send them to docs@​sideviewapps.​com.