Troubleshooting the Data Flow

Health Checks and Confirmation

First let’s confirm the problem is what it seems. Run our health check pages:

• Open our app.
• Click Admin, then Run health checks.
• Let those finish.

If you get warnings about there being no recent data, continue with checking the SFTP server.

The SFTP server

Look in the folder where the files should be written to on the SFTP server.

1. If there are recent files in that folder (files newer than when the problem started) 
   • The data flow is working up to the SFTP server, so UCM and SFTP are working fine and the problem is somewhere *after* that.
   • Continue by checking the forwarder.
2. If there are only very old files in that folder (files from way before the problem started) 
   • First, double check that you are in fact looking in the right place. 
   • If you are in the correct folder, these very old files are probably just leftovers from ages ago — they are not generally a problem and can be addressed later. 
     
     • Feel free to contact us about them while you work through these other troubleshooting steps.
       
   • In any case, continue troubleshooting below as if there are ​“No files”.
     
3. If there are no files, stop your UF/​server for a few minutes so that it won’t read and delete new files while we test, then look again or refresh. 
   • If files begin to show up (it might take one or two minutes), UCM and the SFTP server are working fine and the problem is somewhere *after* that.
     
     • Go ahead and restart the UF/​server. Those new files should disappear again within a few moments.
     • Continue by checking the forwarder, especially with regards to the index and sourcetype settings of our TA. 
   • If no files appear even after stopping the UF/​server, then the SFTP server is not working or UCM isn’t trying to send files.
     
     • If it’s possible there just weren’t any completed calls in the last few minutes, try placing a test call and waiting another minute or two to force a file to be created.
     • If that doesn’t work to make a file appear, Double check that the SFTP server software is in fact still running. 
     • Confirm that the username and password for SFTP is correct.
     • Confirm the local operating system’s firewall is not blocking this traffic.
       
     • Check your SFTP server logs for additional information.
     • If you have found no apparent issues here continue with checking UCM and the Billing Server entry

The Billing Server inside CUCM

• Navigate to the Billing Server entry in Cisco Unified Serviceability
  • (Our setup docs might remind you where that is if you don’t remember)
    
• Confirm our entry still exists, isn’t disabled, and confirm where it points to.

• Restart the ​“Cisco CDR Repository Manager” service inside CUCM.
• Check to see if the data starts flowing into Splunk and our app again.
  • It can take some time for the system to catch up, especially if it was down for more than just a short time. 
    
  • You could revisit the previous step on checking the SFTP server, temporarily turn off the UF and watch to see if files now start showing up.
    

If data is still not making it to the SFTP server, it’s time to check the network.

The forwarder

There are two main choices here, one is if your SFTP server lives on a different server than your Splunk server, and the other if your SFTP server is the *same* server as your Splunk server.

1. If your SFTP server and forwarder are on a separate server from your Splunk server, check: 
   • That the Splunk forwarder is still running. You may have just stopped/​started it from previous instructions, so this might already be ​“known”. If so, just continue below.
   • That the TA_​cisco_​cdr add-on still exists. 
   • That the inputs.conf file exists — most likely at “$SPLUNK_​HOME/​etc/​apps/​TA_​cisco_​cdr/​local/inputs.conf” or failing that in “$SPLUNK_​HOME/​etc/​apps/​TA_​cisco_​cdr/​local/inputs.conf”
     (If there is no input.conf anywhere then the admin may have accidentally deleted the entire TA when they updated it). 
     
   • If the file does exist, then check that its contents are correct for your environment. 
     • It has the correct path for the files
     • It has the correct index=... and sourcetype=... settings
     • And that the index=... setting is reflected in our custom_​index macro
   • That there are no broken networks or misconfigured/​reconfigured firewalls between the UF and the Indexer.
2. If your SFTP server saves files on the Splunk server itself, check: 
   • The input file specification is correct for your environment.
     
     • It has the correct path for the files
       
     • It has the correct index=... and sourcetype=... settings
     • And that the index=... setting is reflected in our custom_​index macro

There are actually more possibilities for breakage here, but the above are the most common. If you are unsure about any of them, feel free to shoot us an email at support@​sideviewapps.​com!

The network

If you are have been directed to this section, we’ve mostly ruled out other causes and the problem is likely somewhere in the network.

• Double-check both the firewall settings and the firewall logs.* 
  
• Find out if any changes have been made to the network and confirm those did not interrupt the data flow.
• And additional networking checks, like tcpdump/​wireshark, which we leave as an exercise for the reader.
  

*You are putting those logs into Splunk aren’t you? It is a fantastic and common use case!

Our app’s custom_​index macro

If everything up to this point is working but our app can no longer find the data, there may be a mismatch between our app’s custom_index macro and which index the data is in.

• Check the forwarder to confirm which index is set in its index=... setting
  
• Confirm the macro custom_index points to the right index:
  • Go to Splunk’s Settings page.
  • Go to Advanced Search, then Search Macros.
  • Search for ​“custom_​index”
  • Edit it to reflect the index your data is in.
    

If you have any comments at all about the documentation, please send them to docs@​sideviewapps.​com.