# Sinkhole vs monitor inputs

Our installation and setup docs specify that you create a “sinkhole” input to index the CDR and CMR data, and a side effect of this is that the raw files on disk will be deleted as the records are ingested (aka indexed) into Splunk.  Sometimes this causes a hiccup though, when customers or trial prospects don’t want the raw data to be deleted from disk.

### Why might you want to keep the files around instead of deleting them?

Some possible reasons to not sinkhole the files may include:

1. You might want to keep the raw data around as an emergency backup or crosscheck
2. There is a preexisting External Billing Server config in CUCM, and deleting the files would break some other integration

When it’s reason #1, it may be enough to know that historical CDR and CMR can be exported out of CUCM quite far back, and they come out of UCM in such a way that the historical files are fairly easy to index. (see “Loading Historical Data”)

When it’s reason #2, one easy workaround is to not reuse that other external billing server config. Instead create a second separate External Billing Server entry. This is after all what they are for and why UCM allows you to have several of them. It also offers a great advantage that if an administrator of that other integration makes a change and forgets you were also using the same files, your data won’t stop coming in.

### What are the risks of keeping the files?

Splunk’s monitor inputs are great, but they have a feature where they constantly check all the matching files for appended changes. In the case of CDR and CMR, this serves no purpose – once UCM FTP’s a file over, it will never FTP the same file ever again nor will it ever append data into an existing file. However there’s no way to configure a Splunk monitor input to not always be checking those files.

With only 50 or even 500 files, Splunk and the operating system can work together to check all the files without using many resources. However when there are 50,000 files or more this task starts to become extremely onerous. It’s a long story about OS file descriptors and I/O contention, but the long story short is that your Splunk server will start to run extremely slowly. Searches will take seemingly forever to run, indexing will fall behind by hours and then days or even weeks. Eventually when you have about a million files accumulated you might have difficulty even accessing the server at all. What is happening is that all the server’s considerable resources are being thrown at constantly and pointlessly opening and seeking through tens of thousands of files.  With one CDR file per CallManager node per minute by default, the number of files in the destination directories can rise to 50,000 surprisingly quickly.

### Now that I understand the risks, how can I set this up?

If you want to go ahead anyway, you can technically create a monitor input instead of a sinkhole input.  There are some critical extra steps that need to be taken when you do this.  Here’s how, along with some recommendations on dealing with the resulting files.

Please follow the appropriate section below for a new install or for conversion of an existing installation either direction.

1) Create the monitor input on the Universal Forwarder by adding this config to an inputs.conf file located at “$SPLUNK_HOME/etc/apps/TA_cisco_cdr/local/inputs.conf”. You may have to create the directory “local” and the file “inputs.conf” the first time. If your Universal Forwarder is on Windows, the inputs will look like these: [monitor://D:\path\to\files\cdr_*] index = cisco_cdr sourcetype = cucm_cdr [monitor://D:\path\to\files\cmr_*] index = cisco_cdr sourcetype = cucm_cmr  If your Universal Forwarder is on Linux or Unix, the inputs will look like these: [monitor:///path/to/files/cdr_*] index = cisco_cdr sourcetype = cucm_cdr [monitor:///path/to/files/cmr_*] index = cisco_cdr sourcetype = cucm_cmr  NOTE: It is critical that no mistakes be made here. Only the sections in bold should be edited. Leave everything else exactly as it is written above. Make sure : 1. if your forwarder is on a linux or unix host, that you use appropriate slashes, ie “/foo/bar/cdr_*” vs “C:\foo\bar\cdr_*”. 2. that the index specified in both lines matches exactly the single index specified in the “custom_index” macro on the Search Heads’ apps. Index names in Splunk are case-sensitive. 3. that “cdr_*” and “cmr_*” are present respectively on the end of each path, and that they correspond to the “_cdr” and “_cmr” suffixes on the given sourcetype being set. 4. that both stanzas specify the exact same index. “cisco_cdr” is assumed here although you may have chosen a different index name. 2) Set up a script to run every day to delete or move away any and all files whose modified dates are more than 3 days from the current time. The key goal is to make sure the files thus moved no longer match the path of the data inputs. On windows an example that deletes all files older than 3 days is: forfiles /p "E:\CDR" /D -3 /C "cmd /c del @path"  On Linux, an example is: find /path/to/cdr/files* -mtime +3 -exec rm {} \;  NOTE: Change the path in either case to match your situation. You may also need a different time period before deleting – if you want to shorten it go right ahead, but please be very careful if you want to lengthen it. A lot of other examples can be found on the web. # Convert a sinkhole input into a monitor input If you find you need to keep the raw files around even though you already have a sinkhole input set up for them, the conversion is easy. 1) Open the inputs.conf file containing the data input. It will typically be at$SPLUNK_HOME/etc/system/local/inputs.conf or perhaps $SPLUNK_HOME/etc/apps/cisco_cdr/local/inputs.conf on a standalone indexer. 2) Find the two stanzas for the batch input – (one will be to match CDR and one for CMR). They will look like this: [batch://C:\path\to\files\cdr_*] index = cisco_cdr sourcetype = cucm_cdr move_policy = sinkhole  3) Carefully change the word “batch” in those 2 stanzas to “monitor“. 4) Delete the entire line with “move_policy = sinkhole“. The result (in this example!) would look like this (AND REMEMBER there’s two stanzas that need to be changed – CDR and CMR both!): [monitor://C:\path\to\files\cdr_*] index = cisco_cdr sourcetype = cucm_cdr  5) Save the file. 6) Restart splunk on this host. It may be a universal forwarder, an HF, or a standalone indexer but in any event it must be restarted to pick up the data input change. 7) Set up a script to run every day to delete or move away any and all files whose modified dates are more than 3 days from the current time. The key goal is to make sure the files thus moved no longer match the path of the data inputs. On windows an example that deletes all files older than 3 days is: forfiles /p "E:\CDR" /D -3 /C "cmd /c del @path"  On Linux, an example is: find /path/to/cdr/files* -mtime +3 -exec rm {} \;  NOTE: Change the path in either case to match your situation. You may also need a different time period before deleting – if you want to shorten it go right ahead, but please be very careful if you want to lengthen it. A lot of other examples can be found on the web. # Safely convert a monitor input to a sinkhole input There are a variety of reasons you may want to move a monitor input to a sinkhole input. The monitor input could be legacy from long ago. It could be that you need to remove the extra point of failure of having a cron job or scheduled task. Perhaps you just realize that keeping the files around is more hassle than it’s worth since you can export any potentially missing data from CUCM and import it again. In any of those cases, the crucial issue in doing this is that if you just switch the input config and restart, all of the monitored files remaining on disk will get indexed a second time. Monitor inputs use a splunk mechanism called the “fishbucket” to make sure they never index any events in any file twice — unfortunately sinkhole inputs do NOT pay any attention to the monitor input’s fishbucket so the new batch input will re-read the already indexed files and you will have duplicated data. The trick I recommend involves temporarily interrupting the flow of files coming in. 1) SSH or login to the host to which the files are being SFTP’ed, and thus the host where the Splunk data input is running. 2) Verify that you have fewer than, say, 50,000 files or so. If you have more than that, extra care will need to be taken to be sure that the CDR data you have actually is “all caught up”. Email us if this might be an issue. 3) Shutdown the SFTP server. This will cause UCM to start queueing up CDR and CMR files – they will not be lost, only delayed until you turn the SFTP server back on. If this is a built-into-linux SSH account, … I’m not sure what to do here, maybe you can just temporarily disable the user or change its password for the time being? 4) Wait a couple minutes. This will allow the Splunk data input to catch up and index every last event in the files that have been SFTP’ed. Feel free to check the CDR app’s “Browse Calls” for recent data. When the date/time of the recent data there seems to match pretty well with the newest file, you should be OK. (They won’t match perfectly, but it should be pretty easy to spot that it’s mostly up to date or not). 5) DELETE OR MOVE THE FILES OUT OF THE DIRECTORY! Don’t forget this step or the new data input will index them all a second time. 6) Open the inputs.conf file containing the data input. It will typically be at$SPLUNK_HOME/etc/system/local/inputs.conf or perhaps \$SPLUNK_HOME/etc/apps/cisco_cdr/local/inputs.conf on a standalone indexer.

7) Find the two stanzas for the monitor input – (one will be to match CDR and one for CMR).  They will look like this:

[monitor://C:\path\to\files\cdr_*]
index = cisco_cdr
sourcetype = cucm_cdr


8) Carefully change the word “monitor” in those 2 stanzas, to “batch“.

9) Carefully add a new line into the stanza –

move_policy = sinkhole


10) Double check it looks like this now:

[batch://C:\path\to\files\cdr_*]
index = cisco_cdr
sourcetype = cucm_cdr
move_policy = sinkhole


11) Save the file.

11) Restart Splunk on this host. It may be a universal forwarder, or a HF, or a standalone indexer but in any event it must be restarted to pick up the data input change.

12) Start up your FTP server again. CUCM should send over the queued CDR and CMR files, and they will get indexed now by the sinkhole input.  (If you changed the password for the user temporarily, I think as long as there weren’t TOO many failures it’ll all start up again.  If it does not, then you probably need to do step 1 in our data troubleshooting document.  (Ignore step 0, and also ignore the admonition to start at step 3, just start at step 1 because that’s what we need to do here).

Note that sinkhole inputs are pretty unintuitive – don’t watch the directory waiting for the files to appear because they never will ! the very moment that the FTP server puts them there, Splunk will whisk them away to be indexed and delete them from the filesystem.

If you have any comments at all about the documentation, please send it in to docs@sideviewapps.com.