Cisco CDR Reporting & Analytics | Administration

Migrating to a New Splunk Deployment

We get asked fairly regularly about moving our app from one server to another. The answer to this question is very dependent on your environment.

In the below we talk about a migration strategy for a single standalone Splunk server with our apps on it, either with the SFTP server software on this same server and part of the migration, or with it separate.

Important notices:

  • While it’s worked multiple times, we obviously cannot guarantee this process will work. You should have tested backups and a restore plan.
  • There are a few spots in the below where you have to confirm operation of systems before moving forward, please don’t skip those steps.
  • Splunk is typically installed at /​opt/​splunk on Linux, or at​“C:\Program Files\Splunk” on Windows. We’ll follow Splunk’s convention and refer to this directory as $SPLUNK_HOME.

Also, if you are completely new to Splunk and your current environment, here’s a great set of docs pages from Splunk on what to do when you have inherited a deployment you know nothing about. For our purpose, the most useful page of this is the one on the deployment topology.

New Server Prep 

This group of steps can be done ahead of time.

  1. Get Splunk, the Cisco CDR app, and Canary all installed on the new server by following our docs for Install Step 1, Splunk and Install Step 2, Sideview Apps.
  2. Enter your existing Cisco CDR license key using the process in our docs for Updating the license.
  3. Confirm any accounts are created in Splunk, email settings are right and so forth. 
  4. Do NOT at this time set up a data input, we’ll use the fact there isn’t one set up later as an easy way to confirm SFTP is working. 
  5. If your SFTP server lives on this host too, then set up the new SFTP server software. 
  6. Once it’s ready, shut off Splunk on it via services or via the command line. 
  7. If the SFTP server software is installed on this host as well, leave the SFTP server running so that if we point CUCM at it the files will start accumulating in the SFTP server’s drop folder. 

Old Server Prep 

This group of steps can be done ahead of time.

  1. In old Splunk, click Splunk’s Settings then indexes. Find out the Home path to where the index cisco_​cdr is. Note this value. 
  2. Make sure all Searches, Reports and Alerts that you want to migrate (e.g. ​“all valid and current ones”) are set to be shared in the app and not just private. This will make them easier to move later. 
    • Use Splunk’s documentation for setting report permissions to make sure all searches that you need are shared at least with ​“App”.

Migration of historical data 

These steps performed at migration time

  1. On the old server, turn off the data inputs for our app.
    • It should be in Splunk’s Settings, Data inputs, Files & Directories.
    • For both the CDR and the CMR input, click Disable
  2. Stop Splunk on the old server.
  3. Open up the cisco_​cdr index location in the file system (found previously in ​“Old Server Prep”)
  4. Copy the folders inside that folder to the same place on the new server.
    • Please confirm neither the old nor the new server have Splunk running at this time.
    • The folders are named like ​“colddb”, ​“datamodel_​summary”, ​“db”, and ​“thaweddb”. It’s likely the only real data will be in the db folder, but copy all of them over just in case.
  5. If it asks to overwrite files, double-check that you are copying files the right direction then go ahead and overwrite them.

Intermediate Confirmation

You can now start Splunk on the new server and confirm that Investigate Calls has your historical data. You won’t have *new* data, but it should be current up to the start of this process.

You can leave Splunk running on the new server at this time. 

Migrate Data Inputs

If your SFTP server software runs on the same host as Splunk

If your SFTP server software runs on the same host as Splunk, then -

  1. Migrate the CUCM billing server to point to the new system. 
    • Generally you have to create a whole new ​“External Billing Server” as per our Installation Step 4, UCM instructions.
  2. You should see files start building up in wherever the SFTP server saves files (assuming you have completed phone calls during the past minute!). 
    • If you do not see any files in a while (you can place a phone call and hang up to trigger it to send in files the next minute), then see step 1 in our Troubleshooting the Data Flow documentation to restart a service in CM
  3. Now that you know CM is sending SFTP files to the new location you can start ingesting those new files, so set up the data input again. 
    • Click our app’s Admin menu then Set up Data Inputs menu item and follow the wizard.
  4. All the files in the SFTP folder should disappear within a minute or so (maybe practically immediately!)
  5. Important ​“fill the gap” step — 
    Go to the old SFTP server’s file location and copy whatever remaining cdr_​* and cmr_​* files from it to the new SFTP server’s folder.

If your SFTP server software is on another system entirely

If your SFTP server software is on another system entirely, then — 

  1. On the SFTP server, modify the outputs.conf to point to the new server and restart the Universal Forwarder service. 

Confirmation of data

You should now see Investigate Calls not only having old data, but also having new data. (It might take a few minutes to ​“catch up” from the backlog.)

Assuming that all looks good, now we can migrate other things.

Migrate ​“system” stuff (groups, sites, etc…)

  1. From your old server in the $SPLUNK_HOME\etc\apps\cisco_cdr\lookups folder there are a few files to copy into their new location:
    • cidr.csv
    • groups.csv
    • clusters.csv
    • devices.csv
    • call_quality_thresholds.csv
  2. Just put them in the same location on the new server, overwriting whatever’s there.

Migrate custom reports, dashboards and alerts

There are two layers to this: app-level config and user-level config. 

App-level config

The most relevant config will be at the app level and located in two top-level folders inside $SPLUNK_​HOME/​cisco_​cdr – ​“default” and ​“local”. (There are several layers to Splunk’s configuration file precedence, we’re just brushing the surface here). 

The default doesn’t need migration and is part of what the apps install. 

The local configs are things you mostly want to move, consisting of various customizations, dashboards and reports that have been shared at the app level.

  1. On your old server, open up the folder $SPLUNK_​HOME/​etc/​apps/​cisco_​cdr/​local
  2. Copy the files and folders in there to the new server, same location except do not copy the following files:
    1. Do not copy: inputs.conf
    2. Do not copy: indexes.conf
    3. Do not copy: sideview_license.conf

User-level config

When you are logged into Splunk and are using the app to create dashboards and reports, if the end results weren’t shared to ​“App” (again see Splunk’s docs on setting report permissions), those items may live in $SPLUNK_​HOME/​etc/​users/​USERNAME/​cisco_​cdr/​local/​.

There are several ways to deal with this:

  • Copy the entire $SPLUNK_​HOME/​etc/​users to the new server
    • If the migration to the new server involves ​“cleaning things up”, this will not accomplish that goal.
  • Copy individual users as necessary
  • Instruct users how to change permissions on the reports, dashboards and items as ​“shared in App”, which will move those reports out of the users folders and into the regular app-level

These strategies for migrating user content are only the tip of the iceberg — you’ll have to come up with your own plans. But as always, if you run into problems or need to bounce ideas around, contact us and we can help you out.



If you have any comments at all about the documentation, please send them to docs@​sideviewapps.​com.

Related

Administration

Define Quality Thresholds

Administration

Troubleshooting the Data Flow

Administration

Sinkhole vs Monitor Inputs

Administration

UF vs HF