General Splunk
Migrations, and Moving Our App to a New System
June 19th, 2020
NOTICE as of December 2, 2022, this entire topic has been updated and merged into our official docs on the page Migrating to a New Splunk Deployment.
We get asked fairly regularly about moving our app from one server to another. The answer to this question is very dependent on your environment.
In the below we talk about a migration strategy for a single standalone Splunk server with our apps on it, that’s acting as your SFTP server for your CUCM implementation, and doing no other duties.
We also include the reasonably easy extension to that situation of having your SFTP server be on another server which uses the Splunk Universal Forwarder (UF) to send the data in to the Splunk server.
Important notice:
Really, these are important, because they are quite load bearing.
- We cannot guarantee this process will work. You should have tested backups and a restore plan.
- If you haven’t tested your backups, it’s as good as not having backups at all. Test them!
- If you haven’t tested your *restore* plan, it’s also just as good as having no backups at all.
- Much of the below relies on steps being done in the right order.
- You can wipe it all out if you do it in the wrong order, or copy in the wrong direction, so pay attention to the order and general flow of things.
- There are many spots in the below where you have to confirm operation of systems before moving forward.
- Don’t skip those steps.
- Don’t think you can come back to them later (see #2 above!)
- Carefully confirm everything at each step, investigate and resolve any inconsistencies or anomalies before proceeding.
Also, if you are completely new to Splunk and your current environment, here’s a great set of docs pages from Splunk on what to do when you have inherited a deployment you know nothing about. For our purpose, the most useful page of this is the one on the deployment topology.
Enough with the scary bits, Rich. Can you just tell me what to do?
Yes, I can, but just remember, we’re not responsible for this going sideways. 🙁
New Server Prep (can be done ahead of time)
Get Splunk, the Cisco CDR app, and Canary all installed on the new server following these two sections of our docs:
and
Put in your Cisco CDR license key using our app’s Settings/Update License
Confirm any accounts are created in Splunk, email settings are right and so forth. For *this* step, most of this can all be done later too.
One of these important steps: Do NOT at this time set up a data input, we’ll use the fact there isn’t one set up later as an easier way to confirm SFTP is working.
If your SFTP server lives on this server too, then DO set up the SFTP server with the same user and password as the old one. You can change the username or the password for the SFTP account at this time if you want, but I’m recommending you not do that to keep our change surface area as small as possible.
Confirmation
Confirm you have Splunk running, our two apps on it, that you can log into it, that it is not in any way ingesting any CDR or CMR files.
Once it’s ready, shut off Splunk on it via services or via the command line. Leave the SFTP server running so that if we point CUCM at it the files will start accumulating in the SFTP server’s drop folder.
Old Server Prep (can be done ahead of time)
In old Splunk, click Splunk’s Settings/indexes. Find out the physical path to where the index cisco_cdr is. Note this value.
Make sure all Searches, Reports and Alerts that you want to migrate (e.g. “all valid and current ones”) are set to be shared in the app and not just private. This will make them easier to move later.
- Click Splunk’s Settings/Searches Reports and Alerts
- For each report you know you need, check the column Sharing.
- If it’s “Private”, then for that report click Edit, then Edit Permissions, change it to “Display for… App” and click save.
- This moves the actual report definition (from somewhere deep in a weird user folder) into the app’s “local” folder, so it’s easier to find and copy. (more info on that later)
Migration of historical data (At Migration time!)
- On the old server, turn off the data inputs for our app.
- It should be in Splunk’s Settings, Data inputs, Files & Directories.
- For both the CDR and the CMR input, click Disable
- Stop Splunk on the old server.
- Open up and look in the cisco_cdr index location (found previously, likely to be c:\program files\splunk\var\lib\splunk\cisco_cdr or /opt/splunk/var/lib/splunk/cisco_cdr).
- Copy the folders inside that folder to the same place on the new server.
- Remember, the new server does NOT have Splunk actually running at this time!
- The folders are named like “colddb”, “datamodel_summary”, “db”, and “thaweddb”. It’s likely the only real data will be in the db folder, but copy all of them over just in case.
- It may, but shouldn’t, ask to overwrite some files. Overwrite them if it does ask (of course, only after double-checking you are doing it the right direction!)
Intermediate Confirmation
You can now start Splunk on the new server and confirm that Investigate Calls has your historical data. (Note customized reports and sites and stuff aren’t moved yet).
You can leave Splunk running on the new server at this time.
Once confirmed, continue:
- Migrate the CUCM billing server to point to the new system.
- Generally you have to create a whole new “External Billing Server” as per the specific section here:
- https://sideviewapps.com/apps/cisco-cdr-reporting-and-analytics/documentation/installation-and-setup/callmanager-configuration/
- You should see files start building up in wherever the SFTP server saves files (assuming you have completed phone calls during the past minute!).
- If you do not see any files in a while (you can place a phone call and hang up to trigger it to send in files the next minute), then see step 1 here on restarting a service in CM
- https://sideviewapps.com/documentation/cisco-cdr-reporting-analytics-administration-troubleshooting-data-flow
- Now that you know CM is sending SFTP files to the new location you can start ingesting those new files, so set up the data input again.
- I’d give you a link, but you should just need to fill in the data path in our app’s Setup/Set up Data Inputs page.
- All the files in the SFTP folder should disappear within a minute or so (maybe practically immediately!)
- Finally, copy whatever data files came into the old system since we started this process into that same folder and that should catch the system up with all data.
Confirmation of data
After this, you should see Browse calls not only having old data, but also having new data. (It might take a few minutes to “catch up” from the backlog.)
Assuming that all looks good, now we can migrate other things.
Migrate “system” stuff (groups, sites, etc…)
- In your c:\program files\splunk\etc\apps\cisco_cdr\lookups are a few files to copy into their new location:
- cidr.csv
- groups.csv
- clusters.csv
- devices.csv
- Just put them in the same location on the new server, overwriting whatever’s there.
Migrate custom reports and alerts
- Most, but not all, of the following folder’s contents should be copied:
- c:\program files\splunk\etc\apps\cisco_cdr\local
- EXCEPTIONS – do not copy indexes.conf, sideview_license.conf
- Otherwise copy all the rest, including that data subfolder.
Migrate user stuff.
Do note you may not want to do this wholesale. You might want to pick and choose, leave the old server around for a while (alternate to that below) and just copy things as you need them.
Also note that if you had moved everything that’s needed to being shared in “App”, then mostly everything you want is in the local folder we just moved.
So maybe you don’t want to do this at all!
Anyway, if you did still want to do this, see “user-level config” at https://sideviewapps.com/documentation/cisco-cdr-reporting-analytics-administration-migrating-new-splunk-deployment
Completion
At this time, it should all be done.