We get asked fairly regularly about moving our app from one server to another. The answer to this question is very dependent on your environment.
In the below we talk about a migration strategy for a single standalone Splunk server with our apps on it, that’s acting as your SFTP server for your CUCM implementation, and doing no other duties.
We also include the reasonably easy extension to that situation of having your SFTP server be on another server which uses the Splunk Universal Forwarder (UF) to send the data in to the Splunk server.
Really, these are important, because they are quite load bearing.
- We cannot guarantee this process will work. You should assume it won’t, and that you should have tested backups and a restore plan.
- If you haven’t tested your backups, it’s as good as not having backups at all. Test them!
- If you haven’t tested your *restore* plan, it’s also just as good as having no backups at all.
- Much of the below relies on steps being done in the right order.
- Most important is how and when you copy/move the data between systems.
- Really. You can wipe it all out if you do it in the wrong order.
- So pay attention.
- But also there are other gotchas, so you *still* have to pay attention on all the other steps.
- There are many spots in the below where you have to confirm operation of systems before moving forward.
- Don’t skip those steps.
- Don’t think you can come back to them later (see #2 above!)
- Carefully confirm everything at each step, investigate and resolve any inconsistencies or anomalies before proceeding.
Also, if you are completely new to Splunk and your current environment, here’s a great set of docs pages from Splunk on what to do when you have inherited a deployment you know nothing about. For our purpose, the most useful page of this is the one on the deployment topology.
Anyway, with that out of the way, let’s get started!
Wait – did I mention we’re not responsible if this goes awry? It should work, it HAS worked, it can work again, but we don’t know your environment or your skill level! There may be tiny steps I’ve missed that rely on you having a pretty good idea what all these steps are accomplishing. Luckily, in most environments, if push comes to shove you can probably export a lot of historical data from CUCM again and just re-ingest it. There’s a few caveats to that, but it generally works fine.
New Server Prep (can be done ahead of time)
Get Splunk, the Cisco CDR app, Sideview Utils and Canary all installed on the new server following these two sections of our docs:
Put in your Cisco CDR license key using our app’s Settings/Update License
Confirm any accounts are created in Splunk, email settings are right and so forth. For *this* step, most of this can all be done later too.
One of these important steps: Do NOT at this time set up a data input, we’ll use the fact there isn’t one set up later as an easier way to confirm SFTP is working.
If your SFTP server lives on this server too, then DO set up the SFTP server with the same user and password as the old one. You can change the username or the password for the SFTP account at this time if you want, but I’m recommending you not do that to keep our change surface area as small as possible.
Confirm you have Splunk running, our three apps on it, that you can log into it, that it is not in any way ingesting any CDR or CMR files.
Once it’s ready, shut off Splunk on it via services or via the command line. Leave the SFTP server running.
Old Server Prep (can be done ahead of time)
In old Splunk, click Splunk’s Settings/indexes. Find out the physical path to where the index cisco_cdr is. Note this value.
Make sure all Searches, Reports and Alerts that you want to migrate (e.g. “all valid and current ones”) are set to be shared in the app and not just private. This will make them easier to move later.
- Click Splunk’s Settings/Searches Reports and Alerts
- For each report you know you need, check the column “Sharing”.
- If it’s “Private”, then for that report click Edit, then Edit Permissions, change it to “Display for… App” and click save.
- This moves the actual report definition (from somewhere deep in a weird user folder) into the app’s “local” folder, so it’s easier to find and copy. (more info on that later)
Migration of historical data (At Migration time!)
- Shut off the old SFTP server. CUCM will just queue up files for hours at least (default settings store 5 GB I think, which is like several weeks’ worth).
- Wait a minute or two (shouldn’t take more than a few seconds, so a minute or two is enough) for the last files to get ingested.
- Stop Splunk on the old server. Wait until it’s stopped.
- Open up and look in the cisco_cdr index location (found previously, likely to be c:\program files\splunk\var\lib\splunk\cisco_cdr or /opt/splunk/var/lib/splunk/cisco_cdr).
- Copy the folders inside that folder to the same place on the new server. (They’re named like “colddb”, “datamodel_summary”, “db”, and “thaweddb”. It’s likely the only real data will be in colddb and db, but copy all of them over just in case.
- It may, but shouldn’t, ask to overwrite some files. Overwrite them if it does ask.
That right there should let you start Splunk on the new server and confirm that it Browse Calls has your historical data. (Note customized reports and sites and stuff aren’t moved yet)
Once confirmed, continue:
- Migrate the CUCM billing server to point to the new SFTP server.
- You should only have to change the “External Billing Server’s” settings to point to the new location, as per the specific section here:
- You should see files start building up in wherever the SFTP server saves files (assuming you have completed phone calls during the past minute!).
- If you do not see any files in a while (you can place a phone call and hang up to trigger it to send in files the next minute), then see step 1 here on restarting a service in CM
- Now that you know CM is sending SFTP files to the new location, you can start ingesting those new files, so set up the data input again.
- I’d give you a link, but you should just need to fill in the data path in our app’s Setup/Set up Data Inputs page.
- All the files in the SFTP folder should disappear within a minute or so (maybe practically immediately!)
Confirmation of data
After this, you should see Browse calls not only having old data, but also having new data. (It might take a few minutes to “catch up” from the backlog.)
Assuming that all looks good, now we can migrate other things.
Migrate “system” stuff (groups, sites, etc…)
- In your c:\program files\splunk\etc\apps\cisco_cdr\lookups are a few files to copy into their new location:
- Just put them in the same location on the new server, overwriting whatever’s there.
Migrate custom reports and alerts
- Most, but not all, of the following folder’s contents should be copied:
- c:\program files\splunk\etc\apps\cisco_cdr\local
- EXCEPTIONS – do not copy indexes.conf, sideview_license.conf
- Otherwise copy all the rest, including that data subfolder.
Migrate user stuff.
Do note you may not want to do this wholesale. You might want to pick and choose, leave the old server around for a while (alternate to that below) and just copy things as you need them.
Also note that if you had moved everything that’s needed to being shared in “App”, then mostly everything you want is in the local folder we just moved.
So maybe you don’t want to do this at all!
Anyway, if you did still want to do this, see “user-level config” at https://sideviewapps.com/apps/cisco-cdr-reporting-and-analytics/documentation/administrative-concepts/migrating-to-a-new-splunk-deployment/
At this time, it should all be done.
Read more posts in the Cisco CDR General Splunk category