If you have the hardware budget, we recommend setting up Distributed Search, i.e. deploying multiple Splunk instances set up as Splunk Indexers, and one or more Splunk instances set up as dedicated Splunk Search Heads.
The Splunk docs will say it all better than we can. Just REMEMBER: on the indexing side, our load will be almost negligible. It’s on the search side that our load may be significant.
For more information, see the Splunk documentation on Distributed Deployment.
If you have any comments at all about the documentation, please send them to firstname.lastname@example.org.