If you do have the hardware budget we recommend setting up Distributed Search, ie deploying multiple Splunk instances set up as Splunk Indexers and one or more Splunk instances set up as dedicated Splunk Search Heads.
The Splunk docs will say it all better than we can. Just REMEMBER: on the indexing side our load will be almost negligible. It is on the search side that our load may be significant.
For more information, see the Splunk documentation on Distributed Deployments.
If you have any comments at all about the documentation, please send it in to email@example.com.