We are currently sending tons of log via HSL to our splunk server using UDP protocol. The reason we choose UDP is because we think that UDP have less performance impact than using TCP if target is unreachable.
We just upgraded our splunk installation to Splunk 5.0 and we found shortly after that Splunk 5.0 has issues with UDP input ( see : http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues#Data_input_issues )
I would like to find out what will be the impact if we switch our HSL to TCP mode especially when the target server is unreachable ?
The documentation scare me a bit by stating that TCP pool members has no way to limit or control the number of TCP connections opened to the target pool members.
The time before a pool marked as down is about 15 seconds.
Does anyone have any recommendation or any peculiarities information that you can share when running HSL in TCP mode ?