Using two Virtual Servers to limit duplicate syslog messages reaching a Splunk Forwarder.

I have a requirement to limit duplicate syslog messages reaching our dual site splunk deployment and to only forward logging messages received at a second site if the first site has an issue.

 

All my current syslogging devices forward to two LogLogic sylog servers. however, i am replacing Loglogic with a splunk solution and these duplicated logging messages will eat twice as many splunk volume licenses when they both hit the forwarding layer. To save removing one logging destination from each network device, i'd like to harness the power of our dual site BIG-IP-2200s to solve the problem.

 

I'd like to set up one VS at each site to assume the current logging destination IP address of the existing Loglogic deployment, i'd then like one VS to forward syslog to its local Splunk deployment, but i'd like the other VS to forward syslog to the bin bucket/NULL whilst the health of the first VS is good. However, if the first VS fails (with its own health monitor of the local splunk service), the second VS will need to stop sending received syslog messages to the bin bucket/NULL and instead pick a second pool member which will forward to the second instance of splunk.

 

My first question is around how to effectively throw network traffic in the 'bin' if I'm happy with the state of an attached health monitor.

 

My second question is around using health monitors to assess the health of another Virtual Server on another F5.

 

Hopefully if the attached images are visible, they should explain more of the details