Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
code share

UDP/TCP_Packet_Duplication

Problem this snippet solves:

This iApp provides full configuration of UDP/TCP packet duplication. It is commonly used to duplicate Syslog, SNMP Traps, Netflow, and Sflow data streams to multiple vendor solutions or customers. It also provides fault tolerance capabilities within each duplicated destination. By pointing Network devices, Appliances, and Servers to a VIP distributing network management traffic modifying distribution of streams can be done in one centralized location. UDP packets retain the original source address when sending to the destination locations.

Notes:

  • Prior to 11.5 you must add an IPv6 address to any interface to allow for HSL traffic to be sent to the distribution virtual fdf5::1/64 fdf5::2/64 for an HA pair would do it.* TCP traffic does not maintain original source
  • Internal F5 Resources can demo this solution within the UDF environment using the blueprint named "Traffic Duplication Demo"

Contributed by: Ken Bocchino

Comments on this Snippet
Comment made 27-Oct-2015 by Patricia Gonzalez 0
I am very interested in this! Has anyone tested?
0
Comment made 12-Nov-2015 by Ken Bocchino
Yes, this is in may working environments (and just updated to version 2.0) let me know if you have any issues using it.
0
Comment made 19-Nov-2015 by bigipjr28 371
How would you configure syslog for instance that require UDP duplication. What would be the desintation and the primary IPs..I only see the primary IP text box Any help is great thanks
0
Comment made 19-Nov-2015 by bigipjr28 371
How would you configure syslog for instance that require UDP duplication. What would be the desintation and the primary IPs..I only see the primary IP text box Any help is great thanks
0
Comment made 04-Mar-2016 by Runo Førrisdahl 0
Hi Ken, Thanks for an interesting solution. I've had som issues with this on 11.6 duplication UDP syslogs. It starts off just fine and can work great for X time. Then something causes it to leak packages. It only leaks packages related to the duplication VIPs. Have you seen this kind of behavior?
0
Comment made 05-Jul-2016 by ep 0
Ken, I'm trying to use v2.2 of this iApp to duplicate snmptraps to multiple trap receivers. For some traps, it is working great. For others, though, they aren't getting duplicated. I have a packet capture showing that two nearly identical traps behave differently on the F5. What is the best way to troubleshoot the iApp? Thanks, Brian
0
Comment made 08-Jul-2016 by ep 0

Looks like my issue disappeared. It is working quite well at the moment. Thanks! ep

0
Comment made 08-Aug-2016 by Sanjeev N G 0

Hi Ken,

I have installed iApp on 11.4.1 HF8 but i am getting below error when trying to implement.

Error parsing template:can't eval proc: "script::run" can't find package iapp 1.1.0 while executing "package require iapp 1.1.0" (procedure "script::run" line 2) invoked from within "script::run" line:1

0
Comment made 26-Sep-2016 by Scott Crawford 1

Anyone using this with route domains? I'm playing with it (in route domains) and not having luck. Unsure if it's the RD or something else.

0
Comment made 11-Nov-2016 by Sanjeev N G 0

Hi Ken,

I have installed Version 2.2 on 11.6.0 HF6, i am not able to get this working. When i grep for log i see below error in log.

warning mcpd[5663]: 01071859:4: Warning generated : /Common/Splunk_duplication.app/ir_Splunk_duplication_udp_spray:17: warning: [use curly braces to avoid double substitution][[string length $destination]] warning mcpd[5663]: 01071859:4: Warning generated : /Common/Splunk_duplication.app/ir_Splunk_duplication_distribute:14: warning: [use curly braces to avoid double substitution][![ catch { pool [lindex $nodeandport 0] } ]]

Add i do not see any traffic or any activity happening.Please let me know how to solve the issue.

0
Comment made 09-Dec-2016 by Sp33dy 1

Hi,

I'm also using this duplicator and it works fine. The only thing is that it uses source port 0 for traffic sent to the 2 destinations. According to RFC Firewalls don't allow this traffic with source port 0. Do you guys have the same issue or am i doing something wrong?

Please let me know.

Thx.

0
Comment made 14-Dec-2016 by Mauz 158

Does this IApp works if the clone is in a different subnet from the LTM's subnet

0
Comment made 14-Dec-2016 by Mauz 158

Does this IApp works if the clone pool member is in a different subnet from the LTM's subnet?

0
Comment made 15-Dec-2016 by Sp33dy 1

I have it working to two different IP's in different subnets. Just make sure you're routing is ok.

0
Comment made 01-Mar-2017 by edolton 0

@Sp33dy

I see the same thing with the source port being 0. Its an issue for me since they want the backend servers to ACK the traffic. Let me know if you found a solution

0
Comment made 02-Mar-2017 by Sp33dy 1

@edolton

I fixed this by stripping of the restriction from the IAPP and change the sourceport to preserve on the Virtual Server (both the virtual servers created by the IAPP). Now everything works fine!

Regards,

Maarten

0
Comment made 02-Mar-2017 by edolton 0

Thanks! i made the same change. It doesn't seem to keep the original source but increments a non-zero port 9011 then 9012 then 9013 etc. I'll see if this works for me. Thanks!

0
Comment made 19-Jun-2017 by kdt0078 16

Having an issue with this iAPP on 11.6.1. It looks like it creates two virtual servers xxx_distribute and xxx_udp. Looks like the destination address on the xxx_distribute is a dummy ipv6 address and the xxx_udp virtual server is not forwarding traffic.

Has this been plug-n-play for those of you who have it working?

0
Comment made 08-Nov-2017 by Jacob Creech 13

I am wanting to use this IApp as well and would like to know if there is some documentation on this to explain the setup process a little more.

Currently, I have UDP traffic coming in on 7 different ports to a server. When these packets are successfully written to a database table the service will send an ACK back to the device. So I would like to use this IApp to keep current traffic going to my production environment as well as duplicate this traffic to a QA server but not allowing the ACK to be sent back from the QA server. Will this be possible with this IApp.

0
Comment made 08-Dec-2017 by Jacob Creech 13

I finally got this IApp working.... FYI I had to remove tags from the VLANs if I used tags it would not replicate the traffic once I removed them it started working.

Now I have a new issue I am getting the incoming traffic but the ACT is not making it back to the device. I can see that the ACT is being generated and sent from the server but it is not making it back to the device. Any help would be greatly appreciated.

0
Comment made 11-Dec-2017 by Jacob Creech 13

@ Ken Bocchino,

Any help would be greatly appreciated. I am not able to get the ACT back to the device.

0
Comment made 14-Dec-2017 by Jacob Creech 13

kdt0078 the dummy IPV6 actually is used like a loop back to duplicate the packet. I had a sumulare issue I was able to resolve the issue by making my vlans untagged. Dont know why this fixed my issue but it did. Try it

0
Comment made 14-Feb-2018 by tdelamatre 0

Does this iApp support multiple "profiles"? For example, suppose we have sources A, B, C defined by loopback subnets and destinations X, Y, Z as NetFlow collectors. Can I send A->XY, B->XYZ and C->YZ or similar combinations all using a single VIP?

0
Comment made 24-Mar-2018 by DamonL 64

We have a situation where we want to duplicate TCP packets, but our receivers can only receive UDP. Would it be possible to protocol convert before duplication?

0
Comment made 13-Apr-2018 by Sergi0 63

I tried v2.2 on tmos v13.1, does not work for me. Does anybody use it with v13?

0
Comment made 16-Apr-2018 by Jacob Creech 13

No, I have not tried this on v13 the last version I tried this on was v12.1.1. And would not expect F5 to updated this iApp. I worked extensively with F5 to get this iApp to send and ACT back to the device and at the end of a 2 month, POC was told this is not a supported F5 iApp.

0
Comment made 4 months ago by ChuckR 53

We plan to upgrade to 13.1, has anyone gotten this to work on 13.1? Or has anyone come up with another way maybe? Thank you,

0
Comment made 3 weeks ago by Ryan 380

I took Ken's excellent work and made it work for my particular use case. Sharing here in case it helps somebody else.

I un-iApp-ified it, added route domain support, fixed the pool problem, and it works great on 13.1 for me.

-- Create two VIPs

Image Text

-- Create datagroup (nf_destinations.dg) and add IPs you want to send netflow/syslog to with string as the IP and port as the value

-- Create pool (nf_distribute.pool) that has a member of the distribute VIP

-- Create UDP profile and assign to both VIPs (collector and distributor) assign immediate timeout and enable datagram lb

-- Create two iRules, and assign to the VIPs accordingly

# nf_collector.irule 
# Acquire UDP Netflow packet from collector and distribute
when CLIENT_ACCEPTED {
    # Get source IP and break-out into variables
    scan [IP::client_addr] %d.%d.%d.%d a b c d
    # Insert placeholder in UDP datagram for our source/dest embed
    UDP::payload replace 0 0 [binary format ssssa256 255 255 255 255 [string repeat "~" 256]]
    # Set HSL distribute pool side-channel
    set hsl [HSL::open -proto UDP -pool nf_distribute.pool]
    # Iterate over Netflow Destinations (via established datagroup)
    set id [class startsearch nf_destinations.dg]
    while { [class anymore nf_destinations.dg $id] } {
        set destinationelement [class nextelement nf_destinations.dg $id]
        set destination [lindex $destinationelement 0]
        set destinationwithpad "$destination[string repeat "~" [expr 256 - [string length $destination]]]"
        # Embed source/dest and send to side-channel
        UDP::payload replace 0 264 [binary format ssssa256 $a $b $c $d $destinationwithpad]
        HSL::send $hsl "[UDP::payload]"
        # Uncomment to help debug the collector
        # log local0. "\[NF_COLLECTOR\] :: $destinationwithpad"
    }
    # Drop packet... no longer need
    discard
}

# nf_distribute.irule 
# Acquire UDP Netflow packet from collector and distribute
when CLIENT_ACCEPTED {
    # Get embedded source/dest information from UDP payload, assign to variables
    binary scan [UDP::payload] ssssa256a* a b c d destinationwithpad data
    # Assign destination (and remove padding)
    set destination [findstr $destinationwithpad "" 0 "~"]
    # Source NAT packet so it comes from original source -- Add %route_domain after $d if you need route domain support
    snat "$a.$b.$c.$d"
    # Remove embedded source/dest information from UDP payload, leave original data
    UDP::payload replace 0 [UDP::payload length] $data
    # Send to embedded node (add %route_domain after $destination if you need route domain support and you do not include in the datagroup)
    node $destination:9996
    # Uncomment to help debug the distributor
    # log local0. "\[NF_DISTRIBUTOR\] :: $a.$b.$c.$d \-\-\> $destination 9996"
}

Boom. Netflow Replicator without paying 20k for a replication VM.

To be determined if this is resource-prohibitive however...

0