Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Answers

iRule - URI/Referer Rate limit per minute

Dears,

We have a customer which is getting some troubles with access coming from an specific referer/uri. These access are overloading the application, causing a lot of troubles to the business. As we could not find an ASM countermeasure to avoid this behavior, we are trying to create an iRule to redir these access to another webserver with a static page. Before using it in production environment we are performing some tests, but it´s not working properly.

The completely code follows. The intention is to allow a maximum of 3 access per minute from an specific URI or Referer string. But when we activate the iRule no access are droped after three requests during a minute.

Could you help us with this troubleshooting?

Regards

when RULE_INIT { set static::maxRate 3 set static::windowSecs 60 }

when HTTP_REQUEST { if { ([HTTP::method] eq "GET") and ([class match [string tolower [HTTP::uri]] ends_with URI_LIST_TO_LIMIT] ) } {

     #whitelist
    if { [class match [IP::client_addr] equals ips_whitelist] }{
       return
    }

    # set variables
    set limiter [string tolower [HTTP::uri]]
    set get_count [table key -count -subtable $limiter]

    # main condition
    if { $get_count < $static::maxRate } {
        incr get_count 1
         table set -subtable $limiter $get_count $limiter indefinite $static::windowSecs
    } else {
        log local0. "$limiter has exceeded the number of requests allowed."
        drop
        return
    }
}

}

0
Rate this Question

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

The problem is that you are creating a record with an "indefinite" timeout period meaning, it will never go away.

The idea is that once a record is created for a URI in your case, that record should expire and automatically get deleted after a period of no Hits (when the URI is not seen/used).

So, change this command:
table set -subtable $limiter $get_count $limiter indefinite $static::windowSecs

Make it look like this:

table set -subtable $limiter $get_count $limiter $static::windowSecs $static::windowSecs

Here is a link to a throttling irule very similar to the one you are using:
https://devcentral.f5.com/codeshare/http-request-throttle

Note: The iRule provided at the link does not use subtables and, rather than adding records and counting them, it increments the value held within a record. This is a much better approach. Better performance and saves a lot of memory. Also the iRule at the link throttles by IP address, it should be easy for you to modify it to throttle by URI. Your iRule may cause the Bigip to run out of memory under extremely heavy Load.

HTH

0
Comments on this Answer
Comment made 09-Dec-2015 by Bruno Esteves 12
Hi John, Thank you for your time. I tried adjust the table set like you said, but didn't work. It's possible see on var/log/ltm the increasing of get_count? I'm going to test this throttling irule today and see If everything works fine. Thanks. Bruno
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Hi Bruno,

the optimal TCL code for rate_limiting algorythms do strongly depend on your functional requirements. Someone may prefer a very fast but not that accurate algorithm (aka. see John's link) and others may want to use a very accurate algorithm using sliding time frames (aka. your posted code).

If your functional requirement mandates an algorythm that "garantees" less than X_Number of request to pass in a time frame of the "last" X_Seconds in every situation, then you have to stick with your solution.

But if you don't require such an accurate algorythm, then you should adopt the sample code provided by John to increase the overall performance. But be aware that the accuracy will then be decreased to some extend so that burst traffic may ocour (e.g. a settings of 3 request within 60 sec allows a well-timed request burst of up to 5 request within 1 secs (e.g. Burst Rate = rate_limit * 2 - 1 ).

Regarding your coding issues....

At least for the "#set variable" and "#main condition" part I can't find any coding issues. In my opinion your code represents is a good implementation of a typical sliding time frame algorithm. The only stuff I would like to recommentd, is to set a different KEY_VALUE. Instead of storing the URI as KEY_VALUE I would recommend to store a rather simple "1" (e.g. "table set -subtable $limiter $get_count 1 indef $static::windowSecs").

BTW: If the coding problems still persist then add some additional debug lines, to see whats happening behind the scenes and to allow pinpointing the section, command or variable that is causing trouble?

Cheers, Kai

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Hi,

Here a working irule (inspired by http request throttle irule) :

when RULE_INIT {
    set static::maxReqs 3;
    set static::timeout 60;
}

when HTTP_REQUEST {
        if { [HTTP::header exists "X-Forwarded-For"] } {
            set client_IP_addr [getfield [lindex  [HTTP::header values X-Forwarded-For]  0] "," 1]
        } else {
            set client_IP_addr [IP::client_addr]
        }
        if { ([HTTP::method] eq "GET") and ([class match [string tolower [HTTP::uri]] ends_with URI_LIST_TO_LIMIT] ) } {

            #whitelist
            if { [class match [IP::client_addr] equals ips_whitelist] }{
               return
            }
            set getcount [table lookup -notouch "$client_IP_addr:[HTTP::uri]"]
            if { $getcount equals "" } {
                table set "$client_IP_addr:[HTTP::uri]" "1" $static::timeout $static::timeout
            } else {
                if { $getcount < $static::maxReqs } {
                    table incr -notouch "$client_IP_addr:[HTTP::uri]"
                } else {
                    reject
                }
            }
        }
}
0
Comments on this Answer
Comment made 12-Apr-2017 by Bruno Esteves 12

Dears,

I'm trying to include a sliding window of time to counting requests. (e.g. if we have 8 req in 60 sec, block for 600 sec.)

So, I did change ($static::timeout $static::timeout) to ($static::timeout $static::winsec) and include this variable (set static::winsec 60). But, didn't work. Have I missed something here ?

Here a working iRule that in using:

when HTTP_REQUEST {

if { [HTTP::uri] ends_with "/URI" and [HTTP::method] eq "POST"}{

    set static::maxRate 8
    set static::timeout 600
    set client_IP_addr [IP::client_addr]

    set getcount [table lookup -notouch "$client_IP_addr:[HTTP::uri]"]
    if { $getcount equals "" } {
    table set "$client_IP_addr:[HTTP::uri]" "1" $static::timeout $static::timeout
    } else {
    if { $getcount < $static::maxRate } {
    table incr -notouch "$client_IP_addr:[HTTP::uri]"
    } else {
    log -noname local0. "REQUEST Rejected: current requestCount for $client_IP_addr"
    reject
}
}
}
}

Cheers, Bruno

0