Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Clear all filters
Answers

Rate limit HTTP requests based on cookie

Hi guys

Below you will see an irule designed to rate limit HTTP requests to spesific URIs. The detection is being performed based on client IP address. On the other hand, the client IP could represent much more than 1 unique client because of NATted IP addresses. Could anybody advice how to modify this irule to catch unique clients? Would it be possible to modify this based on cookie control? Or is there any different irules for this purpose? Thanks a lot.

when RULE_INIT { set static::maxRate 3 set static::windowSecs 1 }

when HTTP_REQUEST { if { ([HTTP::method] eq "GET") and ([class match [string tolower [HTTP::uri]] starts_with DATAGROUP-RATELIMIT-URI] ) } {

    # whitelist
    if { [class match [IP::client_addr] equals DATAGROUP-RATELIMIT-WHITELIST] }{
       return
    }

    # set variables
    set limiter [string tolower [HTTP::uri]]
    set clientip_limitervar [IP::client_addr]:$limiter
    set get_count [table key -count -subtable $clientip_limitervar]

    # main condition
    if { $get_count < $static::maxRate } {
        incr get_count 1
         table set -subtable $clientip_limitervar $get_count $clientip_limitervar indefinite $static::windowSecs
    } else {
        log local0. "$clientip_limitervar has exceeded the number of requests allowed."
        drop
        return
    }
}

}

0
Rate this 0

Replies to this 0

placeholder+image

You may use a hash of the following info : - Host header - Accept-Language - Accept-Encoding - Accept-Charset - User-Agent - URI

you can use sha1, sha256, etc. command to hash :

set myhash [sha1 "[HTTP::header Host][HTTP::header Accept-Language][HTTP::header Accept-Encoding][HTTP::header Accept-Charset][HTTP::header User-Agent][HTTP::uri]"]

You can set the hash the key for your subtable.

One another tip would be to insert a javascript on client side to inspect browser settings. This is commonly used by bot detection systems. Have a look at the ASM Bot Detection technique to see how it works.

0
Comments on this Reply
Comment made 27-Aug-2014 by Erol Dogan 2
What is the use of generating a hash? That hash is not unique per client as far as I understand. How could it resolve the issue? Could you kindly elaborate your proposal please?
0
Comment made 27-Aug-2014 by Yann Desmarest 4489
As you can't rely only on IP address, given that X-Forwarded-xxx is not a mandatory header and that firewall don't insert this header, you can just rely on a session cookie or a browser fingerprint. I let you log [HTTP::header Host][HTTP::header Accept-Language][HTTP::header Accept-Encoding][HTTP::header Accept-Charset][HTTP::header User-Agent][HTTP::uri] to see what kind of value you can expect. That's why, doing a hash will help you normalize your fingerprint. You just have to do the hash operation for each request. the command I proposed can help you doing rate limiting on URI.
0
placeholder+image

Could anybody advice how to modify this irule to catch unique clients?

is there x-forwarded-for header? if yes, can we use it (instead of source ip)?

Would it be possible to modify this based on cookie control?

can we check and generate cookie similar to the one in the following article?

CodeShare Refresh: HTTP Session Limit by Colin Walker
https://devcentral.f5.com/articles/codeshare-refresh-http-session-limit

e.g.

root@(ve11a)(cfg-sync In Sync)(Active)(/Common)(tmos)# list ltm rule qux
ltm rule qux {
    when RULE_INIT {
  set static::maxRate 3
  set static::windowSecs 1
}
when HTTP_REQUEST {
  if { ([HTTP::method] eq "GET") and
    ([class match [string tolower [HTTP::uri]] starts_with DATAGROUP-RATELIMIT-URI] ) } {

    # whitelist
    if { [class match [IP::client_addr] equals DATAGROUP-RATELIMIT-WHITELIST] }{
      return
    }

    if { [HTTP::cookie exists "ClientID"] } {
      set need_cookie 0
      set client_id [HTTP::cookie "ClientID"]

      # set variables
      set limiter [string tolower [HTTP::uri]]
      set clientip_limitervar ${client_id}:${limiter}
      set get_count [table key -count -subtable $clientip_limitervar]

      # main condition
      if { $get_count < $static::maxRate } {
        incr get_count 1
        table set -subtable $clientip_limitervar $get_count $clientip_limitervar indefinite $static::windowSecs
      } else {
        log local0. "$clientip_limitervar has exceeded the number of requests allowed."
        drop
        return
      }

    } else {
      set need_cookie 1
      set client_id [format "%08d" [expr { int(100000000 * rand()) }]]
    }
  }
}
when HTTP_RESPONSE {
  if { $need_cookie } {
    HTTP::cookie insert name "ClientID" value $client_id path "/"
  }
}
}
0
Comments on this Reply
Comment made 27-Aug-2014 by Erol Dogan 2
The clients would be NATted behind their corporate firewall/ home modem/ ISP routers etc. So even if we use the x-forwarded-for header info, it would never mean that an IP is equal to a unique client. I will try to understand and test the irule you suggested.
0
Comment made 28-Aug-2014 by Erol Dogan 2
I have tested the irule provided above but unfortunately the connection is always successfull for all requests. Have you ever made this irule work? Thanks for your time.
0
placeholder+image

I have tested the irule provided above but unfortunately the connection is always successfull for all requests.

were you accessing the same url and providing the same cookie in every request?

Have you ever made this irule work?

i never used it in production. i did test it in lab using ab (apachebench).

0
Comments on this Reply
Comment made 28-Aug-2014 by Erol Dogan 2
I was accessing the exact same URL and providing the same cookie in every request. What I actually did is open the application and when I accessed to the index page I refreshed the page (CTRL+F5) several times. I can see the below statistics for the corresponding irule executions. Does it give you any clue to tshoot :) Event Type Total Executions Failures Aborts HTTP_RESPONSE 662 0 0 RULE_INIT 1 0 0 CLIENT_CLOSED 1664 1662 0 HTTP_REQUEST 662 0 0
0
Comment made 28-Aug-2014 by nitass 13357
is there nothing in /var/log/ltm? can you try to add some logging in the irule?
0
Comment made 28-Aug-2014 by Erol Dogan 2
Sth like this: ug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: Resuming log processing at this invocation; held 158 messages. Aug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: TCL error: /Common/Cookie <CLIENT_CLOSED> - can't read "timer": no such variable while executing "after cancel $timer" Aug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: TCL error: /Common/Cookie <CLIENT_CLOSED> - can't read "timer": no such variable while executing "after cancel $timer" Aug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: TCL error: /Common/Cookie <CLIENT_CLOSED> - can't read "timer": no such variable while executing "after cancel $timer" Aug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: TCL error: /Common/Cookie <CLIENT_CLOSED> - can't read "timer": no such variable while executing "after cancel $timer" Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: TCL error: /Common/Cookie <CLIENT_CLOSED> - can't read "timer": no such variable while executing "after cancel $timer" Aug 28 07:15:04 f5demo err tmm[9909]: 01220001:3: Per-invocation log rate exceeded; throttling. Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509 Aug 28 07:15:04 f5demo info tmm[9909]: Rule /Common/Cookie <HTTP_REQUEST>: 24601509
0
placeholder+image

i think i do not see after command in the irule you posted. what irule are you using? can you post it here?

# tmsh list ltm rule (name)
0
Comments on this Reply
Comment made 28-Aug-2014 by Erol Dogan 2
root@(f5demo)(cfg-sync Standalone)(Active)(/Common)(tmos)# list ltm rule Cookie ltm rule Cookie { when RULE_INIT { set static::max_active_clients 10 } when HTTP_REQUEST { # test cookie presence if {[HTTP::cookie exists "ClientID"]} { set need_cookie 0 set client_id [HTTP::cookie "ClientID"] log local0. "$client_id" # if cookie not present & connection limit not reached, set up client_id } else { if {not ([table keys -subtable httplimit] > $static::max_active_clients)} { set need_cookie 1 set client_id [format "%08d" [expr { int(100000000 * rand()) }]] # Only count this request if it's the first on the TCP connection if {[HTTP::request_num] == 1}{ table set -subtable httplimit [IP::client_addr]:[TCP::client_port] "blocked" set timer [after 60000 -periodic { table lookup -subtable httplimit [IP::client_addr]:[TCP::client_port] } ] } } else { HTTP::redirect "http://sorry.domain.com/" event CLIENT_CLOSED disable return } } } when HTTP_RESPONSE { # insert cookie if needed if {$need_cookie == 1} { HTTP::cookie insert name "ClientID" value $client_id path "/" } } when CLIENT_CLOSED { # decrement current connection counter for this client_id after cancel $timer table delete -subtable httplimit [IP::client_addr]:[TCP::client_port] } } (END)
0
placeholder+image

i think you misread what i posted. the Colin irule does not do the same thing as what your irule does (his irule does session limit but you do rate limit). what i meant is to show you how he manages cookie.

0
placeholder+image

Sorry Nitass, my fault.

When I use the irule without the last part quoted below it seems to be working, but when I use it as a whole the page never comes up. Could you advice for one more time. Thanks.

when HTTP_RESPONSE { if { $need_cookie } { HTTP::cookie insert name "JSESSIONID" value $client_id path "/login.php" } }

The whole irule is:

when RULE_INIT { set static::maxRate 3 set static::windowSecs 1 } when HTTP_REQUEST { log local0. "You're here" if { ([HTTP::method] eq "GET") and ([class match [string tolower [HTTP::uri]] starts_with DATAGROUP-RATELIMIT-URI] ) } { log local0. "You're here1"

# whitelist
if { [class match [IP::client_addr] equals DATAGROUP-RATELIMIT-WHITELIST] }{
  return

log local0. "You're here2"
}



if { [HTTP::cookie exists "JSESSIONID"] } {
log local0. "You're here3"
  set need_cookie 0
log local0. "You're here4"
  set client_id [HTTP::cookie "JSESSIONID"]
log local0. "You're here5"



  # set variables
  set limiter [string tolower [HTTP::uri]]
  set clientip_limitervar ${client_id}:${limiter}
  set get_count [table key -count -subtable $clientip_limitervar]
log local0. "$get_count"  

  # main condition
  if { $get_count < $static::maxRate } {
    incr get_count 1
    table set -subtable $clientip_limitervar $get_count $clientip_limitervar indefinite $static::windowSecs
  } else {
    log local0. "$clientip_limitervar has exceeded the number of requests allowed."
    drop
    return
  }

} else {
  set need_cookie 1
  set client_id [format "%08d" [expr { int(100000000 * rand()) }]]
}

} }

when HTTP_RESPONSE { if { $need_cookie } { HTTP::cookie insert name "JSESSIONID" value $client_id path "/login.php" } }

0
placeholder+image

but when I use it as a whole the page never comes up. Could you advice for one more time.

JSESSIONID cookie is generated by server, isn't it? if yes, i do not think you need the HTTP_RESPONSE part because it will insert one more JSESSIONID cookie (with different value) which may affect application behavior.

0
Comments on this Reply
Comment made 29-Aug-2014 by Erol Dogan 2
Yes, you're right. JSESSIONID is the cookie generated by server. So, it works. Many thanks.
0