You may recall me writing about iRules building heatmaps to allow you to graphically track your traffic.  I then expanded on the simple concept to allow you to view the world rather than just the US, and even went so far as to allow you to click around an interface to select which region of the world you wanted to zoom to for easier viewing. The next step in my planned evolution of this project is to allow you to even further refine your stats view to make the tool more useful than just a general “yep, we’re getting some traffic” view by allowing you to not only zoom to regions, but to filter based on URLs to see traffic for specific portions of your site.

If you take a minute and think about how that might be useful you, like me, will likely come up with several ideas for what you might want to track on a URL basis.  Add campaigns, user group viewing, regional localization of content, new site functionality or features, etc. can often all be tracked by clicks to a certain URL. Now with nothing but an iRule you can get a graphical view of who from around the world is making use of any number of things in your application.

Enough talk, let’s see the guts of this thing. First, I strongly recommend you check out at least Part 2 of this series for some reference, if not Part 1 & Part 2, because we’re going to be building off the iRule that was already completed there. Without pasting the code here, at the end of Part2 we had a working iRule that would collect stats for either countries or states, store that info in two separate tables, and take a number of different arguments to allow viewing a heatmap of different world regions, calling up and presenting the data from those tables as needed to google’s charting API.

In this iRule, we introduce a new and as it turns out very interesting challenge. How do we store multiple tiers of data in a table system? You see, tables are a fantastically powerful thing, but you can only have one layer of sub-tables.  What I needed to do was store both the country (or state) of origin for the request, as well as the URI that user was requesting, and then increment a counter for that set of data (I.E.  CA => /myURL => 15).  Due to the fact that I couldn’t nest tables I nearly scrapped tables all together and went with arrays, but that would lose a considerable amount of performance and scalability plus, what’s the fun without a challenge?

The solution is relatively simple once you get your head around it, all you need to do is build what’s generally known as a pointer table.  What is a pointer table?  Allow me to digress a moment.  To store the data I wanted, I would ideally have liked to create a table that looked like:

countries =>      
  CA =>    
    url1 => 15
    url2 => 7
    url3 => 27
  US =>    
    url1 => 15

And so on and so forth. Basically nested name value pairs so that if I want to recall all the data for CA or US I can do so by iterating through a loop.  Since I can’t do that in tables, I had to figure something else out. A pointer table is simply a table of table names.  My solution was to us the name of each table as the country or state, and within that table I’d store the URLs requested from that location, along with their counters.  What this does is give me my two layers of logical separation with two different tables one layer deep each.  So I end up with something like:

mytables =>  


state:CA =>    
  url1 => 15
country:CA =>    
  url1 => 27


So above you have a table named mytables with the names of each table that will do the actual tracking, prefixing each with either state or country so we know which is which, and then you have examples of two tables, state:CA and country:CA for California and Canada respectively, with the URLs they’re each tracking for their origin requests, and the number of requests for that URL from that origin.  That’s pointer tables in a nutshell.

With me so far? Okay…now let’s get to some actual code.

To start with, we have to change how we’re inserting the info into the tables for each request that comes through. The actual whereis lookups are the same, but the table addition part is different.  Let’s take a look at how we have to do this now:

   1: if {[table incr -subtable country:$cloc -mustexist [HTTP::uri]] eq ""} {
   2:   table set -subtable country:$cloc [HTTP::uri] 1 indefinite indefinite
   3: }  
   4: if {[table incr -subtable mytables -mustexist country:$cloc] eq ""} {
   5:   table set -subtable mytables country:$cloc 1 indefinite indefinite
   6: }  
   7: if {$cloc eq "US"} {  
   8:   if {[table incr -subtable state:$sloc -mustexist [HTTP::uri]] eq ""} {
   9:       table set -subtable state:$sloc [HTTP::uri] 1 indefinite indefinite
  10:   }  
  11:   if {[table incr -subtable mytables -mustexist state:$sloc] eq ""} {
  12:       table set -subtable mytables state:$sloc 1 indefinite indefinite
  13:   }    
  14: }    

The concept stays the same, as you can see, but we’re now doubling up our inserts. Every time we insert create or increment an entry for a URL into a table, we’re making sure there’s an entry for that country or state in the pointer table “mytables”. This way we can iterate through mytables and find all the tables that we need to scour for entries later. It’s important to note that we’re using the state: and country: naming scheme here when creating the names of the tables so we can differentiate between the two later, and also that we’re adding entries tracked by URL not their origin, since the table itself tells us the origin.

Next comes the tricky bit, how do we iterate through the different tables as necessary when someone wants to view the actual heatmap? Before it was 4 simple lines of code. I’m afraid it’s grown to a bit more than that since we have to nest loops and have different cases for countries vs. states.  Let’s take a look at the simplest form of the nested loop we need to make this happen:

   1: foreach mysub [table keys -subtable mytables] {
   2:   foreach myurl [table keys -subtable $mysub] {
   3:     append chld "[getfield $mysub ":" 2]"
   4:     append chd "[table lookup -subtable $mysub $myurl],"
   5:   }
   6: }

First, you have the foreach mysub line that loops through each entry in the mytables table, which if you recall is full of entries that look like state:WA and country:DE which are really just table names as I stated above when creating the table entries.  For each of the tables within that pointer table, you’re looping through every name/value pair and appending the pertinent info to the same chld and chd variables.  Note that I had to use getfield for the actual origin since the table’s name starts with a state: or country: prefix.

So that’s a simple version, but that would just give us all the info for every URI, which isn’t what we want. The entire purpose of building this more advanced solution was to select individual URIs to filter on.  To do this, we have to introduce a couple things. One is the concept of zoom, which we’ve worked with before. This is simply the region of the world the user wants to zoom to. This was handled with the simple switch based on the URI requested before. To this, we’re adding an argument, separated from the zoom URI by a “?”, which will be the URI we’d like to filter on.  So before to see all requests for the US you would have requested    Now, to see all requests to “/myURL” from europe, you’d request  Simple, right? So how does this work in the code.

Well, first of all we need to actually get those parameters from the URI and split them apart. That part’s pretty simple:

   1: set zoom ""
   2: set zoomURL ""  
   3: set zoom [getfield [string map {"heatmap" "world"} [HTTP::uri]] "?" 1]
   4: set zoomURL [getfield [string map {"heatmap" "world"} [HTTP::uri]] "?" 2]

So we now have our zoom and zoomURL. We now have to implement those into the logic somewhere. This is where that nice, little foreach you saw above gets hairy. We have to take into account US vs all other regions so we know whether to search tables with either the state: or country:  prefix. We also have to ensure that we’re only appending info to the chld and chd that is pertinent to the our zoomURL filter.  This ends up looking like:

   1: foreach mysub [table keys -subtable mytables] {
   2:   if {$zoom eq "usa"} {
   3:     if {$mysub starts_with "state:"} { 
   4:       foreach myurl [table keys -subtable $mysub] {
   5:         if {$zoomURL ne ""} { 
   6:           if {$myurl eq $zoomURL} {  
   7:             append chld "[getfield $mysub ":" 2]"
   8:             append chd "[table lookup -subtable $mysub $myurl],"
   9:           }
  10:         } else {
  11:           append chld "[getfield $mysub ":" 2]"
  12:           append chd "[table lookup -subtable $mysub $myurl],"  
  13:         }
  14:       }  
  15:     } 
  16:   } else {
  17:     if {$mysub starts_with "country:"} { 
  18:       foreach myurl [table keys -subtable $mysub] {
  19:         if {$zoomURL ne ""} { 
  20:           if {$myurl eq $zoomURL} {  
  21:             append chld "[getfield $mysub ":" 2]"
  22:             append chd "[table lookup -subtable $mysub $myurl],"
  23:           }
  24:         } else {
  25:           append chld "[getfield $mysub ":" 2]"
  26:           append chd "[table lookup -subtable $mysub $myurl],"  
  27:         }
  28:       }  
  29:     } 
  30:   }
  31: }

That’s not really as scary as it looks, it’s just the same simple nested foreach from above with the zoom and zoomURL conditionals in it which means we’re double down with the same logic, once for states and once for countries.

Next we need to be able to add these URLs that we want to filter on to the HTML being sent to the client for actual filtering by clicking the appropriate link.  I first hard coded these in but that’s amazingly anti-scaling, so I went with a class approach instead.  This way, anyone can easily add a URL to the trackingurls class and it will automagically show up in the HTML the next time a user requests one of the heatmap pages. This just required one more loop and a little HTML trickery using the zoom variable from above:


   1: set filters ""  
   2: foreach mytrackingurl [class names trackingurls] {
   3:   append filters "<a href='${zoom}?${mytrackingurl}'>${mytrackingurl}</a> | "
   4: }  
   5: set filters [string trimright $filters " | "]   
   7: Combine the above generated HTML with the static HTML in RULE INIT and respond to the client
   8: HTTP::respond 200 content "${static::resp1}${zoom}&chd=t:${chd}&chld=${chld}${static::resp2} \ 
   9:  Filter by URL: <a href='/$zoom'>All URLs</a> | $filters \ 
  10:  $static::resp3"
  11: }

Last but not least, cleanup. We have to alter the resetmap “command” to clear all of our tables, which requires yet more nested looping, but at least they’re simple this time:


   1: foreach pointertable [table keys -subtable mytables] {
   2:   foreach entry [table keys -subtable $pointertable] {
   3:     table delete -subtable $pointertable $entry
   4:   }  
   5: } 
   6: foreach pointerentry [table keys -subtable mytables] {
   7:   table delete -subtable mytables $pointerentry
   8: } 
   9: HTTP::respond 200 Content "<HTML><center><br><br><br><br><br><br>Table Cleared.\
  10: <br><br><br> <a href='/heatmap'>Return to Map</a></HTML>"


And with that, you now have a solution built 100% in iRules that puts out pretty pictures and even allows you to filter on URLs. Example screenshots below:








And of course the code in it’s entirety (note that a few lines were chopped down for viewing with the use of \, this is untested, undo if necessary when implementing):

   1: when RULE_INIT {
   2: ##  Configure static portions of the HTML response for the heatmap pages
   3:   set static::resp1 "<HTML><center><font size=5>Here is your site's usage by Country:</font><br><br><br>\
   4: <img src='"
   5:   set static::resp2 "&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'><br><br>\
   6: Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> \
   7: | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | \
   8: <a href='/usa'>United States</a> | <a href='/heatmap'>World</a><br><br>"
   9:   set static::resp3 "<br><br><br><a href='/resetmap'>Reset All Counters</a></center></HTML>"
  10: }
  12: when HTTP_REQUEST timing on {
  13:   switch -glob [string tolower [HTTP::uri]] {  
  14:     "/asia*" - 
  15:     "/africa*" - 
  16:     "/europe*" - 
  17:     "/middle_east*" - 
  18:     "/south_america*" - 
  19:     "/usa*" -
  20:     "/world*" - 
  21:     "/heatmap*" {
  22:       set chld ""  
  23:       set chd ""
  24:       set zoom ""
  25:       set zoomURL ""  
  26: ##  Split apart the zoom region from the filter URL in the request  
  27:       set zoom [getfield [string map {"heatmap" "world"} [HTTP::uri]] "?" 1]
  28:       set zoomURL [getfield [string map {"heatmap" "world"} [HTTP::uri]] "?" 2]
  29: ##  Get a list of all states or countries, applying the URL filter where necessary
  30: ##  and retrieve the associated count of requests from that area to that URL
  31:   ##  First step through the mytables table, which is a pointer table referencing all subtables with counter values in them
  32:       foreach mysub [table keys -subtable mytables] {
  33:   ##  Next determine whether to search state or country tables
  34:         if {$zoom eq "usa"} {
  35:           if {$mysub starts_with "state:"} { 
  36:   ##  For each state sub table step through each key, which will be a URL, and count the request to that URL.
  37:   ##  This is also where URL filtering is applied if applicable
  38:             foreach myurl [table keys -subtable $mysub] {
  39:               if {$zoomURL ne ""} { 
  40:                 if {$myurl eq $zoomURL} {  
  41:                   append chld "[getfield $mysub ":" 2]"
  42:                   append chd "[table lookup -subtable $mysub $myurl],"
  43:                 }
  44:               } else {
  45:                 append chld "[getfield $mysub ":" 2]"
  46:                 append chd "[table lookup -subtable $mysub $myurl],"  
  47:               }
  48:             }  
  49:           } 
  50:         } else {
  51:           if {$mysub starts_with "country:"} { 
  52:             foreach myurl [table keys -subtable $mysub] {
  53:               if {$zoomURL ne ""} { 
  54:                 if {$myurl eq $zoomURL} {  
  55:                   append chld "[getfield $mysub ":" 2]"
  56:                   append chd "[table lookup -subtable $mysub $myurl],"
  57:                 }
  58:               } else {
  59:                 append chld "[getfield $mysub ":" 2]"
  60:                 append chd "[table lookup -subtable $mysub $myurl],"  
  61:               }
  62:             }  
  63:           } 
  64:         }
  65:       }
  67: ##  Send back the pre-formatted response, set in RULE_INIT, combined with the map zoom, list of areas, and request count
  68:       set chd [string trimright $chd ","] 
  69:   ##  First loop through the trackingurls class to get a list of all URLs to be tracked and format HTML around them for links
  70:       set filters ""  
  71:       foreach mytrackingurl [class names trackingurls] {
  72:         append filters "<a href='${zoom}?${mytrackingurl}'>${mytrackingurl}</a> | "
  73:       }  
  74:       set filters [string trimright $filters " | "]   
  76:   ##  Combine the above generated HTML with the static HTML in RULE INIT and respond to the client
  77:       HTTP::respond 200 content "${static::resp1}${zoom}&chd=t:${chd}&chld=${chld}${static::resp2} \ 
  78:        Filter by URL: <a href='/$zoom'>All URLs</a> | $filters \ 
  79:        $static::resp3"
  80:     }
  82:     "/resetmap" {
  83: ##  Reset all table counters to zero        
  84:       foreach pointertable [table keys -subtable mytables] {
  85:         foreach entry [table keys -subtable $pointertable] {
  86:           table delete -subtable $pointertable $entry
  87:         }  
  88:       } 
  89:       foreach pointerentry [table keys -subtable mytables] {
  90:         table delete -subtable mytables $pointerentry
  91:       } 
  92:       HTTP::respond 200 Content "<HTML><center><br><br><br><br><br><br>\
  93: Table Cleared.<br><br><br> <a href='/heatmap'>Return to Map</a></HTML>"
  94:     }
  96:     default {
  97: ##  Look up country & state locations 
  98:       set cloc [whereis [IP::client_addr] country]
  99:       set sloc [whereis [IP::client_addr] abbrev]
 101: ##  If the IP doesn't resolve to anything, pick a random IP (useful for testing on private networks)
 102:       if {($cloc eq "") and ($sloc eq "")} { 
 103:         set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
 104:         set cloc [whereis $ip country]
 105:         set sloc [whereis $ip abbrev]
 106:         if {($cloc eq "") or ($sloc eq "")} {  
 107:             set cloc "US"
 108:           set sloc "WA"
 109:         }
 110:       }
 112: ##  Create a new table named country:location or state:location
 113:       if {[table incr -subtable country:$cloc -mustexist [HTTP::uri]] eq ""} {
 114:         table set -subtable country:$cloc [HTTP::uri] 1 indefinite indefinite
 115:       }  
 116: ##  Update the mytables pointer table with the new country or state table name  
 117:       if {[table incr -subtable mytables -mustexist country:$cloc] eq ""} {
 118:         table set -subtable mytables country:$cloc 1 indefinite indefinite
 119:       }  
 120: ##  Same as above for states, not countries.
 121:       if {$cloc eq "US"} {  
 122:         if {[table incr -subtable state:$sloc -mustexist [HTTP::uri]] eq ""} {
 123:             table set -subtable state:$sloc [HTTP::uri] 1 indefinite indefinite
 124:         }  
 125:         if {[table incr -subtable mytables -mustexist state:$sloc] eq ""} {
 126:             table set -subtable mytables state:$sloc 1 indefinite indefinite
 127:         }    
 128:       }    
 129:       HTTP::respond 200 Content "Added - Country: $cloc State: $sloc"
 130:     }
 131:   }
 132: }



What’s to come in the next and probably last installment of the Heatmaps, iRules Style series? Well, pretty pictures are great and all, but what about the actual numbers? Next time I’ll make things more useful in the real world by giving you the pretty pictures along with the cold, hard data you so crave.




Related Articles