Leslie_South_55
Nimbostratus
NimbostratusCollecting website statistics
Greetings DevCentral,
So the challange was to be able to gather website statistics (client IP, pages hit, spiders, crawlers, search text, etc, etc, etc). There is seemingly useful tool called AWStats, available on sourceforge. Very nice interface.
AWStats builds it's stats page using web server logs recording all the METHODS and REPONSES. So now the challange; we have 8 web servers with log files wraping every hour, PLUS we have some iRules in front of this pool to either redirect or block certain request types, so the web servers don't see all the traffic.
Enter another iRule...and here is the code (partially built from several examples found here)
when HTTP_REQUEST {
set curtime [clock seconds]
set formattedtime [clock format $curtime -format {%d/%b/%Y:%T %z}]
set referer [HTTP::header value Referer]
set useragent [HTTP::header value User-Agent]
set log_format "[IP::client_addr] - - \[$formattedtime\] \"[HTTP::method] [HTTP::uri] HTTP/[HTTP::version]\" \"$referer\" \"$useragent\""
}
when HTTP_RESPONSE {
log local1.info "$log_format [HTTP::status] [HTTP::payload length]"
}
notice that I am using local1 which I defined in my syslog-ng.conf to send to a remote_server, as not to log to local disk on the bigip.
Testing was great, when fine BUT when I put this in production...LOOK OUT..my 6400 went bonkers..alertd and syslog-ng daemons consumed 99.5% of the 2nd CPU and the UNIX remote syslog IO was equally utilized.
In 5 min, we accumilated a 17MB log file with over 53,000 lines (each line representing one HTTP_RESPONSE)needless to say I had to remove the rule.
The VS that we are trying to gather stats on maintains an average 1000 HTTP connections.
I am looking for any advice or words of wisdom that could help me complete my challange.
Thanks
-L
