David_Remington
Mar 13, 2008Employee
dilemma: run slow or crash on load
So I have a rule I am writing that has to parse a very large (6000+) entry class file for every URI that traverses the virtual server. Because of the constraints on possibilities for a match I basically have to loop through the class and do a string match (not regexp). So the rule is dog slow... I mean, really really slow. I mean, a single iteration uses about 3% of the CPU giving me a scorching 40 requests per second, give or take. So it is non-viable.
So I decided to take the class file and parse it into an array so that I could just loop through a small portion of the overall class file based on the first couple characters in the requested uri. During runtime this makes a huge difference. Instead of 3% the rule drops to less than .05% which is well within our performance requirements.
Unforutnately, the RULE_INIT event does so much work building the array that tmm's heartbeat times out when you do a 'b load' and the box fails over. It also seems to suspend all network activity during the load, which causes monitors to time out, etc.
After the tmm restart, the rule loads fine and runs great.
Any ideas how I can do the following faster or do something to give tmm enough breathing room to not lock up? Is there a way to break up the work so that tmm can increment it's heartbeat or does the entire event have to be parsed before tmm can do some other task?
when RULE_INIT {
set debug 1
if { $debug > 0 } {log local0.notice "rule initializing"}
set list_map [lsort -unique -decreasing -ascii $::cl_map_v10]
array set ::map { }
array set ::keylist { }
if { $debug > 0 } {log local0.notice "starting to load array"}
set idxelement { a b c d e f g h i j k l m n o p q r s t u v w x y z 0 1 2 3 4 5 6 7 8 9 }
foreach idx $idxelement {
if { $debug > 1 } { log local0.notice "idx: $idx" }
foreach idx2 $idxelement {
if { $debug > 1 } { log local0.notice " idx2: $idx2" }
set n 0
foreach element $list_map {
if { $element starts_with "/${idx}${idx2}" } {
incr n
set ::map(${idx}${idx2},${n}) $element
}
}
set ::listsize(${idx}${idx2}) $n
}
}
if { $debug > 0 } {log local0.notice "map(ab,1): $::map(ab,1)"}
if { $debug > 0 } {log local0.notice "done load array"}
if { $debug > 0 } {log local0.notice "rule initialized"}
}
One suggestion I have thought of/been suggested is to make a bunch of class files instead of using the array, but unfortunately the above rule creates what would basically be 36x36 class files. Even if I pared that down a bit I'd at least need several hundred class files which will be a huge pain for the customer to manage.
Thoughts? Suggestions? Condolences?
(ps - if there is tcl magic to do the "set idxelement" line without typing the alphanumerics I'd love to hear it although I think it is not directly related to my core problem.. all I could find were extended tcl commands that don't seem to be supported in irules.)
Thanks