iRule　- escape period in Double Quotes,Braces

Try this:

This evaluates to true  
when RULE_INIT {  
    if { "aa.html" matches_regex "aa\\.html" } { log local0. "true" } else { log local0. "false" }  
}

This evaluates to false:
when RULE_INIT {
    if { "aahtml" matches_regex "aa\\.html" } { log local0. "true" } else { log local0. "false" }
} 

This also evaluates to false:
when RULE_INIT {
if { "aa-html" matches_regex "aa\\.html" } { log local0. "true" } else { log local0. "false" }
}