Question on syntax of iRule

I think the problem is you've enclosed the expression operators with brackets () rather than braces {}, this should work:

when HTTP_REQUEST {
    if { 
            {
                [HTTP::header value "User-Agent"] contains "Mozilla"
                || [HTTP::header value "User-Agent"] contains "Opera"
            }
        && [string tolower [HTTP::uri]] matches_regex {restservicestest}
        && { not [HTTP::cookie names] contains ".test" }
    } then { 
        reject
        log local0. "Client browser trying to connect to REST Host:[HTTP::host]; URI = [HTTP::uri] No SSO Cookie Detected, Client IP:[IP::client_addr] has been blocked" 
    }
}