Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Clear all filters
Answers

HTTP::respond and header request modifications

Hey everyone.

Trying to federate AWS S3 and an on premises S3 compliant storage box with iRules/iRulesLX. The calling client connecting to the BigIP will by default have the access key and secret key of the on premises S3 storage, but using the javascript aws-sdk with iRulesLX and determining that the object is in AWS S3 (HEAD request) I generate a new signature for AWS S3 and respond to iRules with that information.

I use the signature with an HTTP::respond as such: HTTP::respond 302 noserver Location "$host" Authorization "$authorization" X-Amz-Date "$xamzdate" X-Amz-Content-Sha256 "$xamzcontentsha256"

Problem I am seeing is that it does not appear that the HTTP::respond action with the headers is actually modifying the headers (Authorization, X-Amz-Date etc.) request during the 302. I get back a HTTP 403 with InvalidAccessKeyId: The Access Key ID you provided does not exist in our records.

Looking at the client response, I see the raw request still has the Authentication string for the on premises S3 storage.

Am I going about this wrong? Still new to iRules and the F5. Thanks for the help!

0
Rate this Question
Comments on this Question
Comment made 30-May-2018 by S Blakely

Without seeing the (sanitized) irule it is hard to say what might be going on.

Add some logging statements so you can track the path through the irule, to validate any decision logic.

0
Comment made 30-May-2018 by mblachford 22

Good points on seeing the sanitized iRule. I don't think there is anything proprietary. Here it is (the relevant bit is in the GET case under when HTTP_REQUEST. iRulesLX returns an array of data that we want to HTTP::respond with)

when RULE_INIT {
    set static::debug 0
}

proc logger { mthd httpc msg } {
    switch -exact -- $mthd {
        request {
            if { $static::debug != 0 } {
                log -noname local0.debug "=START HTTP REQUEST=============================="
                log -noname local0.debug "Client [IP::client_addr]:[TCP::client_port] -> [HTTP::method] [HTTP::host] [HTTP::uri]"
                foreach reqHeader [HTTP::header names] {
                    log -noname local0.debug "$reqHeader: [HTTP::header value $reqHeader]"
                }
                log -noname local0.debug "=END HTTP REQUEST==============================="
            } else {
                log -noname local0.info "Client [IP::client_addr] -> [HTTP::method] [HTTP::host] [HTTP::uri] (HTTP REQUEST)"
                log -noname local0.info $msg
            }
        }
        response {
            if { $static::debug != 0 } {
                log -noname local0.debug "=START HTTP RESPONSE=============================="
                log -noname local0.debug "HTTP RESPONSE - status: $httpc"
                foreach resHeader $msg {
                    set key [ getfield $resHeader "|" 1]
                    set value [ getfield $resHeader "|" 2]
                    log -noname local0.debug "$key: $value"
                }
                log -noname local0.debug "=END HTTP RESPONSE=============================="
            } else {
                log -noname local0.info "HTTP RESPONSE - status: $httpc"
            }
        }
        default {
            log -noname local0.info "$msg"
        }
    }
}

when HTTP_REQUEST {
    set rpc_handle [ILX::init syncp-plugin syncp-extension]

    switch [HTTP::method] {
        HEAD {
            if {[ catch {ILX::call $rpc_handle -timeout 12000 "syncp_http_head_req" [HTTP::method] [HTTP::uri] } result ]} {
                call logger request null "ILX syncp_http_head_req timeout. Reason: $result"
            }

            call logger request null null

            set httpresponse [ lindex $result 0 ]
            set httpheaders [ lindex $result 1 ]
            set httpcontent [ lindex $result 2 ]

            if { [string length $httpcontent] == 0 } {
                set stringtoeval "HTTP::respond $httpresponse -version auto noserver"
            } else {
                set stringtoeval "HTTP::respond $httpresponse -version auto content {$httpcontent} noserver"
            }

            foreach hdrs $httpheaders {
                set key [ getfield $hdrs "|" 1]
                set value [ getfield $hdrs "|" 2]
                append stringtoeval " {$key} {$value} "
            }

            call logger response $httpresponse $httpheaders
            eval $stringtoeval
        }
        GET {
            if {[ catch {ILX::call $rpc_handle -timeout 12000 "syncp_http_get_req" [HTTP::method] [HTTP::uri] } result ]} {
                call logger request null "ILX syncp_http_get_req timeout. Reason: $result"
            }
            call logger request null null

            set host [ lindex $result 0 ]
            set xamzcontentsha256 [ lindex $result 1 ]
            set xamzdate [ lindex $result 2 ]
            set authorization [ lindex $result 3 ]

          HTTP::respond 302 noserver Location "$host" Authorization "$authorization" X-Amz-Date "$xamzdate" X-Amz-Content-Sha256 "$xamzcontentsha256"
        }
        PUT {
            if {[ catch {ILX::call $rpc_handle -timeout 12000 "syncp_http_put_req" [HTTP::method] [HTTP::uri] } result ]} {
                call logger request null "ILX syncp_http_put_req timeout. Reason: $result"
            }

            call logger request null null
        }
        DELETE {
            if {[ catch {ILX::call $rpc_handle -timeout 12000 "syncp_http_delete_req" [HTTP::method] [HTTP::uri] } result ]} {
                call logger request "ILX syncp_http_delete_req timeout. Reason: $result"
            }

            call logger request null null
        }
    }
}
0
Comment made 30-May-2018 by S Blakely

Well, first log $host and the other variables to ensure that you are getting the appropriate values out of the irulesLX -
$host is mapped to the Location header, and should be the full request for the federated server and the complete URL protocol://hostname/url

Use the Browser Network tools to track the REQUEST/RESPONSE path.

Make sure no other attached irule is generating a HTTP::redirect/response

0
Comment made 30-May-2018 by mblachford 22

Little about the environment. Its a VE setup on my local machine in ESX for development purposes. The initial request to the BigIP with the creds for the local S3 storage is as follows (just a sample from the logs today):

May 30 03:17:31 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: GET /testbucket1/shot.png HTTP/1.1 User-Agent: aws-sdk-nodejs/2.229.1 darwin/v8.7.0 callback X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c89AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPP Content-Length: 0 Host: 10.128.10.240 X-Amz-Date: 20180530T031731Z Authorization: AWS4-HMAC-SHA256 Credential=1316923@ecstd.emc.local/20180530/us-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3cd6b5f4159cff60c601c2491722096d68a7aec5390bf663291bb0997fc6a1dc Connection: close

Return values printed as requested via iRules. I am returning these in a list from iRulesLX:

set host [ lindex $result 0 ]
set xamzcontentsha256 [ lindex $result 1 ]
set xamzdate [ lindex $result 2 ]
set authorization [ lindex $result 3 ]

log local0. "HOST: $host"
log local0. "X-Amz-Content-Sha256: $xamzcontentsha256"
log local0. "X-Amz-Date: $xamzdate"
log local0. "Authorization: $authorization"

May 31 00:14:21 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: HOST: s3.amazonaws.com
May 31 00:14:21 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
May 31 00:14:21 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: X-Amz-Date: 20180531T001421Z
May 31 00:14:21 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: Authorization: AWS4-HMAC-SHA256 Credential=n9pzu5kt6nggeneric/20180531/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3a0ad3e459c338b362a5e7dec941319f9d207d725b7fbcedcdf88efad059b9ef
0
Comment made 30-May-2018 by mblachford 22

Oh, and by the way the client side here is nodejs with the AWS SDK for testing purposes from the console on my mac.

0

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

So the 302 response from your irule has an incorrectly formatted Location header with no protocol or URI, as I mentioned earlier.

Your new location header should be

https://s3.amazonaws.com/testbucket1/shot.png

HTTP::respond 302 noserver Location "https://$host[HTTP::uri]" Authorization "$authorization" X-Amz-Date "$xamzdate" X-Amz-Content-Sha256 "$xamzcontentsha256"

0
Comments on this Answer
Comment made 30-May-2018 by mblachford 22

Thanks for the help, and I did try modifying the location header as you suggested with no success either. I am wondering if the original request's Authorization header is not being overwritten.

ltm log entry showing $host as you stated it should be plus the newly signed authorization header:

May 31 01:55:02 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: HOST: http://s3.amazonaws.com/testbucket1/shot.png
May 31 01:55:02 bigip01 info tmm[1929]: Rule /Common/syncp-plugin/syncp-irule <HTTP_REQUEST>: Authorization: AWS4-HMAC-SHA256 Credential=nn9pzu5kt6nggeneric/20180531/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=89c958cce3e747f47b4b60efb9bee4c01021ce80db3d67c6708b73fa9b88f767
`</pre>

Original client side HTTP Request to the BigIP. Notice the Authorization string:

<pre>`HttpRequest {
      method: 'GET',
      path: '/testbucket1/shot.png',
      headers: 
       { 'User-Agent': 'aws-sdk-nodejs/2.229.1 darwin/v8.7.0 callback',
         'X-Amz-Content-Sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
         'Content-Length': 0,
         Host: '10.128.10.240',
         'X-Amz-Date': '20180531T015136Z',
         Authorization: 'AWS4-HMAC-SHA256 Credential=1316923@ecstd.emc.local/20180531/us-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f9a0f39b895af77cdfc12cc7dad590c0487b4da9512a68ceec2cdae16eb1e615' },

Client side HTTP Response after the 302 redirect (from the BigIP):

header: 'GET /testbucket1/shot.png HTTP/1.1\r\nUser-Agent: aws-sdk-nodejs/2.229.1 darwin/v8.7.0 callback\r\nX-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nContent-Length: 0\r\nHost: s3.amazonaws.com \r\nX-Amz-Date: 20180531T020713Z\r\nAuthorization: AWS4-HMAC-SHA256 Credential=1316923@ecstd.emc.local/20180531/us-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f9a0f39b895af77cdfc12cc7dad590c0487b4da9512a68ceec2cdae16eb1e615'\r\nConnection: close\r\n\r\n'

So it looks like some of the changes took place on the redirect (host etc.) but not others (Authorization) Wondering if this is a function of the HTTP 1.1 spec, or if its syntax with how im returning it from the BigIP....

0
Comment made 30-May-2018 by S Blakely

The 302 response will have the correct information, but the client won't use the newly supplied Authorization header for the next request - that comes from within the client itself, so it keeps using the one it has.

What you probably need to do is get the new Authorization header, modify the existing request, and then pass that request directly from the LTM to the AWS server (using the pool or node command), and then modify and pass the response back to the client.

0
Comment made 31-May-2018 by mblachford 22

That just might work. Right now, I have the AWS node definition and the onsite node definition in the same pool called by FQDN so the responses are ephemeral and tagged as _auto.

Would you recommend that I split this up into two distinct pools and just target the pool name vs. having to figure out the pool members?

0
Comment made 31-May-2018 by S Blakely

Would you recommend that I split this up into two distinct pools and just target the pool name vs. having to figure out the pool members?

Yes

0