Creating a Secure AWS S3 Proxy with F5 iRulesLX

AWS S3 provides a secure method of storing files, but can pose a challenge of how to access S3 from a restricted environment that does not allow direct access to the Public Internet. The following article is a variation of “How do I access VPC endpoints from outside my VPC” to create a secure S3 proxy solution. The solution provides a method of converting unauthenticated requests in a restricted environment to authenticated requests to a S3 bucket.

Secure Access to AWS S3

AWS S3 provides “Buckets” that can be used to store files. It is possible to have the files stored publicly, but that has caused accidental exposure to personal data. Using Signed requests one can ensure that files can only be accessed by authorized users. This poses a challenge of how to provide Intranet type of access to files. The goal is to provide access to users in non-routable address space (RFC 1918) and not require that each end-user send requests that look something like this:

 GET /secure/employee-benefits.pdf?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Expires=1141889120&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D

Ideally there should be a process of transparently converting an end-user request

From:

 https://s3-proxy/secure/employee-benefits.pdf

To

 https://secure-bucket.s3.amazonaws.com/secure/employee-benefits.pdf?AWSAccessKeyId=XXXXX&Expires=1234&Signature=YYYY

Creating Signed Requests with iRulesLX and AWS-SDK

Using the AWS JavaScript SDK and iRulesLX we can create the conversion process.

First we need to extract the requested path that we want to sign. From a TCL iRule we can retrieve the information and send it to an iRules LX process.

...
when HTTP_REQUEST {
    set orig_path [string trimleft [HTTP::path] "/"]

    set RPC_HANDLE [ILX::init aws_s3_rpc_plugin aws_s3_rpc_ext]
    set rpc_response [ILX::call $RPC_HANDLE aws_s3_rpc_add_creds $orig_path]
...

Then we need to generate the Signed request using the JavaScript SDK in iRulesLX.

...
ilx.addMethod('aws_s3_rpc_add_creds', function(req, res) {
  var path = req.params()[0];
  var params = {Bucket:"secure-bucket",
                   Key: path
                };
  var signed_url = s3.getSignedUrl('getObject',params);
...

Once we get the results back we can modify the request to change the hostname from “s3-proxy” to “s3.amazonaws.com”, add the name of the bucket, and add the secure signature. This is done in a TCL iRule.

...
when HTTP_REQUEST_SEND {
    clientside {

        HTTP::header replace Host "secure-bucket.s3.amazonaws.com"
        # specify bucket / signature
        HTTP::uri "/$orig_path?$qs"
    }
}
...

The complete code can be found here: iRule code, iRule LX code.

A few more details

In addition to transparently signing the requests it is also possible to cache the signature/files and use OneConnect to improve the performance and reduce the cost by fetching fewer resources from S3.

This meets the original requirements of creating a secure proxy as well as provide additional performance benefits.

The solution can speed-up requests by roughly 4x under the right conditions. Testing the solution from AWS us-west-1 to an S3 bucket in us-east-1 reduced the request time from ~350 ms to ~80 ms. Testing from the same region (us-east-1 to us-east-1) was ~110 ms to ~90 ms (these were taken from only 3 samples, very rough estimates).