AS3 Best Practice
Introduction AS3 is a declarative API that uses JSON key-value pairs to describe a BIG-IP configuration. From virtual IP to virtual server, to the members, pools, and nodes required, AS3 provides a simple, readable format in which to describe a configuration. Once you've got the configuration, all that's needed is to POST it to the BIG-IP, where the AS3 extension will happily accept it and execute the commands necessary to turn it into a fully functional, deployed BIG-IP configuration. If you are new to AS3, start reading the following references: Products - Automation and orchestration toolchain(f5.com; Product information) Application Services 3 Extension Documentation(clouddocs; API documentation and guides) F5 Application Services 3 Extension(AS3) (GitHub; Source repository) This article describes some considerations in order to efficiently deploy the AS3 configurations. Architecture In the TMOS space, the services that AS3 provides are processed by a daemon named 'restnoded'. It relies on the existing BIG-IP framework for deploying declarations. The framework consists of httpd, restjavad and icrd_child as depicted below (the numbers in parenthesis are listening TCP port numbers). These processes are also used by other services. For example, restjavad is a gateway for all the iControl REST requests, and is used by a number of services on BIG-IP and BIG-IQ. When an interaction between any of the processes fails, AS3 operation fails. The failures stem from lack of resources, timeouts, data exceeding predefined thresholds, resource contention among the services, and more. In order to complete AS3 operations successfully, it is advised to follow the Best Practice outlined below. Best Practice Your single source of truth is your declaration Refrain from overwriting the AS3-deployed BIG-IP configurations by the other means such as TMSH, GUI or iControl REST calls. Since you started to use the AS3 declarative model, the source of truth for your device's configurations is in your declaration, not the BIG-IP configuration files. Although AS3 tries to weigh BIG-IP locally stored configurations as much as it can do, discrepancy between the declaration and the current configuration on BIG-IP may cause the AS3 to perform less efficiently or error unexpectedly. When you wish to change a section of a tenant (e.g., pool name change), modify the declaration and submit it. Keep the number of applications in one tenant to a minimum AS3 processes each tenant separately.Having too many applications (virtual servers) in a single tenant (partition) results in a lengthy poll when determining the current configuration. In extreme cases (thousands of virtuals), the action may time out. When you want to deploy a thousand or more applications on a single device, consider chunking the work for AS3 by spreading the applications across multiple tenants (say, 100 applications per tenant). AS3 tenant access behavior behaves as BIG-IP partition behavior.A non-Common partition virtual cannot gain access to another partition's pool, and in the same way, an AS3 application does not have access to a pool or profile in another tenant.In order to share configuration across tenants, AS3 allows configuration of the "Shared" application within the "Common" tenant.AS3 avoids race conditions while configuring /Common/Shared by processing additions first and deletions last, as shown below.This dual process may cause some additional delay in declaration handling. Overwrite rather than patching (POSTing is a more efficient practice than PATCHing) AS3 is a stateless machine and is idempotent. It polls BIG-IP for its full configuration, performs a current-vs-desired state comparison, and generates an optimal set of REST calls to fill the differences.When the initial state of BIG-IP is blank, the poll time is negligible.This is why initial configuration with AS3 is often quicker than subsequent changes, especially when the tenant contains a large number of applications. AS3 provides the means to partially modify using PATCH (seeAS3 API Methods Details), but do not expect PATCH changes to be performant.AS3 processes each PATCH by (1) performing a GET to obtain the last declaration, (2) patching that declaration, and (3) POSTing the entire declaration to itself.A PATCH of one pool member is therefore slower than a POST of your entire tenant configuration.If you decide to use PATCH,make sure that the tenant configuration is a manageable size. Note: Using PATCH to make a surgical change is convenient, but using PATCH over POST breaks the declarative model. Your declaration should be your single source of truth.If you include PATCH, the source of truth becomes "POST this file, then apply one or more PATCH declarations." Get the latest version AS3 is evolving rapidly with new features that customers have been wishing for along with fixes for known issues. Visitthe AS3 section of the F5 Networks Github.Issuessection shows what features and fixes have been incorporated. For BIG-IQ, check K54909607: BIG-IQ Centralized Management compatibility with F5 Application Services 3 Extension and F5 Declarative Onboarding for compatibilities with BIG-IQ versions before installation. Use administrator Use a user with the administrator role when you submit your declaration to a target BIG-IP device. Your may find your role insufficient to manipulate BIG-IP objects that are included in your declaration. Even one authorized item will cause the entire operation to fail and role back. See the following articles for more on BIG-IP user and role. Manual Chapter : User Roles (12.x) Manual Chapter : User Roles (13.x) Manual Chapter : User Roles (14.x) Prerequisites and Requirements(clouddocs AS3 document) Use Basic Authentication for a large declaration You can choose either Basic Authentication (HTTP Authorization header) or Token-Based Authentication (F5 proprietary X-F5-Auth-Token) for accessing BIG-IP. While the Basic Authentication can be used any time, a token obtained for the Token-Based Authentication expires after 1,200 seconds (20 minutes). While AS3 does re-request a new token upon expiry, it requires time to perform the operation, which may cause AS3 to slow down. Also, the number of tokens for a user is limited to 100 (since 13.1), hence if you happen to have other iControl REST players (such as BIG-IQ or your custom iControl REST scripts) using the Token-Based Authentication for the same user, AS3 may not be able to obtain the next token, and your request will fail. See the following articles for more on the Token-Based Authentication. Demystifying iControl REST Part 6: Token-Based Authentication(DevCentral article). iControl REST Authentication Token Management(DevCentral article) Authentication and Authorization(clouddocs AS3 document) Choose the best window for deployment AS3 (restnoded daemon) is a Control Plane process. It competes against other Control Plane processes such as monpd and iRules LX (node.js) for CPU/memory resources. AS3 uses the iControl REST framework for manipulating the BIG-IP resources. This implies that its operation is impacted by any processes that use httpd (e.g., GUI), restjavad, icrd_child and mcpd. If you have resource-hungry processes that run periodically (e.g., avrd), you may want to run your AS3 declaration during some other time window. See the following K articles for alist of processes K89999342 BIG-IP Daemons (12.x) K05645522BIG-IP Daemons (v13.x) K67197865BIG-IP Daemons (v14.x) K14020: BIG-IP ASM daemons (11.x - 15.x) K14462: Overview of BIG-IP AAM daemons (11.x - 15.x) Workarounds If you experience issues such as timeout on restjavad, it is possible that your AS3 operation had resource issues. After reviewing the Best Practice above but still unable to alleviate the problem, you may be able to temporarily fix it by applying the following tactics. Increase the restjavad memory allocation The memory size of restjavad can be increased by the following tmsh sys db commands tmsh modify sys db provision.extramb value <value> tmsh modify sys db restjavad.useextramb value true The provision.extramb db key changes the maximum Java heap memory to (192 + <value> * 8 / 10) MB. The default value is 0. After changing the memory size, you need to restart restjavad. tmsh restart sys service restjavad See the following article for more on the memory allocation: K26427018: Overview of Management provisioning Increase a number of icrd_child processes restjavad spawns a number of icrd_child processes depending on the load. The maximum number of icrd_child processes can be configured from /etc/icrd.conf. Please consult F5 Support for details. See the following article for more on the icrd_child process verbosity: K96840770: Configuring the log verbosity for iControl REST API related to icrd_child Decrease the verbosity levels of restjavad and icrd_child Writing log messages to the file system is not exactly free of charge. Writing unnecessarily large amount of messages to files would increase the I/O wait, hence results in slowness of processes. If you have changed the verbosity levels of restjavad and/or icrd_child, consider rolling back the default levels. See the following article for methods to change verbosity level: K15436: Configuring the verbosity for restjavad logs on the BIG-IP systemEmbracing AS3: Foundations
Many F5 teams are on the North American leg of the F5 Academy BIG-IP Next Roadshow, where we are sharing the BIG-IP story, where we're going with it, and then diving into some hands-on labs with the attendees. I got to attend in person in Kansas City, St. Louis, Cincinnati, and Columbus, and talked with customers at all stages of the automation journey. Some haven't automated much of anything. Some have been using a variety of on-/off-box scripts, and some are all-in, baby! AS3 is a critical component of BIG-IP Next, but it has also been around for several years for BIG-IP and can be learned and implemented ahead of your transition to BIG-IP Next. So over the next three weeks, I'm going to host a five-part live-stream series on AS3 Foundations. There are a lot of resources I'll link to and key in on during the event for reference and further study. Here are the shows I'm doing with descriptions: Beyond Imperatives—What the heck is AS3? (Dec 4@ 9am PST) In this first episode, I'll cover the history of automation on BIG-IP, the differences between imperative and declarative models, and the basics of AS3 from data structure and systems architecture perspectives. Top 10 Features to Know in the VSCode F5 Extension (Dec 7@ 12pm PST) Next, special guest Ben Novak, author of the F5 Extension for VSCode, will join me to dig into the features most important to know and learn to use for understanding how to take stock BIG-IP configurations and turn applications into declarations. Migrating and Deploying Applications in VSCode (Dec 14@ 12pm PST) In this episode, armed with the knowledge I learned from Ben in the last episode, I'll finally get to the brass tacks of AS3! I'll review diagnostics and migrate several applications from standard configuration to AS3 declarations and deploy a few. For migrating active workloads, I'll discuss the steps necessary to reduce transition impact. Creating New Apps and Using Shared Objects (Dec 18@ 9am PST) I've always tried to alter existing things before creating new things with tech, and this is no different. Now that I have a few migrations under my belt, I'll attack a net-new application and I'll look at shared objects and how and when to use them. Best Practices (Dec 21@ 12pm PST) Finally, I'll close this series with a look at several best practices when working with AS3. The conclusion to this series, though, is hopefully just the stepping off point for everyone new to AS3, and we'll be able to continue the conversation right here on DevCentral.Getting Started with BIG-IP Next: Migrating an Application Workload
So far in this article series, the focus has been completely on the operational readiness of BIG-IP Next as a system. In this article, I'll walk through migrating an application currently supported by my classic BIG-IP running TMOS version 15.1.x. The application is just a simple instance of an NGINX web server fronted on LTM with basic load balancing, TLS offloading, and a basic WAF policy. There are a lot of screenshots in this article, which might seem overwhelming. Doing your own walkthrough, however, will put your mind at ease; it actually moves pretty quickly in realtime. Existing Application Workload on TMOS We'll start with the GUI representation of the application workload. It is secured with TLS, which is offloaded at the BIG-IP with a clientssl profile and not re-encrypted to the server. There are custom TCP and HTTP profiles defined as well as the aforementioned custom clientssl profile. Snat automap is enabled, and a specific VLAN is configured to allow connections. On the security tab, an application security policy is enabled, and the log illegal requests log profile is enabled as well. Finally, under resources, the default pool is defined and a policy is in place to map requests to the applied security policy. On the CLI, that virtual server along with all the other referenced BIG-IP objects are defined in the tmsh version of that configuration. ltm virtual nginx-vip-tls { destination 172.16.101.50:https ip-protocol tcp mask 255.255.255.255 policies { asm_auto_l7_policy__nginx-vip-tls { } } pool nginx-pool profiles { ASM_testpol { } cssl.TestSuite { context clientside } customHTTP { } customTCP { } websecurity { } } security-log-profiles { "Log illegal requests" } source-address-translation { type automap } vlans { vlan.br1 } vlans-enabled } ltm policy asm_auto_l7_policy__nginx-vip-tls { controls { asm } last-modified 2024-03-20:13:25:13 requires { http } rules { default { actions { 1 { asm enable policy /Common/testpol } } ordinal 1 } } status legacy strategy first-match } ltm pool nginx-pool { members { 172.16.102.5:http { address 172.16.102.5 session monitor-enabled state up } } monitor http } security bot-defense asm-profile ASM_testpol { app-service none clientside-in-use disabled flags 0 inject-javascript disabled persistent-data-validity-period 0 send-brute-force-challenge disabled send-javascript-challenge disabled send-javascript-efoxy disabled send-javascript-fingerprint disabled } ltm profile client-ssl cssl.TestSuite { app-service none cert-key-chain { default { cert default.crt key default.key } } cipher-group cg_TLSv1.3 ciphers none defaults-from clientssl inherit-ca-certkeychain true inherit-certkeychain true options { dont-insert-empty-fragments } } ltm cipher group cg_TLSv1.3 { allow { cr_TLSv1.3 { } } } ltm cipher rule cr_TLSv1.3 { cipher TLSv1_3 dh-groups DEFAULT signature-algorithms DEFAULT } ltm profile http customHTTP { app-service none defaults-from http enforcement { known-methods { PATCH DELETE GET POST PUT } max-header-count 32 max-header-size 16384 rfc-compliance enabled } hsts { mode enabled } insert-xforwarded-for enabled proxy-type reverse } ltm profile tcp customTCP { app-service none congestion-control bbr defaults-from f5-tcp-progressive idle-timeout 600 ip-tos-to-client pass-through keep-alive-interval 2100 pkt-loss-ignore-burst 3 pkt-loss-ignore-rate 10 proxy-options enabled } ltm profile web-security websecurity { } You can see that I have some non-standard options in some of that configuration, such as specifying the congestion-control algorithm algorithm in the TCP profile, enabling HSTS in the HTTP profile, and setting cipher rules and groups for use in my SSL profile. Now that we have an idea of the workload we're going to migrate, let's create a UCS of the system for use in the migration. If you are already comfortable with this part on classic BIG-IP systems, you can skip down to the next section header. First, login to your classic BIG-IP and navigate to System->Archives and click Create. Give it a name and click Finished. I named mine next-migration. Click OK after the UCS has been generated and saved. In the archive list, click the name of the UCS you created. Click the Download button. Migrating the Workload in Central Manager Upload UCS and Analyze the Workloads Armed with your UCS, login to Central Manager and on the welcome screen, click Go to Application Workspace. If you have not added any applications yet, you'll see a screen like this with a Start Adding Apps button. If you already have something defined, you'll see a list of applications. Click the + Add Application button instead. On this screen, we'll bypass creating a new application service and select New Migration. Name your session as you'll be able to come back to it to migrate other applications later if your intent is to just migrate a single application for now (as is the case with this walkthrough.) I added a description but it is not necessary. Click Next. Here you'll select your UCS archive and group your application services by IP addresses OR by virtual server. I stuck with the recommended default. Click Next. Your UCS will now upload and then Central Manager will analyze and group the package. An enhanced version of the JOURNEYS tool available in the f5devcentral organization on GitHub is used here. Select Add Application. Application 5 is the one we are interested in analyzing and migrating for this walkthrough, so I selected that one. Notice in the status column the applications that have warnings, and that ours is one of them. Hovering over the triangle icon it indicates the app can be migrated, but without some of the functionality from our classic iteration of this workload. Next, click Analyze at the top right so we can see what can't be migrated. In the Configuration Analyzer screen, there are 3 files with areas of concern. First, that the websecurity profile is not supported. This is ok, the mechanisms to support attaching policies in Next are slightly different. Next from what was the bigip_base.conf file, it's not supporting the vlan as defined. This is included in the migration analysis as the vlans are specified in my virtual server, but the mechanisms for doing so are different in Next. (Note: I don't fully grok this change yet. This article will be updated once I have confidence I'm communicating the functionality accurately.) And finally, from the bigip.conf file, there are few areas of concern, shown in the animated gif below. Standalone bot-defense is not a thing in BIG-IP Next, it's part of the overall policy, so that object is not supported. Also not supported yet are local traffic policies and cipher groups. Note that even though these objects aren't supported, I can still migrate the application, and it should "just work." I guess we'll see later in this article, right? :) At this point, select the </> Preview AS3 and copy that to a file. We'll compare that to the classic BIG-IP version of AS3 in a later section. Add an Application Service After closing the AS3 preview, select the application again and click Add. Click Next For this particular application, we need a couple shared objects: the certificate/key pair for the SSL profile and the WAF policy. Click Import. After those are imported, click the numbered icon (2 in my case) under the Shared Objects column, which will open a listing of those objects that you imported. Review the objects (optional) and click Exit. At this step, if your existing application migration is accurate to the object level, you can deploy to an instance directly. But I have some changes to make to the IPs so I'm going to deploy as a draft instead. After seeing that my deployment was successfully deployed as a draft in Central Manager, I click Finish. Update the Draft and Deploy In My Application Services, click the application we just migrated. Here we can tweak the AS3 declaration. I need to update the vlan as my vlan.br1 from my TMOS BIG-IP system is not defined on my Next instance. I also have different client/server address ranges, so I updated the virtual server and pool member addresses as well. You will likely want to change your application name from the generically-migrated "application_5" but I left it as is for this exercise. Once I completed those changes, I clicked Save & Deploy. I was then asked to select an instance to deploy the application server. I only have one currently, so I selected that. This failed due to my vlan configuration. As I mentioned during the migration process, I don't yet fully grok the vlan referencing requirements in Next, so this is a point for me to be educated on and follow up with updates here in this article. Instead, I removed the allowVlans attributed altogether (after another attempt) and then clicked Save & Deploy again and (after re-selecting the deploy location as shown above) found success. Clicking on the application, you get a visual representation of the application objects. Testing and Observing the Migrated Application Now that we have an honest to goodness deployed application on BIG-IP Next (WOO HOO!!) let's test it to make sure things are working as expected. I have a ubuntu test server with connections into my external and internal traffic networks for my Next instance so it can be the client (curl) and the server (NGINX). First, a request that should work: curl -sk https://10.0.2.50/ <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> Huzzah! That's a successful test. I ran a simple bash script with repetitive wget calls to push just a little load to populate the instance traffic graph: Now let's test the WAF policy by sending some nefarious traffic: curl -sk --config requests.txt https://10.0.2.50/ <html> <head> <title>Request Rejected</title> </head> <body>The requested URL was rejected. Please consult with your administrator.<br><br> Your support ID is: 16177875355615369771<br><br> <a href='javascript:history.back();'>[Go Back]</a> </body> </html> Sweet! Exactly what we wanted to see. Now let's take a look at the WAF Dashboard for blocks. Ok, that's a wrap on migrating the application. Functionally, it is a success! Comparing BIG-IP classic AS3 with BIG-IP Next AS3 If you are moving from classic BIG-IP configuration to BIG-IP Next, you likely will not have any context for comparing AS3 and so you might miss that some of the features you configured in classic are not present in Next. Some of those features aren't there at all yet, and some of them are just not exposed yet. Under the hood, TMM is still TMM with BIG-IP Next, and all of that core functionality is there, it's just a matter of prioritizing what gets exposed and tested and ready to support. Despite a myriad of features in classic BIG-IP, a surprising number of features went either unused or under-used and maintaining support for those will depend on future use requirements. Anyway, one way to build context for AS3 is to useVisual Studio Code and the F5 Extension to take your classic configuration and convert that to AS3 declarations with the AS3 configuration converter. In this section, I'm going to look at a few snippets to compare between classic and Next. Declaration Header The header for classic is a essentially a wrapper (lines 2-4) that isn't necessary in Next at all. That's because in classic, AS3 is not the only declaration class, you also have declarative onboarding and telemetry streaming. Classic: { "$schema": "https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/master/schema/latest/as3-schema.json", "class": "AS3", "declaration": { "class": "ADC", "schemaVersion": "3.37.0", "id": "urn:uuid:4339ea7d-094b-4950-b029-ac6344b03a2b", "label": "Converted Declaration", } } Next: { "class": "ADC", "schemaVersion": "3.0.0", "id": "urn:uuid:715aa8d8-c2b0-4890-9e77-5f6131ee9efd", "label": "Converted Declaration", } Profiles One thing to keep in mind with migration is that the migration assistant currently provides detailed analysis to the class level, not the class attribute level. This means that some of the attributes that are supported in classic that are not supported in Next will fly under the radar and be removed with no notification. There is work underway in this regard, but you'll need to evaluate each of your applications as you migrate and plan accordingly. For the app I migrated here, this was evident in the following profiles. ClientSSL Here, the cipher groups and rules from classic are not yet available, and the ability to establish only TLSv1.3 seems to not be configurable at this time. Classic: "cssl.TestSuite": { "certificates": [ { "certificate": "foo.acmelabs.com" } ], "cipherGroup": { "use": "cg_TLSv1.3" }, "class": "TLS_Server", "tls1_0Enabled": true, "tls1_1Enabled": true, "tls1_2Enabled": true, "tls1_3Enabled": true, "singleUseDhEnabled": false, "insertEmptyFragmentsEnabled": false }, Next: "cssl.TestSuite": { "authenticationFrequency": "one-time", "certificates": [ { "certificate": "/tenant87f7bd9913a51/application_5/foo.acmelabs.com" } ], "class": "TLS_Server" }, TCP In the TCP profile, the most notable changes are the loss of QoS settings and the ability to select the congestion control algorithm. Classic: "customTCP": { "congestionControl": "bbr", "idleTimeout": 600, "ipTosToClient": "pass-through", "keepAliveInterval": 2100, "pktLossIgnoreBurst": 3, "pktLossIgnoreRate": 10, "proxyOptions": true, "class": "TCP_Profile" } Next: "customTCP": { "idleTimeout": 600, "pktLossIgnoreBurst": 3, "pktLossIgnoreRate": 10, "proxyBufferHigh": 262144, "proxyBufferLow": 196608, "proxyOptions": true, "sendBufferSize": 262144, "class": "TCP_Profile" }, HTTP In my HTTP profile, it seems I lost all my personally-selected options, such that I'd likely be fine with the default profile. Also, since I'm using the WAF, I can manage the allowed request methods there, and whereas I can't auto-insert strict transport security in the profile directly yet, I can manage that in an iRule as well, so I do have a path to workarounds for both cases. Classic: "customHTTP": { "knownMethods": [ "PATCH", "DELETE", "GET", "POST", "PUT" ], "maxHeaderCount": 32, "maxHeaderSize": 16384, "hstsInsert": true, "xForwardedFor": true, "proxyType": "reverse", "class": "HTTP_Profile" }, Next: "customHTTP": { "requestChunking": "sustain", "responseChunking": "sustain", "class": "HTTP_Profile" } Final Thoughts I point out the differences in my before and after to show a complete picture of the migration process. Some things changed, some went away, but the bottom line is I have a working application service. Before working on this article, I've done a migration in a couple step-by-step controlled labs and have played with but not finished deploying a working, tested, functional application in my own lab. Don't make that same mistake. Get your classic configurations migrated ASAP even if only as a draft in Central Manager, so you can start to evaluate and analyze What work you have on your end to tweak and tune where features have changed Where you need to start engaging your account team to inquire about your MUST HAVE features that may or may not be scoped currently. Next time out, we'll take a look at creating a net-new application service. Until then, stay active out there community and start digging into BIG-IP Next!Advanced WAF v16.0 - Declarative API
Since v15.1 (in draft), F5® BIG-IP® Advanced WAF™ canimport Declarative WAF policy in JSON format. The F5® BIG-IP® Advanced Web Application Firewall (Advanced WAF) security policies can be deployed using the declarative JSON format, facilitating easy integration into a CI/CD pipeline. The declarative policies are extracted from a source control system, for example Git, and imported into the BIG-IP. Using the provided declarative policy templates, you can modify the necessary parameters, save the JSON file, and import the updated security policy into your BIG-IP devices. The declarative policy copies the content of the template and adds the adjustments and modifications on to it. The templates therefore allow you to concentrate only on the specific settings that need to be adapted for the specific application that the policy protects. ThisDeclarative WAF JSON policyis similar toNGINX App Protect policy. You can find more information on theDeclarative Policyhere : NAP :https://docs.nginx.com/nginx-app-protect/policy/ Adv. WAF :https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-declarative-security-policy.html Audience This guide is written for IT professionals who need to automate their WAF policy and are familiar with Advanced WAF configuration. These IT professionals can fill a variety of roles: SecOps deploying and maintaining WAF policy in Advanced WAF DevOps deploying applications in modern environment and willing to integrate Advanced WAF in their CI/CD pipeline F5 partners who sell technology or create implementation documentation This article covershow to PUSH/PULL a declarative WAF policy in Advanced WAF: With Postman With AS3 Table of contents Upload Policy in BIG-IP Check the import Apply the policy OpenAPI Spec File import AS3 declaration CI/CD integration Find the Policy-ID Update an existing policy Video demonstration First of all, you need aJSON WAF policy, as below : { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false } } } 1. Upload Policy in BIG-IP There are 2 options to upload a JSON file into the BIG-IP: 1.1 Either youPUSHthe file into the BIG-IP and you IMPORT IT OR 1.2 the BIG-IPPULLthe file froma repository (and the IMPORT is included)<- BEST option 1.1PUSH JSON file into the BIG-IP The call is below. As you can notice, it requires a 'Content-Range' header. And the value is 0-(filesize-1)/filesize. In the example below, the file size is 662 bytes. This is not easy to integrate in a CICD pipeline, so we created the PULL method instead of the PUSH (in v16.0) curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/file-transfer/uploads/policy-api.json' \ --header 'Content-Range: 0-661/662' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --header 'Content-Type: application/json' \ --data-binary '@/C:/Users/user/Desktop/policy-api.json' At this stage,the policy is still a filein the BIG-IP file system. We need toimportit into Adv. WAF. To do so, the next call is required. This call import the file "policy-api.json" uploaded previously. AnCREATEthe policy /Common/policy-api-arcadia curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/javascript' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "filename":"policy-api.json", "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 1.2PULL JSON file from a repository Here, theJSON file is hosted somewhere(in Gitlab or Github ...). And theBIG-IP will pull it. The call is below. As you can notice, the call refers to the remote repo and the body is a JSON payload. Just change the link value with your JSON policy URL. With one call, the policy isPULLEDandIMPORTED. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }' Asecond versionof this call exists, and refer to the fullPath of the policy.This will allow you to update the policy, from a second version of the JSON file, easily.One call for the creation and the update. As you can notice below, we add the"policy":"fullPath" directive. The value of the "fullPath" is thepartitionand thename of the policyset in the JSON policy file. This method is VERY USEFUL for CI/CD integrations. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 2. Check the IMPORT Check if the IMPORT worked. To do so, run the next call. curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ You should see a 200 OK, with the content below (truncated in this example). Please notice the"status":"COMPLETED". { "kind": "tm:asm:tasks:import-policy:import-policy-taskcollectionstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy?ver=16.0.0", "totalItems": 11, "items": [ { "isBase64": false, "executionStartTime": "2020-07-21T15:50:22Z", "status": "COMPLETED", "lastUpdateMicros": 1.595346627e+15, "getPolicyAttributesOnly": false, ... From now, your policy is imported and created in the BIG-IP. You can assign it to a VS as usual (Imperative Call or AS3 Call).But in the next session, I will show you how to create a Service with AS3 including the WAF policy. 3. APPLY the policy As you may know, a WAF policy needs to be applied after each change. This is the call. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/apply-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{"policy":{"fullPath":"/Common/policy-api-arcadia"}}' 4. OpenAPI spec file IMPORT As you know,Adv. WAF supports OpenAPI spec (2.0 and 3.0). Now, with the declarative WAF, we can import the OAS file as well. The BEST solution, is toPULL the OAS filefrom a repo. And in most of the customer' projects, it will be the case. In the example below, the OAS file is hosted in SwaggerHub(Github for Swagger files). But the file could reside in a private Gitlab repo for instance. The URL of the projectis :https://app.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 The URL of the OAS file is :https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 This swagger file (OpenAPI 3.0 Spec file) includes all the application URL and parameters. What's more, it includes the documentation (for NGINX APIm Dev Portal). Now, it ispretty easy to create a WAF JSON Policy with API Security template, referring to the OAS file. Below, you can notice thenew section "open-api-files"with the link reference to SwaggerHub. And thenew templatePOLICY_TEMPLATE_API_SECURITY. Now, when I upload / import and apply the policy, Adv. WAF will download the OAS file from SwaggerHub and create the policy based on API_Security template. { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3" } ] } } 5. AS3 declaration Now, it is time to learn how we cando all of these steps in one call with AS3(3.18 minimum). The documentation is here :https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/application-security.html?highlight=waf_policy#virtual-service-referencing-an-external-security-policy With thisAS3 declaration, we: Import the WAF policy from a external repo Import the Swagger file (if the WAF policy refers to an OAS file) from an external repo Create the service { "class": "AS3", "action": "deploy", "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.2.0", "id": "Prod_API_AS3", "API-Prod": { "class": "Tenant", "defaultRouteDomain": 0, "API": { "class": "Application", "template": "generic", "VS_API": { "class": "Service_HTTPS", "remark": "Accepts HTTPS/TLS connections on port 443", "virtualAddresses": ["10.1.10.27"], "redirect80": false, "pool": "pool_NGINX_API_AS3", "policyWAF": { "use": "Arcadia_WAF_API_policy" }, "securityLogProfiles": [{ "bigip": "/Common/Log all requests" }], "profileTCP": { "egress": "wan", "ingress": { "use": "TCP_Profile" } }, "profileHTTP": { "use": "custom_http_profile" }, "serverTLS": { "bigip": "/Common/arcadia_client_ssl" } }, "Arcadia_WAF_API_policy": { "class": "WAF_Policy", "url": "http://10.1.20.4/root/as3-waf-api/-/raw/master/policy-api.json", "ignoreChanges": true }, "pool_NGINX_API_AS3": { "class": "Pool", "monitors": ["http"], "members": [{ "servicePort": 8080, "serverAddresses": ["10.1.20.9"] }] }, "custom_http_profile": { "class": "HTTP_Profile", "xForwardedFor": true }, "TCP_Profile": { "class": "TCP_Profile", "idleTimeout": 60 } } } } } 6. CI/CID integration As you can notice, it is very easy to create a service with a WAF policy pulled from an external repo. So, it is easy to integrate these calls (or the AS3 call) into a CI/CD pipeline. Below, an Ansible playbook example. This playbook run the AS3 call above. That's it :) --- - hosts: bigip connection: local gather_facts: false vars: my_admin: "admin" my_password: "admin" bigip: "10.1.1.12" tasks: - name: Deploy AS3 WebApp uri: url: "https://{{ bigip }}/mgmt/shared/appsvcs/declare" method: POST headers: "Content-Type": "application/json" "Authorization": "Basic YWRtaW46YWRtaW4=" body: "{{ lookup('file','as3.json') }}" body_format: json validate_certs: no status_code: 200 7. FIND the Policy-ID When the policy is created, a Policy-ID is assigned. By default, this ID doesn't appearanywhere. Neither in the GUI, nor in the response after the creation. You have to calculate it or ask for it. This ID is required for several actions in a CI/CD pipeline. 7.1 Calculate the Policy-ID Wecreated this python script to calculate the Policy-ID. It is an hash from the Policy name (including the partition). For the previous created policy named"/Common/policy-api-arcadia",the policy ID is"Ar5wrwmFRroUYsMA6DuxlQ" Paste this python codein a newwaf-policy-id.pyfile, and run the commandpython waf-policy-id.py "/Common/policy-api-arcadia" Outcome will beThe Policy-ID for /Common/policy-api-arcadia is: Ar5wrwmFRroUYsMA6DuxlQ #!/usr/bin/python from hashlib import md5 import base64 import sys pname = sys.argv[1] print 'The Policy-ID for', sys.argv[1], 'is:', base64.b64encode(md5(pname.encode()).digest()).replace("=", "") 7.2 Retrieve the Policy-ID and fullPath with a REST API call Make this call below, and you will see in the response, all the policy creations. Find yours and collect thePolicyReference directive.The Policy-ID is in the link value "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0" You can see as well, at the end of the definition, the "fileReference"referring to the JSON file pulled by the BIG-IP. And please notice the"fullPath", required if you want to update your policy curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Range: 0-601/601' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ { "isBase64": false, "executionStartTime": "2020-07-22T11:23:42Z", "status": "COMPLETED", "lastUpdateMicros": 1.595417027e+15, "getPolicyAttributesOnly": false, "kind": "tm:asm:tasks:import-policy:import-policy-taskstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy/B45J0ySjSJ9y9fsPZ2JNvA?ver=16.0.0", "filename": "", "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "endTime": "2020-07-22T11:23:47Z", "startTime": "2020-07-22T11:23:42Z", "id": "B45J0ySjSJ9y9fsPZ2JNvA", "retainInheritanceSettings": false, "result": { "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "message": "The operation was completed successfully. The security policy name is '/Common/policy-api-arcadia'. " }, "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }, 8 UPDATE an existing policy It is pretty easy to update the WAF policy from a new JSON file version. To do so, collect from the previous call7.2 Retrieve the Policy-ID and fullPath with a REST API callthe"Policy" and"fullPath"directive. This is the path of the Policy in the BIG-IP. Then run the call below, same as1.2 PULL JSON file from a repository,but add thePolicy and fullPath directives Don't forget to APPLY this new version of the policy3. APPLY the policy curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' TIP : this call, above, can be used in place of the FIRST call when we created the policy "1.2PULL JSON file from a repository". But be careful, the fullPath is the name set in the JSON policy file. The 2 values need to match: "name": "policy-api-arcadia" in the JSON Policy file pulled by the BIG-IP "policy":"fullPath" in the POST call 9 Video demonstration In order to help you to understand how it looks with the BIG-IP, I created this video covering 4 topics explained in this article : The JSON WAF policy Pull the policy from a remote repository Update the WAF policy with a new version of the declarative JSON file Deploy a full service with AS3 and Declarative WAF policy At the end of this video, you will be able to adapt the REST Declarative API calls to your infrastructure, in order to deploy protected services with your CI/CD pipelines. Direct link to the video on DevCentral YouTube channel : https://youtu.be/EDvVwlwEFRw
How does F5 AS3 really work under the hood?
Put it simple,AS3 is a way to configure a BIG-IP once a BIG-IP is already provisioned. Full stop! We can also use AS3 to maintain that configuration over time. The way it works is we as a client send a JSON declaration via REST API and AS3 engine is supposed to work out how to configure BIG-IP the way it's been declared. AS3 internal components (parser and auditor) are explained further ahead. For non-DEV audience, AS3 is simply the name we give to an intelligent listener which acts as an interpreter that reads our declaration and translate it to proper commands to be issued on the BIG-IP. AS3 engine may or may not reside on BIG-IP (more on that on section entitled "3 ways of using AS3"). Yes, AS3 is declared in a structured JSON file and there are many examples on how to configure your regular virtual server, profiles, pools, etc,on clouddocs. AS3 uses common REST methods to communicate such as GET, POST andDELETE under the hood. For example, when we send our AS3 declaration to BIG-IP, we're sending an HTTP POST with the AS3 JSON file attached. AS3 is part of the Automation Toolchain whichincludes Declarative Onboarding and Telemetry Steaming. What AS3 is NOT Not a Mechanism for Role-BasedAccess Control (RBAC) AS3 doesn't support RBAC in a way that you can allow one user to configure certain objects and another user to configure other objects. AS3 has to use admin username/password with full access to BIG-IP resources. Not a GUI There's currently no native GUI built on top of AS3. Not an orchestrator AS3 won't and doesn't work out how to connect to different BIG-IPs and automatically figure out which box it needs to send which configuration to. All it does is receive a declaration, forwards it on and configure BIG-IP. Not for converting BIG-IP configuration We can't currently use AS3 to pull BIG-IP configuration and generate an AS3 configuration but I hope this functionality should be available in the future. Not for licensing or other onboarding functions We can't use AS3 for doing things like configuring VLANs or NTP servers. We use AS3 to configure BIG-IP once it's been already initially provisioned. For BIG-IP's initial set up, we useDeclarative Onboarding. Why should we use AS3? To configure and maintain BIG-IPs across multiple versions using the same automated workflow. A simple JSON declaration becomes the source of Truth with AS3, where configuration edits should only be made to the declaration itself. If multiple BIG-IP boxes use the same configuration, a single AS3 declaration can be used to configure the entire fleet. It can also be easily integrated with external automation tools such as Ansible and Terraform. What I find really REALLY cool about AS3 AS3 targets and supports BIG-IP version 12.1 and higher. Say we have an AS3 declaration that was previously used to configure BIG-IP v12.1, right? Regardless if we're upgrading or moving config to another box, we can still use the same declaration to configure BIG-IP v15.1 box in the same way. I'm not joking! Back in the F5 Engineering Services days, I still remember when I used to grab support tickets where the issue was a configuration from an earlier version that was incompatible with newer version, e.g. a profile option was moved to a different profile, or new feature was added that requires some other option to be selected, etc. This is supposed to be a thing of the past with AS3. AS3Key Features Transactional If you're a DBA, you've certainly heard of the term ACIDIC (atomicity, consistency, isolation, and durability). Let's say we send an AS3 declaration with 5 objects. AS3 will either apply the entire declaration or not apply at all. What that means is that if there's one single error, AS3 will never apply part of the configuration and leave BIG-IP in an unknown/inconsistent state. There's no in-between state. Either everything gets configured or nothing at all. It's either PASS or FAIL. Idempotent Say we send a declaration where there's nothing to configure on BIG-IP. In that case, AS3 will come back to client and inform that there's nothing for it to do. Essentially, AS3 won't remove BIG-IP's entire config and then re-apply it. It is smart enough to determine what work it needs to do and it will always do as little work as possible. Bounded AS3 enforces multi-tenancy by default, i.e. AS3 only creates objects in partitions (known as "tenants" in AS3 jargon) other than /Common. If we look at theAS3 declaration examples, we can see that a tenant (partition) is specified before we declare our config. AS3 does not create objects in the /Common partition. The exception to that is /Common/Shared when objects are supposed to be shared among multiple partitions/tenants. An example is when we create a pool member and a node gets automatically created on BIG-IP. Such node is created on /Common/Shared partition because that node might be a pool member in another partition, for example. Nevertheless, AS3 scope is and must always be bounded. The 3 ways of using AS3 Using AS3 through BIG-IP In this case here, we install AS3 RPM on each BIG-IP. BIG-IP is the box that has the "AS3 listener" waiting for us to send our AS3 JSON config file. All we need to do is to download AS3's binary and install it locally. There's a step by step guide for babies (with screenshots)hereusing BIG-IP's GUI. There's also a way to do it using curl if you're a geek like mehere. Using AS3 through BIG-IQ In this case, we don't need to manually install AS3 RPM on each BIG-IP box like in previous step. BIG-IQ does it for us. BIG-IQ v6.1.0+ supports AS3 and we can directly send declarations through BIG-IQ. Apart from installing, BIG-IQ also upgrades AS3 in the target box (or boxes) if they're using an older version. Analytics and RBAC are also supported. Using AS3 through Docker container This is where AS3 is completely detached from BIG-IP. In the Docker container set up, AS3 engine resides within a Docker container decoupled from BIG-IP. Say your environment have Docker containers running, which is not something uncommon nowadays. We can installAS3 in a Docker containerand use that container as the entry-point to send our AS3 declaration to BIG-IP. Yes, we as Cluent send our AS3 JSON file to where the Docker container is running and as long as the Docker container can reach our BIG-IP, then it will connect and configure it. Notice that in this case our AS3 engine runs outside of BIG-IP so we don't have to install AS3 on our BIG-IP fleet here. Docker container communicates with BIG-IP using iControl REST sendingtmshcommands directly. AS3 Internal Components AS3 engine is comprised ofan AS3 parser and AS3 auditor: AS3 Parser This is the front-end part of AS3 that communicates with the client andis responsible for client's declaration validation. AS3 Auditor After receiving validated declaration from AS3 parser, AS3 auditor's job is to compare desired validated declaration with BIG-IP's current configuration. It then determines what needs to be added/removed and forwards the changes to BIG-IP. AS3 in Action The way it works is Client sends a declaration to AS3 parser and config validation process kicks in. If declaration is not valid, it throws an error back to client along with an error code. If valid, declaration is forwarded on to AS3 auditor which then compares declaration with current BIG-IP's config and determines what needs to change. Only the configuration changes are supposed to be sent to BIG-IP, not the whole config. AS3 auditor then converts AS3 declaration totmshcommands and send convertedtmshconfig to BIG-IP via iControl REST. BIG-IP then pushes the changes viatmshcommands and returns success/error to AS3 auditor. If changes are not successful, an error is returned all the way to the client. Otherwise, successful code is returned to client and changes are properly applied to BIG-IP. Here's the visual description of what I've just said: Debugging AS3 AS3 schema validation errors are returned in HTTP Response with a message pointing to the specific error: This includes typos in property names and so on. Logs on BIG-IP are stored on/var/log/restnoded/restnoded.logand by default only errors are logged. Log level can be changed through theControlsobject in AS3 declaration itself. AS3 vs Declarative Onboarding This is usually source of confusion so I'd like to clarify that a bit. AS3 is the way we configure BIG-IP once it's already up and running. Declarative Onboarding (DO) is for the initial configuration of BIG-IP, i.e. setting up licence, users, DNS, NTP and even provisioning modules. Just like AS3,DO is API-only so no GUI on top of it. We can also have AS3 and DO in the same BIG-IP, so that's not a problem at all. Currently, there's no option to run it in a container like AS3 so as far as I'm concerned, it's only RPM based. Resources AS3 CloudDocs GitHub Repo Releases Declarative Onboarding (DO) CloudDocs GitHub Repo Releases I'd like to thank F5 Software Engineers Steven Chadwick and Garrett Dieckmann from AS3 team for providing a brilliant reference material.Introducing the F5 CLI – Easy button for the F5 Automation Toolchain
The F5 command-line interface (F5-CLI) is the latest addition to the F5 automation family and helps to enable the ease of F5 deployment of application services using the F5 Automation toolchain and F5 cloud services. The F5-CLI was built with ease of use in mind and takes its inspiration from other cloud-based command-line interfaces like those delivered via AWS, Azure, and GCP. The beauty of the F5 CLI is that you can easily deploy your applications by using the AS3/DO/TS or the cloud services declarative model without any software development or coding knowledge, or the need to use third-party tools or programming languages in order to make use of the APIS. Benefits of the F5 CLI: • Quickly access and consume F5’s APIs and Services with familiar remote CLI UX • Configurable settings • Include common actions in Continuous Deployment (CD) pipelines • Prototyping Test calls that may be used in more complex custom integrations using the underlying SDK Supports discovery activities/querying of command-line results (for example, “list accounts” to find the desired account which will be used as an input to final automation) • Support quick one-off automation activities (for example, leveraging a bash loop to create/delete large lists of objects) Ease of use We are going to show two examples of using the F5 Command Line Interface (F5 CLI) • Using the F5 CLI as a simple command line interface and • Using the F5 CLI for rapid F5 integration into a simple automation pipeline Using the F5 CLI straight from the command line The F5 CLI is delivered as a docker container or as a Python application and is very simple to run. One command is required to get the F5 CLI going as a docker image; for example: docker run -it -v "$HOME/.f5_cli:/root/.f5_cli" -v "$(pwd):/f5-cli" f5devcentral/f5-cli:latest /bin/bash And from there you can simply type “f5” and you will be presented with the command line options. #f5 --help Usage: f5 [OPTIONS] COMMAND [ARGS]... Welcome to the F5 command line interface. Options: --version Show the version and exit. --help Show this message and exit. Commands: bigip Manage BIG-IP login Login to BIG-IP, F5 Cloud Services, etc. cs Manage F5 Cloud Services config Configure CLI authentication and configuration --help is your friend here as it will show you all of the various options that you have. Let’s look at a simple example of logging in and sending an F5 AS3 declaration to a Big-IP: Login #f5 login --authentication-provider bigip --host 2.2.2.2 --user auser --password apassword { "message": "Logged in successfully" } Next, send a declaration to your BIG-IP, BIGIQ, F5 Cloud Services; for example: The following command creates a VIP on a BIG-IP with 2 pool members and associates a WAF Policy. bash-5.0# f5 bigip extension as3 create --declaration basicwafpolicy.json Reference: - Review the AS3 declaration above here Using the F5 CLI as part of a continuous deployment pipeline From there, it is also simple to think about ways that you could use the F5 CLI as part of a continuous deployment pipeline. This example uses Jenkins. Jenkins is an open-source automation server. It helps automate the parts of software development related to building, testing, deploying, and facilitating continuous integration and continuous delivery. It is relatively simple to use the F5 CLI as part of a Jenkins pipeline. In my example, I run a docker container that is running a container-based Cloud Bees Jenkins distribution which in turn is running docker inside of the container in order to make use of the F5 CLI. Configuring the F5 CLI to run on a Jenkins Docker container In order to get Jenkins running in my lab I use the cloud bees Jenkins distribution. In my case for this demonstration I am running docker on my laptop. A production deployment would be configured slightly differently. - First install docker: For more information on how to install docker, see the following.https://docs.docker.com/get-docker/ - Pull a cloud bees Jenkins distribution.https://hub.docker.com/r/cloudbees/cloudbees-jenkins-distribution/ docker pull cloudbees/cloudbees-jenkins-distribution - Run the following command in a terminal in order to start the initial Jenkins setup. docker run -u root -p 8080:8080 -p 50000:50000 -v ~/tmp/jenkins-data:/var/cloudbees-distribution -v /var/run/docker.sock:/var/run/docker.sock cloudbees/cloudbees-jenkins-distribution - Access the Jenkins interface to perform the initial setup on http://0.0.0.0:8080 You will need the initial admin password to proceed, which in my case is written into the terminal window from where I am running Jenkins. Jenkins initial setup is required. An admin user has been created and a password generated. Please use the following password to proceed to installation: d425a181f15e4a6eb95ca59e683d3411 This may also be found at: /var/cloudbees-jenkins-distribution/secrets/initialAdminPassword - Follow the in browser instructions to create users and then get the recommended plugins installed in Jenkins. - After you have Jenkins configured you must then install the docker plugin inside of the container. Open another terminal window and in the terminal window run docker ps ... make a note of the Jenkins container name ... docker exec -it JENKINSCONTAINERNAME /bin/bash ... inside the container shell ... apt install docker.io Setting up your Jenkins pipeline - Inside of Jenkins, select “new item” and then “pipeline.” You will then name your pipeline and select the “OK” button. - After that, you will be presented with a list of tabs and you should be able to copy the pipeline code below into the pipeline tab. This then enables a fairly simple Jenkins pipeline that shows how easy it is to build an automation process from scratch. The following is an example Jenkins pipeline that automates the F5-CLI: pipeline { /* These should really be in a vault/credentials store but for testing * they'll be added as parameters that can be overridden in the job * configuration. */ parameters { string(name: 'BIG-IP_ADDRESS', defaultValue: '2.2.2.2', description: 'The IP address for BIG-IP management.') string(name: 'BIG-IP_USER', defaultValue: 'auser', description: 'The user account for authentication.') password(name: 'BIG-IP_PASSWORD', defaultValue: 'apassword', description: 'The password for authentication.') booleanParam(name: 'DISABLE_TLS_VERIFICATION', defaultValue: true, description: 'Disable TLS and HTTPS certificate checking.') } agent { docker { image 'f5devcentral/f5-cli:latest' } } stages { stage('Prepare F5 CLI configuration') { steps { script { if(params.DISABLE_TLS_VERIFICATION) { echo 'Configuring F5 CLI to ignore TLS/HTTPS certificate errors' sh 'f5 config set-defaults --disable-ssl-warnings true' } } script { echo "Authenticating to BIG-IP instance at ${params.BIG-IP_ADDRESS}" sh "f5 login --authentication-provider bigip --host ${params.BIG-IP_ADDRESS} --user ${params.BIG-IP_USER} --password ${params.BIG-IP_PASSWORD}" } } } stage('Pull declaration from github (or not)') { steps { script { echo 'Pull Waff Policy from Github' sh 'wget https://raw.githubusercontent.com/dudesweet/simpleAS3Declare/master/basicwafpolicy.json -O basicwafpolicy.json' } } } stage('Post declaration to BIG-IP (or not)') { steps { script { echo 'Post declaration to BIG-IP (or not)' sh 'f5 f5 bigip extension as3 create --declaration basicwafpolicy.json' } } } } } Reference:Review the jenkinsfile here I have also included a 5-minute presentation and video demonstration that covers using the F5 CLI. The video reviews the F5 CLI then goes on to run the above Jenkins pipeline: https://youtu.be/1KGPCeZex6A Conclusion The F5 CLI is very easy to use and can get you up and running fast with the F5 API. You don’t have to be familiar with any particular programming languages or automation tools. You can use a command-line to use F5 declarative APIs for simple and seamless F5 service verification, automation, and insertion that can get you up and running with F5 services. Additionally, we showed how easy it is to integrate the F5 CLI into Jenkins, which is an open-source software development and automation server. You can begin your automation journey and easily include F5 as part of your automation pipelines. Perhaps the final thing I should say is that it doesn’t have to stop here. The next step could be to use the F5 Python SDK or to explore how F5 APIs can be integrated into the F5 Ecosystem like Ansible or Terraform and many more. For more information, your starting point should always be https://clouddocs.f5.com/. Links and References F5 SDK GitHub repo: https://github.com/f5devcentral/f5-sdk-python Documentation: https://clouddocs.f5.com/sdk/f5-sdk-python/ F5 CLI GitHub repo: https://github.com/f5devcentral/f5-cli Docker Container: https://hub.docker.com/r/f5devcentral/f5-cli Documentation: https://clouddocs.f5.com/sdk/f5-cli/ If you want to talk about F5 cloud solutions we have a Slack channel: Slack / F5cloudsolutions : https://f5cloudsolutions.slack.com Cloud bees Jenkins distribution: https://hub.docker.com/r/cloudbees/cloudbees-jenkins-distribution/ Docker: https://docs.docker.com/get-docker/Declarative Advanced WAF policy lifecycle in a CI/CD pipeline
The purpose of this article is to show the configuration used to deploy a declarative Advanced WAF policy to a BIG-IP and automatically configure it to protect an API workload by consuming an OpenAPI file describing the application. For this experiment, a Gitlab CI/CD pipeline was used to deploy an API workload to Kubernetes, configure a declarative Adv. WAF policy to a BIG-IP device and tuning it by incorporating learning suggestions exported from the BIG-IP. Lastly, the F5 WAF tester tool was used to determine and improve the defensive posture of the Adv. WAF policy. Deploying the declarative Advanced WAF policy through a CI/CD pipeline To deploy the Adv. WAF policy, the Gitlab CI/CD pipeline is calling an Ansible playbook that will in turn deploy an AS3 application referencing the Adv.WAF policy from a separate JSON file. This allows the application definition and WAF policy to be managed by 2 different groups, for example NetOps and SecOps, supporting separation of duties. The following Ansible playbook was used; --- - hosts: bigip connection: local gather_facts: false vars: my_admin: "xxxx" my_password: "xxxx" bigip: "xxxx" tasks: - name: Deploy AS3 API AWAF policy uri: url: "https://{{ bigip }}/mgmt/shared/appsvcs/declare" method: POST headers: "Content-Type": "application/json" "Authorization": "Basic xxxxxxxxxx body: "{{ lookup('file','as3_waf_openapi.json') }}" body_format: json validate_certs: no status_code: 200 The Advanced WAF policy 'as3_waf_openapi.json' was specified as follows: { "class": "AS3", "action": "deploy", "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.2.0", "id": "Prod_API_AS3", "API-Prod": { "class": "Tenant", "defaultRouteDomain": 0, "arcadia": { "class": "Application", "template": "generic", "VS_API": { "class": "Service_HTTPS", "remark": "Accepts HTTPS/TLS connections on port 443", "virtualAddresses": ["xxxxx"], "redirect80": false, "pool": "pool_NGINX_API", "policyWAF": { "use": "Arcadia_WAF_API_policy" }, "securityLogProfiles": [{ "bigip": "/Common/Log all requests" }], "profileTCP": { "egress": "wan", "ingress": { "use": "TCP_Profile" } }, "profileHTTP": { "use": "custom_http_profile" }, "serverTLS": { "bigip": "/Common/arcadia_client_ssl" } }, "Arcadia_WAF_API_policy": { "class": "WAF_Policy", "url": "http://xxxx/root/awaf_openapi/-/raw/master/WAF/ansible/bigip/policy-api.json", "ignoreChanges": true }, "pool_NGINX_API": { "class": "Pool", "monitors": ["http"], "members": [{ "servicePort": 8080, "serverAddresses": ["xxxx"] }] }, "custom_http_profile": { "class": "HTTP_Profile", "xForwardedFor": true }, "TCP_Profile": { "class": "TCP_Profile", "idleTimeout": 60 } } } } } The AS3 declaration will provision a separate Administrative Partition ('API-Prod') containing a Virtual Server ('VS_API'), an Adv. WAF policy ('Arcadia_WAF_API_policy') and a pool ('pool_NGINX_API'). The Adv.WAF policy being referenced ('policy-api.json') is stored in the same Gitlab repository but can be downloaded from a separate location. { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "transparent", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "http://xxxx/root/awaf_openapi/-/raw/master/App/openapi3-arcadia.yaml" } ] }, "modifications": [ ] } The declarative Adv.WAF policy is referencing in turn the OpenAPI file ('openapi3-arcadia.yaml') that describes the application being protected. Executing the Ansible playbook results in the AS3 application being deployed, along with the Adv.WAF policy that is automatically configured according to the OpenAPI file. Handling learning suggestions in a CI/CD pipeline The next step in the CI/CD pipeline used for this experiment was to send legitimate traffic using the API and collect the learning suggestions generated by the Adv.WAF policy, which will allow a simple way to customize the WAF policy further for the specific application being protected. The following Ansible playbook was used to retrieve the learning suggestions: --- - hosts: bigip connection: local gather_facts: true vars: my_admin: "xxxx" my_password: "xxxx" bigip: "xxxxx" tasks: - name: Get all Policy_key/IDs for WAF policies uri: url: 'https://{{ bigip }}/mgmt/tm/asm/policies?$select=name,id' method: GET headers: "Authorization": "Basic xxxxxxxxxxx" validate_certs: no status_code: 200 return_content: yes register: waf_policies - name: Extract Policy_key/ID of Arcadia_WAF_API_policy set_fact: Arcadia_WAF_API_policy_ID="{{ item.id }}" loop: "{{ (waf_policies.content|from_json)['items'] }}" when: item.name == "Arcadia_WAF_API_policy" - name: Export learning suggestions uri: url: "https://{{ bigip }}/mgmt/tm/asm/tasks/export-suggestions" method: POST headers: "Content-Type": "application/json" "Authorization": "Basic xxxxxxxxxxx" body: "{ \"inline\": \"true\", \"policyReference\": { \"link\": \"https://{{ bigip }}/mgmt/tm/asm/policies/{{ Arcadia_WAF_API_policy_ID }}/\" } }" body_format: json validate_certs: no status_code: - 200 - 201 - 202 - name: Get learning suggestions uri: url: "https://{{ bigip }}/mgmt/tm/asm/tasks/export-suggestions" method: GET headers: "Authorization": "Basic xxxxxxxxx" validate_certs: no status_code: 200 register: result - name: Print learning suggestions debug: var=result A sample learning suggestions output is shown below: "json": { "items": [ { "endTime": "xxxxxxxxxxxxx", "id": "ZQDaRVecGeqHwAW1LDzZTQ", "inline": true, "kind": "tm:asm:tasks:export-suggestions:export-suggestions-taskstate", "lastUpdateMicros": 1599953296000000.0, "result": { "suggestions": [ { "action": "add-or-update", "description": "Enable Evasion Technique", "entity": { "description": "Directory traversals" }, "entityChanges": { "enabled": true }, "entityType": "evasion" }, { "action": "add-or-update", "description": "Enable HTTP Check", "entity": { "description": "Check maximum number of parameters" }, "entityChanges": { "enabled": true }, "entityType": "http-protocol" }, { "action": "add-or-update", "description": "Enable HTTP Check", "entity": { "description": "No Host header in HTTP/1.1 request" }, "entityChanges": { "enabled": true }, "entityType": "http-protocol" }, { "action": "add-or-update", "description": "Enable enforcement of policy violation", "entity": { "name": "VIOL_REQUEST_MAX_LENGTH" }, "entityChanges": { "alarm": true, "block": true }, "entityType": "violation" } Incorporating the learning suggestions in the Adv.WAF policy can be done by simple copy&pasting the self-contained learning suggestions blocks into the "modifications" list of the Adv.WAF policy: { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "transparent", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "http://xxxxxx/root/awaf_openapi/-/raw/master/App/openapi3-arcadia.yaml" } ] }, "modifications": [ { "action": "add-or-update", "description": "Enable Evasion Technique", "entity": { "description": "Directory traversals" }, "entityChanges": { "enabled": true }, "entityType": "evasion" } ] } Enhancing Advanced WAF policy posture by using the F5 WAF tester The F5 WAF tester is a tool that generates known attacks and checks the response of the WAF policy. For example, running the F5 WAF tester against a policy that has a "transparent" enforcement mode will cause the tests to fail as the attacks will not be blocked. The F5 WAF tester can suggest possible enhancement of the policy, in this case the change of the enforcement mode. An abbreviated sample output of the F5 WAF Tester: ................................................................ "100000023": { "CVE": "", "attack_type": "Server Side Request Forgery", "name": "SSRF attempt (AWS Metadata Server)", "results": { "parameter": { "expected_result": { "type": "signature", "value": "200018040" }, "pass": false, "reason": "ASM Policy is not in blocking mode", "support_id": "" } }, "system": "All systems" }, "100000024": { "CVE": "", "attack_type": "Server Side Request Forgery", "name": "SSRF attempt - Local network IP range 10.x.x.x", "results": { "request": { "expected_result": { "type": "signature", "value": "200020201" }, "pass": false, "reason": "ASM Policy is not in blocking mode", "support_id": "" } }, "system": "All systems" } }, "summary": { "fail": 48, "pass": 0 } Changing the enforcement mode from "transparent" to "blocking" can easily be done by editing the same Adv. WAF policy file: { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "http://xxxxx/root/awaf_openapi/-/raw/master/App/openapi3-arcadia.yaml" } ] }, "modifications": [ { "action": "add-or-update", "description": "Enable Evasion Technique", "entity": { "description": "Directory traversals" }, "entityChanges": { "enabled": true }, "entityType": "evasion" } ] } A successful run will will be achieved when all the attacks will be blocked. ......................................... "100000023": { "CVE": "", "attack_type": "Server Side Request Forgery", "name": "SSRF attempt (AWS Metadata Server)", "results": { "parameter": { "expected_result": { "type": "signature", "value": "200018040" }, "pass": true, "reason": "", "support_id": "17540898289451273964" } }, "system": "All systems" }, "100000024": { "CVE": "", "attack_type": "Server Side Request Forgery", "name": "SSRF attempt - Local network IP range 10.x.x.x", "results": { "request": { "expected_result": { "type": "signature", "value": "200020201" }, "pass": true, "reason": "", "support_id": "17540898289451274344" } }, "system": "All systems" } }, "summary": { "fail": 0, "pass": 48 } Conclusion By adding the Advanced WAF policy into a CI/CD pipeline, the WAF policy can be integrated in the lifecycle of the application it is protecting, allowing for continuous testing and improvement of the security posture before it is deployed to production. The flexible model of AS3 and declarative Advanced WAF allows the separation of roles and responsibilities between NetOps and SecOps, while providing an easy way for tuning the policy to the specifics of the application being protected. Links UDF lab environment link. Short instructional video link.Automate Application Delivery with F5 and HashiCorp Terraform and Consul
Written by HashiCorp guest author Lance Larsen Today, more companies are adopting DevOps approach and agile methodologies to streamline and automate the application delivery process. HashiCorp enables cloud infrastructure automation, providing a suite of DevOps tools which enable consistent workflows to provision, secure, connect, and run any infrastructure for any application. Below are a few you may have heard of: Terraform Consul Vault Nomad In this article we will focus on HashiCorp Terraform and Consul, and how they accelerate application delivery by enabling network automation when used with F5 BIG-IP (BIG-IP).Modern tooling, hybrid cloud computing, and agile methodologies have our applications iterating at an ever increasing rate. The network, however, has largely lagged in the arena of infrastructure automation, and remains one of the hardest areas to unbottleneck. F5 and HashiCorp bring NetOps to your infrastructure, unleashing your developers to tackle the increasing demands and scale of modern applications with self-service and resilience for your network. Terraform allows us to treat the BIG-IP platform“as code”, so we can provision network infrastructure automatically when deploying new services.Add Consul into the mix, and we can leverage its service registry to catalog our services and enable BIG-IPs service discovery to update services in real time. As services scale up, down, or fail, BIG-IP will automatically update the configuration and route traffic to available and healthy servers. No manual updates, no downtime, good stuff! When you're done with this article you should have a basic understanding of how Consul can provide dynamic updates to BIG-IP, as well as how we can use Terraform for an “as-code” workflow. I’d encourage you to give this integration a try whether it be in your own datacenter or on the cloud - HashiCorp tools go everywhere! Note: This article uses sample IPs from my demo sandbox. Make sure to use IPs from your environment where appropriate. What is Consul? Consul is a service networking solution to connect and secure services across runtime platforms. We will be looking at Consul through the lens of its service discovery capabilities for this integration, but it’s also a fully fledged service mesh, as well as a dynamic configuration store. Head over to the HashiCorp learn portal for Consul if you want to learn more about these other use cases. The architecture is a distributed, highly available system. Nodes that provide services to Consul run a Consul agent. A node could be a physical server, VM, or container.The agent is responsible for health checking the service it runs as well as the node itself. Agents report this information to the Consul servers, where we have a view of services running in the catalog. Agents are mostly stateless and talk to one or more Consul servers. The consul servers are where data is stored and replicated. A cluster of Consul servers is recommended to balance availability and performance. A cluster of consul servers usually serve a low latency network, but can be joined to other clusters across a WAN for multi-datacenter capability. Let’s look at a simple health check for a Nginx web server. We’d typically run an agent in client mode on the web server node. Below is the check definition in json for that agent. { "service": { "name": "nginx", "port": 80, "checks": [ { "id": "nginx", "name": "nginx TCP Check", "tcp": "localhost:80", "interval": "5s", "timeout": "3s" } ] } } We can see we’ve got a simple TCP check on port 80 for a service we’ve identified as Nginx. If that web server was healthy, the Consul servers would reflect that in the catalog. The above example is from a simple Consul datacenter that looks like this. $ consul members Node Address Status Type Build Protocol DC Segment consul 10.0.0.100:8301 alive server 1.5.3 2 dc1 <all> nginx 10.0.0.109:8301 alive client 1.5.3 2 dc1 <default> BIG-IP has an AS3 extension for Consul that allows it to query Consul’s catalog for healthy services and update it’s member pools. This is powerful because virtual servers can be declared ahead of an application deployment, and we do not need to provide a static set of IPs that may be ephemeral or become unhealthy over time. No more waiting, ticket queues, and downtime. More on this AS3 functionality later. Now, we’ll explore a little more below on how we can take this construct and apply it “as code”. What about Terraform? Terraform is an extremely popular tool for managing infrastructure. We can define it “as code” to manage the full lifecycle. Predictable changes and a consistent repeatable workflow help you avoid mistakes and save time. The Terraform ecosystem has over 25,000 commits, more than 1000 modules, and over 200 providers. F5 has excellent support for Terraform, and BIG-IP is no exception. Remember that AS3 support for Consul we discussed earlier? Let’s take a look at an AS3 declaration for Consul with service discovery enabled. AS3 is declarative just like Terraform, and we can infer quite a bit from its definition. AS3 allows us to tell BIG-IP what we want it to look like, and it will figure out the best way to do it for us. { "class": "ADC", "schemaVersion": "3.7.0", "id": "Consul_SD", "controls": { "class": "Controls", "trace": true, "logLevel": "debug" }, "Consul_SD": { "class": "Tenant", "Nginx": { "class": "Application", "template": "http", "serviceMain": { "class": "Service_HTTP", "virtualPort": 8080, "virtualAddresses": [ "10.0.0.200" ], "pool": "web_pool" }, "web_pool": { "class": "Pool", "monitors": [ "http" ], "members": [ { "servicePort": 80, "addressDiscovery": "consul", "updateInterval": 15, "uri": "http://10.0.0.100:8500/v1/catalog/service/nginx" } ] } } } } We see this declaration creates a partition named “Consul_SD”. In that partition we have a virtual server named “serviceMain”, and its pool members will be queried from Consul’s catalog using the List Nodes for Service API. The IP addresses, the virtual server and Consul endpoint, will be specific to your environment.I’ve chosen to compliment Consul’s health checking with some additional monitoring from F5 in this example that can be seen in the pool monitor. Now that we’ve learned a little bit about Consul and Terraform, let’s use them together for an end-to-end solution with BIG-IP. Putting it all together This section assumes you have an existing BIG-IP instance, and a Consul datacenter with a registered service. I use Nginx in this example. The HashiCorp getting started with Consul track can help you spin up a healthy Consul datacenter with a sample service. Let’s revisit our AS3 declaration from earlier, and apply it with Terraform. You can check out support for the full provider here. Below is our simple Terraform file. The “nginx.json” contains the declaration from above. provider "bigip" { address = "${var.address}" username = "${var.username}" password = "${var.password}" } resource "bigip_as3" "nginx" { as3_json = "${file("nginx.json")}" tenant_name = "consul_sd" } If you are looking for a more secure way to store sensitive material, such as your BIG-IP provider credentials, you can check out Terraform Enterprise. We can run a Terraform plan and validate our AS3 declaration before we apply it. $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # bigip_as3.nginx will be created + resource "bigip_as3" "nginx" { + as3_json = jsonencode( { + Consul_SD = { + Nginx = { + class = "Application" + serviceMain = { + class = "Service_HTTP" + pool = "web_pool" + virtualAddresses = [ + "10.0.0.200", ] + virtualPort = 8080 } + template = "http" + web_pool = { + class = "Pool" + members = [ + { + addressDiscovery = "consul" + servicePort = 80 + updateInterval = 5 + uri = "http://10.0.0.100:8500/v1/catalog/service/nginx" }, ] + monitors = [ + "http", ] } } + class = "Tenant" } + class = "ADC" + controls = { + class = "Controls" + logLevel = "debug" + trace = true } + id = "Consul_SD" + schemaVersion = "3.7.0" } ) + id = (known after apply) + tenant_name = "consul_sd" } Plan: 1 to add, 0 to change, 0 to destroy. ------------------------------------------------------------------------ Note: You didn't specify an "-out" parameter to save this plan, so Terraform can't guarantee that exactly these actions will be performed if "terraform apply" is subsequently run. That output looks good. Let’s go ahead and apply it to our BIG-IP. bigip_as3.nginx: Creating... bigip_as3.nginx: Still creating... [10s elapsed] bigip_as3.nginx: Still creating... [20s elapsed] bigip_as3.nginx: Still creating... [30s elapsed] bigip_as3.nginx: Creation complete after 35s [id=consul_sd] Apply complete! Resources: 1 added, 0 changed, 0 destroyed Now we can check the Consul server and see if we are getting requests. We can see log entries for the Nginx service coming from BIG-IP below. consul monitor -log-level=debug 2019/09/17 03:42:36 [DEBUG] http: Request GET /v1/catalog/service/nginx (104.222µs) from=10.0.0.200:43664 2019/09/17 03:42:41 [DEBUG] http: Request GET /v1/catalog/service/nginx (115.571µs) from=10.0.0.200:44072 2019/09/17 03:42:46 [DEBUG] http: Request GET /v1/catalog/service/nginx (133.711µs) from=10.0.0.200:44452 2019/09/17 03:42:50 [DEBUG] http: Request GET /v1/catalog/service/nginx (110.125µs) from=10.0.0.200:44780 Any authenticated client could make the catalog request, so for our learning, we can use cURL to produce the same response. Notice the IP of the service we are interested in. We will see this IP reflected in BIG-IP for our pool member. $ curl http://10.0.0.100:8500/v1/catalog/service/nginx | jq [ { "ID": "1789c6d6-3ae6-c93b-9fb9-9e106b927b9c", "Node": "ip-10-0-0-109", "Address": "10.0.0.109", "Datacenter": "dc1", "TaggedAddresses": { "lan": "10.0.0.109", "wan": "10.0.0.109" }, "NodeMeta": { "consul-network-segment": "" }, "ServiceKind": "", "ServiceID": "nginx", "ServiceName": "nginx", "ServiceTags": [], "ServiceAddress": "", "ServiceWeights": { "Passing": 1, "Warning": 1 }, "ServiceMeta": {}, "ServicePort": 80, "ServiceEnableTagOverride": false, "ServiceProxyDestination": "", "ServiceProxy": {}, "ServiceConnect": {}, "CreateIndex": 9, "ModifyIndex": 9 } ] The network map of our BIG-IP instance should now reflect the dynamic pool. Last, we should be able to verify that our virtual service actually works. Let’s try it out with a simple cURL request. $ curl http://10.0.0.200:8080 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> That’s it! Hello world from Nginx! You’ve successfully registered your first dynamic BIG-IP pool member with Consul, all codified with Terraform! Summary In this article we explored the power of service discovery with BIG-IP and Consul. We added Terraform to apply the workflow “as code” for an end-to-end solution. Check out the resources below to dive deeper into this integration, and stay tuned for more awesome integrations with F5 and Hashicorp! References F5 HashiCorp Terraform Consul Service Discovery Webinar HashiCorp Consul with F5 BIG-IP Learn Guide F5 BIG-IP Docs for Service Discovery Using Hashicorp Consul F5 provider for Terraform Composing AS3 DeclarationsAS3 Foundations: Creating New Apps and Using Shared Objects
I joined in on the fun at several stops on the F5 Academy BIG-IP Next roadshow tour this fall, and in talking to customers, everyone is at various stages of their F5 automation journey, and some aren't automating at all yet. I'm kicking off a six-part series to set some foundational understanding of what AS3 is, how it works, what tools you might use to interact with it, and we'll finish it off with some best practices. If you want to learn or if you want to share your own experience with the class, join me over the next three weeks to finish 2023 in style! Episode Five - Dec 18th@ 9AM PST Demonstrate snippet templates and grab examples from CloudDocs Demonstrate iRule/Cert strings to base 64 encoding Demonstrate cert management options Demonstrate use cases for shared objects, and make clear any gotchas with such approachesManage Infrastructure and Services Lifecycle with Terraform and Ansible + Demo
Working as a Solution Architect for F5, Ioften need to have access to a lab environment. 'Traditionally', the method to implement a lab was to leverage tools like Vagrant,VMWare,or others. A lab environment on a laptop is limited by its computing capacities (CPU/Memory/disk/...).Today we are often asked to show how we can integrate our solutions with many different tools(Orchestration solutions, Version Control systems, CI Servers, containerized environments, ...). Except if your laptop is a powerful one, it's difficult to build such an environment and have itrun smoothly. If the lab requirements are too demanding for my laptop, Iwould access one of our lab facility to do my work. Thisapproach itself is fine but bring some challenges: If you travel like Ido, latency can become a hindrance and be frustrating. Lab facilities leverage "shared resources". Which means you may face issues due toconflicting IP addresses, switch misconfiguration, maintenance operations, ... Some resources may already be reserved/used by another fellow colleague and not be available. You may also face other constraints making both deployment models difficult: Need to share access to the lab. Not easy when it runs on your laptop or in a private cloud that is not always opened to the outside world. People may need to be able to replicate your lab in their own environment. Stability/time needed for maintenance: Using a lab over and over will make it messy. You usually At some point, you'll reach a stage where you want to create a "new" environment that is clean and "trustworthy" (until you played too much with it again) I'm sure i've missed other constraints but you get the idea: maintaining a lab and using it in a collaborativemanner is challenging. Luckily, it's easier today to achieve those objectices: Leverage Public Cloud! Public Cloud gives you access to "unlimited" computing services over Internet that can be automated/orchestrated. With Public Cloud, you have access to an API allowing you to spin up a new environment with all therelevant tools deployed. This way, you may go straight into work (after enjoying a nice cup of coffee/tea while yourinfrastructure is being deployed! ).Once your work is done, you can destroy this environment and save money. When you'll need a lab again, you'll be able to spin a new/clean environment in a matter of minutes and be confident that it's a "healthy lab" When working on Automation/Orchestration of Public cloud environments, I see two dominant tools: Terraform andAnsible. https://www.terraform.io Terraform is an open source command line tool that can be used to provision an infrastructure on dozensof different platforms and services (AWS, Azure, ...).One of the strength of Terraform is that it is declarative: You specify the expected "state" of yourinfrastructure and Terraform will take care of all the underlying complexities (Does it need to be provisioned? Should I update the settings of a component? Which components should be created first? Do we need to deleteresources that are not required anymore, ... ).Terraform will store the "state" of your infrastructure and configuration to be more efficient in its work. https://www.ansible.com Ansible is a provisioning and configuration management tool. It is designed to automate application deployments.One of the strength of Ansible is that it doesn't require any "agents" to run on the targetted systems. Ansibleworks by leveraging "Modules". Those modules are consumed to define the "state" of the targetted systems. They areusually executed over SSH (by default). So how to leverage those tools to have a lab available on-demand? In the following demo, we will: Leverage Terraform to manage the lifecycle of a new AWS environment: manage a dedicated VPC with external/internal subnets, Ubuntu instances, F5 solution) In addition to deploying our infrastructure, it will generate the relevant files for Ansible (inventory file to know theIPs of our systems, ansible variable files to know how to configure the F5 solution with AS3) Use Ansible to manage the configuration of our systems: update our ubuntu instances, install NGINX Web serviceon our Ubuntu instances, deploy a standard F5 configuration to load balance our web application with AS3 Here is a summary for the demo: Demo time! By leveraging tools like Terraform or Ansible (you can achieve the same results with other tools), it is easy to handle thelifecycle of an infrastructure and the services running on top of it. This is what people IaC (Infrastructure as Code) Useful links:- If you want to learn more about the setup of this demo, it is posted on Github: here- F5 provides a list of templates to automate deployment in public cloud. It's available here: AWS Templates, Azure Templates, GCP Templates- F5 Application Services 3 (AS3) documentation/examples: here- If you want to learn more about our API and how to automate/orchestrate F5 solutions (free training): F5 A&O Training
