BIG-IP : iControl : LocalLBDataGroupFile.set_local_path() : swap large file under load
F5 BIG-IP v11.4.1 (Build 635.0) LTM on ESXi I have a .NET C app that uses iControl to perform following sequence : • transfer data-file to staging location on BIG-IP device • recache data-group with contents of this data-file The specific iControl API used for the recache is : LocalLBDataGroupFile.set_local_path() This operation has been 100% consistently successful in non-prod environments with very low traffic to perform data-group-file updates of up to 2M records. Non-prod config : single stand-alone device ( no HA pair ). The operation has also been 100% consistently successful in prod environments with high traffic to update a non-live data-group-file up to 300K records. Prod config : HA pair consisting of 2-node sync-failover device-group with auto-sync disabled. NOTE: By "live data-group-file" I mean an enabled virtual-server has an assigned iRule that references the data-group ( performs matches against data-group maps ). By "non-live data-group-file" I mean that the data-group exists but either is not referenced by any iRules, or iRules that reference it currently are not assigned to any enabled virtual-server. Here is where the problem occurs : When the operation is run in prod environments with high traffic (40-60% baseline cpu utilization, 200-400 Mbps baseline throughput) to update a live data-group-file ( 100K+ records ) the iRule fails. Exactly how the iRule fails is unknown and currently is under investigation by F5 Support, however here are some data-points : • the file-transfer and data-group recache iControl calls return success to the C caller • requests that the iRule normally conditionally rewrites to various backend pools no longer arrive at those servers • BIG-IP logs contain zero errors related to either the iControl operation of the iRule • public client requests that should be processed by the iRule display generic Akamai 500 error-pages NOTE: I have a test that removes Akamai from the equation, but have not yet had an opportunity to run it. My understanding is that LocalLBDataGroupFile.set_local_path() was re-designed/coded for 11.4 and was lab-tested up to 1M records. However, I wonder if any testing was performed in an environment with significant load ? Through trial-and-error I discovered the following workaround (for an HA pair only) : • create a/b pair of data-groups and corresponding set of a/b iRules that are identical except that "a" iRule references "a" data-group, and "b" iRule references "b" data-group • on active node, initially configure virtual-server to use "a" iRule • use C application to update "b" data-group-file ( NOTE: possibly this could also be accomplished via the admin browser, but above 100K records the time-lags and potential impact on prod operations become concerning. ) • if now swap-in "b" iRule to virtual-server ( effectively swapping-in "b" data-group ) the irule will begin to behave strangely (requests swallowed and never routed to backend pool although no errors present in LTM logs) • however, the following "trick" seems to work : sync active to standby promote standby to active on new active, swap-in "b" iRule to the virtual-server reboot new standby* sync new active to new standby Somehow the sync operation "cures" the issues induced by swapping the live iRule to point to a just-updated data-group. So in summary it seems that for a high-load environment attempting to swap new contents into a live data-group somehow induces a failure-case for iRule lookups against that data-group. The failure symptoms are identical both for the technique of re-caching the live data-group with new contents ( iControl API LocalLBDataGroupFile.set_local_path() ), and for the iRule a/b swap technique. However, an active-to-standby sync operation seems to "cure" whatever bad-state the data-group has been put into. Can anyone provide insights as to why swapping-in new contents to a large data-group-file associated with an iRule assigned to a VIP under heavy load would cause iRule data-group lookup failures ?BIG-IP : sync-failover device-group sync options
F5 BIG-IP Virtual Edition v11.4.1 (Build 635.0) LTM on ESXi For a device-group of type sync-failover, the admin browser provides these options : Automatic Sync Full Sync Maximum Incremental Sync Size (KB) Could someone please explain these options ? Are syncs on a time-schedule ? Or are syncs triggered by any change to the primary ? Or exactly what types of changes will trigger a sync ? Specifically, will a change to a data-group file ( e.g. add/delete a line ) trigger a sync ?What is BIG-IQ?
tl;dr - BIG-IQ centralizes management, licensing, monitoring, and analytics for your dispersed BIG-IP infrastructure. If you have more than a few F5 BIG-IP's within your organization, managing devices as separate entities will become an administrative bottleneck and slow application deployments. Deploying cloud applications, you're potentially managing thousands of systems and having to deal with traditionallymonolithic administrative functions is a simple no-go. Enter BIG-IQ. BIG-IQ enables administrators to centrally manage BIG-IP infrastructure across the IT landscape. BIG-IQ discovers, tracks, manages, and monitors physical and virtual BIG-IP devices - in the cloud, on premise, or co-located at your preferred datacenter. BIG-IQ is a stand alone product available from F5 partners, or available through the AWS Marketplace. BIG-IQ consolidates common management requirements including but not limited to: Device discovery and monitoring: You can discovery, track, and monitor BIG-IP devices - including key metrics including CPU/memory, disk usage, and availability status Centralized Software Upgrades: Centrally manage BIG-IP upgrades (TMOS v10.20 and up) by uploading the release images to BIG-IQ and orchestrating the process for managed BIG-IPs. License Management: Manage BIG-IP virtual edition licenses, granting and revoking as you spin up/down resources. You can create license pools for applications or tenants for provisioning. BIG-IP Configuration Backup/Restore: Use BIG-IQ as a central repository of BIG-IP config files through ad-hoc or scheduled processes. Archive config to long term storage via automated SFTP/SCP. BIG-IP Device Cluster Support: Monitor high availability statuses and BIG-IP Device clusters. Integration to F5 iHealth Support Features: Upload and read detailed health reports of your BIG-IP's under management. Change Management: Evaluate, stage, and deploy configuration changes to BIG-IP. Create snapshots and config restore points and audit historical changes so you know who to blame. 😉 Certificate Management: Deploy, renew, or change SSL certs. Alerts allow you to plan ahead before certificates expire. Role-Based Access Control (RBAC): BIG-IQ controls access to it's managed services with role-based access controls (RBAC). You can create granular controls to create view, edit, and deploy provisioned services. Prebuilt roles within BIG-IQ easily allow multiple IT disciplines access to the areas of expertise they need without over provisioning permissions. Fig. 1 BIG-IQ 5.2 - Device Health Management BIG-IQ centralizes statistics and analytics visibility, extending BIG-IP's AVR engine. BIG-IQ collects and aggregates statistics from BIG-IP devices, locally and in the cloud. View metrics such as transactions per second, client latency, response throughput. You can create RBAC roles so security teams have private access to view DDoS attack mitigations, firewall rules triggered, or WebSafe and MobileSafe management dashboards. The reporting extends across all modules BIG-IQ manages, drastically easing the pane-of-glass view we all appreciate from management applications. For further reading on BIG-IQ please check out the following links: BIG-IQ Centralized Management @ F5.com Getting Started with BIG-IQ @ F5 University DevCentral BIG-IQ BIG-IQ @ Amazon Marketplace
Best tool for managing multiple Big-IPs?
Those of you with multiple Big-IP devices to manage, do you use Big-IQ or some other alternative? We have a couple of 7200s with four VMs each in active/standby pairs, plus a 2200 and a few lab VMs. I'd like to be able to go to one place to see all of my nodes, pools, vservers, etc rather than have to swap back and forth between devices.
regex/grok patterns for BigIP Logs
Hi I'm ingesting logs into the ELK stack. quite nice. Only problem is i'm having to write the regex patterns by hand for each log type (tmm/secure/user). Is there a wiki that lists: log syntax (e.g. fields used for TMM log and field order in the log) regexes for these formats I'd be really pleased if there was even just a page that listed (1) log syntax for each type I've seen in the discussions here that log formats also change between versions... any ideas on how often this happens...?BIG-IP : standby auto-sync to active
BIG-IP 11.4.1 Build 635.0 Hotfix HF2 For a sync-failover device-group in active-standby configuration with auto-sync enabled. What is behavior following making a config-change to standby device ( specifically, update data-group contents ) ? Will the changes be immediately sync'd to the active ? Does auto-sync ensure delta-zero between active/standby ? i.e. is it possible for active to have changes not yet sync'd to standby at time change is made to standby ?
