iControl Apps - #05 - Rate Based Statistics
One of the key features of iControl is the ability to monitor statistics of the various objects on the system. Whether it's for capacity planning or auditing and billing purposes, the numbers behind how your applications are being utilized on the network are very important. The iControl API allows for querying of statistics for most of the major objects including Virtual Servers, Pools, Pool Members, and Node Addresses. This data is returned as counters but most often the desired presentation of this data is in the form of a rate such as bits or connections per second. This article will show how to query the counter values and calculate rate based statistics for a given virtual server.
Usage
The arguments for this application are the address, username, and password of the BIG-IP along with optional argument for the specified virtual server. This is declared in the top of the script with the following param statement. There is also a Write-Usage function to display the arguments to the user.
param (
$g_bigip = $null,
$g_uid = $null,
$g_pwd = $null,
$g_virtual = $null
);
$g_gmt_offset = "";
Set-PSDebug -strict;
function Write-Usage()
{
Write-Host "Usage: VirtualRates.ps1 host uid pwd [virtual_server]";
exit;
}
Initialization
As is with all of my PowerShell scripts, the initialization component will look to see if the iControlSnapIn is loaded into the current PowerShell session. If not, the Add-PSSnapIn Cmdlet is called to add the snapin into the runtime. Then a call to the Initialize-F5.iControl cmdlet to setup the connection to the BIG-IP. If this succeeds then success, then the rest of the optional arguments are processed.
If a virtual server name is not specifed, the Get-Virtuals function is called to list out all available virtual servers on the BIG-IP.
If a virtual server name is specified, the Get-VirtualRates function is called which continuously polls the BIG-IP in one second intervals. During each interval it extracts the total bits, packets, and connections and then calculates rates based off of those numbers.
The code for this process is listed below.
function Do-Initialize()
{
if ( (Get-PSSnapin | Where-Object { $_.Name -eq "iControlSnapIn"}) -eq $null )
{
Add-PSSnapIn iControlSnapIn
}
$success = Initialize-F5.iControl -HostName $g_bigip -Username $g_uid -Password $g_pwd;
return $success;
}
if ( ($g_bigip -eq $null) -or ($g_uid -eq $null) -or ($g_pwd -eq $null) )
{
Write-Usage;
}
if ( Do-Initialize )
{
if ( $g_virtual -eq $null )
{
Get-Virtuals;
}
else
{
Get-VirtualRates $g_virtual;
}
}
else
{
Write-Error "ERROR: iControl subsystem not initialized"
}
Listing the Virtuals
The Get-Virtuals function will call the LocalLB.VirtualServer.get_list() method to query the list of virtual servers on the BIG-IP. It will then loop over that list and write the results to the console.
function Get-Virtuals()
{
$vs_list = (Get-F5.iControl).LocalLBVirtualServer.get_list();
Write-Host "Available Virtual Servers:";
foreach ($vs in $vs_list)
{
Write-Host " $vs";
}
}
Reporting Rates
The Get-VirtualRates function does most of the grunt work in this application. It takes as a parameter a virtual server name and then enters a continuous loop.
In this loop, the LocalLB.VirtualServer.get_statistics() method is called. The Common.TimeStamp structure from the result of that call is then converted to a .Net DateTime with the local Get-TimeFromTimeStamp function.
After the time conversion, the returned statistics for the specified virtual server and looks for the STATISTIC_CLIENT_SIDE_BYTES_IN, STATISTIC_CLIENT_SIDE_PACKETS_IN, and STATISTIC_CLIENT_SIDE_TOTAL_CONNECTIONS to use for calculations.
If this is the first time through the loop, there is no reference for a rate, so the values returned are stored in the b0, p0, and c0 variables respectively. Note the Convert-To64Bit function used here. We'll talk about that later but it basically takes the Common.UInt64 structure's high and low 32 bit values and turns it into a 64 bit number.
For all subsequent trips through the loop, a difference is calculated between the new and old counter values and these are stored in the b2, p2, and c2 variables. The difference between the first and current poll is calculated by subtracting the t1 and t0 variables and this results in a .Net TimeSpan object that we extract the total seconds from and store that value in the sec variable. For sanity, if the time difference rounds to zero, we'll set it to one to avoid a divide by zero error.
At this point another sanity check is performed to make sure that the difference in counters is not negative. If it's negative, meaning that the last value is smaller than the first, that would mean that either the statistics have rolled over the 2^64 boundary or that they were manually reset. If this is the case, then we'll throw this poll away and treat it as if the polling started over again.
Finally if we have a valid poll, the rates are calculated by total number over the interval by the time difference resulting in a X/sec rate. The values for bits per second, packets per second, and connections per second are then presented to the console.
Function Get-VirtualRates()
{
param($virtual_server);
$bFirst = 1;
$b0 = 0;
$p0 = 0;
$c0 = 0;
$t0 = [DateTime]::Now;
Write-Host "Rates for Virtual Server: $virtual_server"
while (1)
{
$VirtualServerStatistics = (Get-F5.iControl).LocalLBVirtualServer.get_statistics( (, $virtual_server) );
$t1 = Get-TimeFromTimeStamp $VirtualServerStatistics.time_stamp;
$VirtualServerStatisticEntry = $VirtualServerStatistics.statistics[0];
$Statistics = $VirtualServerStatisticEntry.statistics;
foreach ($Statistic in $Statistics)
{
switch ($Statistic.type) {
"STATISTIC_CLIENT_SIDE_BYTES_IN" {
$b1 = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
}
"STATISTIC_CLIENT_SIDE_PACKETS_IN" {
$p1 = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
}
"STATISTIC_CLIENT_SIDE_TOTAL_CONNECTIONS" {
$c1 = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
}
}
}
if ($bFirst -eq 1 )
{
$bFirst = 0;
$b0 = $b1;
$p0 = $p1;
$c0 = $c1;
$t0 = $t1;
}
else
{
$b2 = $b1 - $b0;
$p2 = $p1 - $p0;
$c2 = $c1 - $c0;
$t2 = $t1 - $t0;
$sec = $t2.Seconds;
if ( $sec -eq 0 ) { $sec = 1 }
if ( ($b2 -lt 0) -or ($p2 -lt 0) -or ($c2 -lt 0) )
{
# either the counters increased past 2^64 or they were reset so start over.
$bFirst = 1;
}
else
{
# Calculate rates
$br = $b2/$sec;
$pr = $p2/$sec;
$cr = $c2/$sec;
Write-Host "$br bps; $pr pps; $cr cps"
}
}
Start-Sleep -s 1
}
}
Converting Time
The Get-TimeFromTimeStamp function takes as input an iControl Common.TimeStamp structure and then creates a native .Net TimeStamp structure from those values and returns it to the calling function.
function Get-TimeFromTimeStamp()
{
param ($TimeStamp);
$dt = new-object -typename System.DateTime
$dt = $dt.AddYears($TimeStamp.year-1)
.AddMonths($TimeStamp.month-1)
.AddDays($TimeStamp.day-1)
.AddHours($TimeStamp.hour)
.AddMinutes($TimeStamp.minute)
.AddSeconds($TimeStamp.second);
return $dt;
}
Converting to 64 bit
The iControl value returned for statistics is the Common.UInt64 structure which has two 32 bit values for the low and high 32 bits of the resulting 64 bit number. Unfortunately, at the time of this article, PowerShell does not support bitwise shifting so we can't create the 64 bit number natively in PowerShell. Never fear! Since PowerShell supports .Net and .Net supports inline compiling and running of .Net code, we can inline compile the C# needed to do the conversion. I found a handy PowerShell script by Lee Holmes/Joel Bennett providing support for inline C# in PowerShell. I converted this script to a powershell function. The Convert-To64Bit function builds the C# 32 bit parts to 64 bit conversion code, calls the Invoke-Inline function with it, and returns the result.
function Convert-To64Bit()
{
param($high, $low);
return Invoke-Inline "result.Add((Convert.ToUInt64($high)<<32) | (Convert.ToUInt64($low)));"
}
Running C# inline in PowerShell
As mentioned above, I took the Invoke-Inline powershell script from Lee Holmes/Joel Bennett and converted it into a inline function for this application. I'm not going to go into the details of this function but in short it takes as input some C# code, builds a cached compiled version of it, executes the supplied code, and then returns the results - if there are any. Very cool!
function Invoke-Inline()
{
param(
[string[]] $code,
[object[]] $arguments,
[string[]] $reference = @()
)
## Stores a cache of generated inline objects. If this library is dot-sourced
## from a script, these objects go away when the script exits.
if(-not (Test-Path Variable:\inlineCode.Cache))
{
${GLOBAL:inlineCode.Cache} = @{}
}
$using = $null;
$source = $null;
if($code.length -eq 1) {
$source = $code[0]
} elseif($code.Length -eq 2){
$using = $code[0]
$source = $code[1]
} else {
Write-Error "You have to pass some code, or this won't do anything ..."
}
## un-nesting magic (why do I need this?)
$params = @()
foreach($arg in $arguments) { $params += $arg }
$arguments = $params
## The main function to execute inline C#.
## Pass the argument to the function as a strongly-typed variable. They will
## be available from C# code as the Object variable, "arg".
## Any values assigned to the "returnValue" object by the C# code will be
## returned to MSH as a return value.
## See if the code has already been compiled and cached
$cachedObject = ${inlineCode.Cache}[$source]
#Write-Verbose "Type: $($arguments[0].GetType())"
## The code has not been compiled or cached
if($cachedObject -eq $null)
{
$codeToCompile =
@"
using System;
using System.Collections.Generic;
$using
public class InlineRunner
{
public List<object> Invoke(Object[] args)
{
List<object> result = new List<object>();
$source
if( result.Count > 0 ) {
return result;
} else {
return null;
}
}
}
"@
Write-Verbose $codeToCompile
## Obtains an ICodeCompiler from a CodeDomProvider class.
$provider = New-Object Microsoft.CSharp.CSharpCodeProvider
## Get the location for System.Management.Automation DLL
$dllName = [PsObject].Assembly.Location
## Configure the compiler parameters
$compilerParameters = New-Object System.CodeDom.Compiler.CompilerParameters
$assemblies = @("System.dll", $dllName)
$compilerParameters.ReferencedAssemblies.AddRange($assemblies)
$compilerParameters.ReferencedAssemblies.AddRange($reference)
$compilerParameters.IncludeDebugInformation = $true
$compilerParameters.GenerateInMemory = $true
## Invokes compilation.
$compilerResults =
$provider.CompileAssemblyFromSource($compilerParameters, $codeToCompile)
## Write any errors if generated.
if($compilerResults.Errors.Count -gt 0)
{
$errorLines = ""
foreach($error in $compilerResults.Errors)
{
$errorLines += "`n`t" + $error.Line + ":`t" + $error.ErrorText
}
Write-Error $errorLines
}
## There were no errors. Store the resulting object in the object
## cache.
else
{
${inlineCode.Cache}[$source] =
$compilerResults.CompiledAssembly.CreateInstance("InlineRunner")
}
$cachedObject = ${inlineCode.Cache}[$source]
}
Write-Verbose "Argument $arguments`n`n$cachedObject"
## Finally invoke the C# code
if($cachedObject -ne $null)
{
return $cachedObject.Invoke($arguments)
}
}
Running the program
The following command will not supply the virtual server name so a list of available virtual servers will be returned.
PS C:\> .\VirtualRates.ps1 theboss admin admin Available Virtual Servers: xpbert-ftp xpbert-http xpbert-ssh xpbert-telnet
For this example, I'll monitor the xpbert-http virtual server. For this test, the virtual is initially idle and then I ran apachebench against the virtual with 10000 total/50 concurrent connections to the default page on the webserver. You'll see starting at poll #3 the cps starts increasing and then when the bit counts start kicking in the bits per second (bps) and packets per second (pps) start increasing. When the apachebench process completes, you will see that the rates will start decreasing over time. Since there is no traffic on the virtual it's fairly obvious that since the time is still increasing but the statistic over that interval remains the same, then the rate will decrease.
PS C:\> .\VirtualRates.ps1 theboss admin admin xpbert-http Rates for Virtual Server: xpbert-http 0 bps; 0 pps; 0 cps 0 bps; 0 pps; 0 cps 0 bps; 0 pps; 77 cps 0 bps; 0 pps; 232 cps 32398.8 bps; 411.6 pps; 350.8 cps 207088.166666667 bps; 2625.5 pps; 453.5 cps 177504.142857143 bps; 2250.42857142857 pps; 528.285714285714 cps 203525.625 bps; 2585.625 pps; 570.5 cps 216543 bps; 2751 pps; 554.9 cps 246497.363636364 bps; 3131.54545454545 pps; 550 cps 298047.083333333 bps; 3783.75 pps; 554.083333333333 cps 255468.928571429 bps; 3243.21428571429 pps; 545.357142857143 cps 253460 bps; 3220 pps; 567.2 cps 271884.0625 bps; 3454.0625 pps; 593.75 cps 287913.705882353 bps; 3657.70588235294 pps; 588.235294117647 cps 301427.611111111 bps; 3829.38888888889 pps; 555.555555555556 cps 275500 bps; 3500 pps; 500 cps 262380.952380952 bps; 3333.33333333333 pps; 476.190476190476 cps 250454.545454545 bps; 3181.81818181818 pps; 454.545454545455 cps 239565.217391304 bps; 3043.47826086957 pps; 434.782608695652 cps 229583.333333333 bps; 2916.66666666667 pps; 416.666666666667 cps 220400 bps; 2800 pps; 400 cps 211923.076923077 bps; 2692.30769230769 pps; 384.615384615385 cps 204074.074074074 bps; 2592.59259259259 pps; 370.37037037037 cps 196785.714285714 bps; 2500 pps; 357.142857142857 cps 190000 bps; 2413.79310344828 pps; 344.827586206897 cps 183666.666666667 bps; 2333.33333333333 pps; 333.333333333333 cps 177741.935483871 bps; 2258.06451612903 pps; 322.58064516129 cps 172187.5 bps; 2187.5 pps; 312.5 cps 166969.696969697 bps; 2121.21212121212 pps; 303.030303030303 cps
Conclusion
In this article, I've presented a way to query a virtual server for it's throughput and connection statistics and use that data to calculate rates based on the time interval between polls. This logic can easily be extended to any other object on the BIG-IP such as pools, pool members, or node addresses.
For a full version of this script, check out the wiki entry for PsRateBasedStatistics in the iControl CodeShare.
