iControl Apps - TMM Statistics

Continuing on with my series of applications on system level statistics, this application will look into the insides of the TMOS processes on the system and dump out the available statistics exposed in the System.Statistics.get_tmm_statistics() method.

Usage

The arguments for this application are the address, username, and password of the BIG-IP.  This is declared in the top of the script with the following param statement.  There is also a Write-Usage function to display the arguments to the user.

param (
  $g_bigip = $null,
  $g_uid = $null,
  $g_pwd = $null
);

Set-PSDebug -strict;

function Write-Usage()
{
  Write-Host "Usage: GlobalIPStats.ps1 host uid pwd";
  exit;
}

Initialization

As is with all of my PowerShell scripts, the initialization component will look to see if the iControlSnapIn is loaded into the current PowerShell session.  If not, the Add-PSSnapIn Cmdlet is called to add the snapin into the runtime.  Then a call to the Initialize-F5.iControl cmdlet is made to setup the connection to the BIG-IP.  If this succeeds, then a call to the Get-TMMStatistics function is called to query the TMM Statistics and output them to the console.

function Do-Initialize()
{
  if ( (Get-PSSnapin | Where-Object { $_.Name -eq "iControlSnapIn"}) -eq $null )
  {
    Add-PSSnapIn iControlSnapIn
  }
  $success = Initialize-F5.iControl -HostName $g_bigip -Username $g_uid -Password $g_pwd;
  
  return $success;
}

#-------------------------------------------------------------------------
# Main Application Logic
#-------------------------------------------------------------------------
if ( ($g_bigip -eq $null) -or ($g_uid -eq $null) -or ($g_pwd -eq $null) )
{
  Write-Usage;
}

if ( Do-Initialize )
{
  Get-TMMStatistics;
}
else
{
  Write-Error "ERROR: iControl subsystem not initialized"
}

Querying TMM Statistics

Since there can be more than one instance of the TMM running on your specific system, this application will poll the statistics for all TMM instances and report on each.  It does so by calling the System.Statistics.get_all_tmm_statistics() method.  Alternately we could have made a call to query all TMM instances and then passed those values to the System.Statistics.get_tmm_statistics() methods, but I figured that I'd save you all a little bandwidth here. 

The code below makes the call to get_all_tmm_statistics() which returns a TMMStatistics structure with the timestamp that the statistics were taken at as well as an array of TMMStatisticEntry structures containing the TMM instance information and all the associated statistics.  I've broken out the processing of the TMMStatisticEntry structure into the Process-TMMStatisticEntry function below.

function Get-TMMStatistics()
{
  param($subset = $null);
  if ( $subset -eq $null ) { $subset = "all"; }
  
  Write-Host "TMM Statistics"
  
  $TMMStatistics = (Get-F5.iControl).SystemStatistics.get_all_tmm_statistics();
  $t = Get-TimeFromTimeStamp $TMMStatistics.time_stamp;
  $Statistics = $TMMStatistics.statistics;
  
  foreach($TMMStatisticEntry in $Statistics)
  {
    Process-TMMStatisticEntry $TMMStatisticEntry
  }
}

Processing the TMMStatisticEntry

The Process-TMMStatisticEntry function takes as input a TMMStatisticEntry structure.  As in my previous applications in this series, I'm using PowerShell's ability to create empty objects and add attributes to them with the Add-Member CmdLet.  The TMMStatisticEntry's statistics array contains all the Common.Statistics structures for each of the specific TMM's statistics.  This array is looped on and user friendly labels are created and then added to the newly create object.

When all of the statistics have been processed, the object containing the user-friendly names and statistic values is passed to the Format-List PowerShell Cmdlet to allow for the nice columnar formatting that it provides.

function Process-TMMStatisticEntry()
{
  param($TMMStatisticEntry);
  
  $stat_obj = New-Object -TypeName System.Object;
  
  $stat_obj | Add-Member -Type noteProperty -Name "Time Stamp" $t;
  $stat_obj | Add-Member -Type noteProperty -Name "--------------------------------" "";
  
  $Statistics = $TMMStatisticEntry.statistics;

  foreach($Statistic in $Statistics)
  {
    $val = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
    switch ($Statistic.type)
    {
      "STATISTIC_TM_PID" {
        $label = "TMOS Processing Agent Process ID";
      }
      "STATISTIC_TM_CPU" {
        $label = "TMOS Processing Agent CPU #";
      }
      "STATISTIC_TM_TMID" {
        $label = "TMOS Processing Agent Instance";
      }
      "STATISTIC_TM_NPUS" {
        $label = "TMOS Processing Agent count";
      }
      "STATISTIC_CLIENT_SIDE_BYTES_IN" {
        $label = "Bytes - Client Side In";
      }
      "STATISTIC_CLIENT_SIDE_BYTES_OUT" {
        $label = "Bytes - Client Side Out";
      }
      "STATISTIC_CLIENT_SIDE_PACKETS_IN" {
        $label = "Packets - Client Side In";
      }
      "STATISTIC_CLIENT_SIDE_PACKETS_OUT" {
        $label = "Packets - Client Side Out";
      }
      "STATISTIC_CLIENT_SIDE_CURRENT_CONNECTIONS" {
        $label = "Connections - Client Side Current";
      }
      "STATISTIC_CLIENT_SIDE_MAXIMUM_CONNECTIONS" {
        $label = "Connections - Client Side Max";
      }
      "STATISTIC_CLIENT_SIDE_TOTAL_CONNECTIONS" {
        $label = "Connections - Client Side Total";
      }
      "STATISTIC_SERVER_SIDE_BYTES_IN" {
        $label = "Bytes - Server Side In";
      }
      "STATISTIC_SERVER_SIDE_BYTES_OUT" {
        $label = "Bytes - Server Side Out";
      }
      "STATISTIC_SERVER_SIDE_PACKETS_IN" {
        $label = "Packets - Server Side In";
      }
      "STATISTIC_SERVER_SIDE_PACKETS_OUT" {
        $label = "Packets - Server Side Out";
      }
      "STATISTIC_SERVER_SIDE_CURRENT_CONNECTIONS" {
        $label = "Connections - Server Side Current";
      }
      "STATISTIC_SERVER_SIDE_MAXIMUM_CONNECTIONS" {
        $label = "Packets - Server Side Max";
      }
      "STATISTIC_SERVER_SIDE_TOTAL_CONNECTIONS" {
        $label = "Packets - Server Side Total";
      }
      "STATISTIC_TM_TOTAL_CYCLES" {
        $label = "TMOS Total CPU Cycles";
      }
      "STATISTIC_TM_IDLE_CYCLES" {
        $label = "TMOS Idle CPU Cycles";
      }
      "STATISTIC_TM_SLEEP_CYCLES" {
        $label = "TMOS Yielded CPU Cycles";
      }
      "STATISTIC_MAINTENANCE_MODE_DENIALS" {
        $label = "Connections - Denials due To Maintenance";
      }
      "STATISTIC_VIRTUAL_ADDRESS_MAXIMUM_CONNECTION_DENIALS" {
        $label = "Connections - Denials Due To Address Limits";
      }
      "STATISTIC_VIRTUAL_SERVER_MAXIMUM_CONNECTION_DENIALS" {
        $label = "Connections - Denials Due To VS Limits";
      }
      "STATISTIC_VIRTUAL_SERVER_NON_SYN_DENIALS" {
        $label = "Packets - Denials Due To SYN Attacks";
      }
      "STATISTIC_NO_HANDLER_DENIALS" {
        $label = "Packets - Denials Due To No Handlers";
      }
      "STATISTIC_LICENSE_DENIALS" {
        $label = "Packets - Denials Due to License Restrictions";
      }
      "STATISTIC_TM_CMP_CONN_REDIRECTED" {
        $label = "Connections - Redirected To Other CMP Processing Agent";
      }
      "STATISTIC_CONNECTION_FAILED_MEMORY_ERRORS" {
        $label = "Connection - Dropped Due To Memory Contstraints";
      }
      "STATISTIC_MEMORY_TOTAL_BYTES" {
        $label = "Memory - TMM Total";
      }
      "STATISTIC_MEMORY_USED_BYTES" {
        $label = "Memory - TMM Used";
      }
      "STATISTIC_DROPPED_PACKETS_TOTAL" {
        $label = "Packets - Total Dropped";
      }
      "STATISTIC_ERRORS_IN" {
        $label = "Errors on Ingress";
      }
      "STATISTIC_ERRORS_OUT" {
        $label = "Errors on Egress";
      }
      "STATISTIC_HTTP_TOTAL_REQUESTS" {
        $label = "Requests - Total HTTP Requests";
      }
      default {
        $type = $Statistic.type;
        Write-Error "Label '$type' not found"
      }
    }
    #Write-Host "$label : $val"
    $stat_obj | Add-Member -Type noteProperty -Name $label $val;
  }
  $stat_obj | format-list
}

Utility functions

Several utility functions are included as well that will do the 64 bit math conversion as well as converting the iControl TimeStamp structure to a .Net DateTime. 

function Get-TimeFromTimeStamp()
{
  param ($TimeStamp);
  $dt = new-object -typename System.DateTime
  $dt = $dt.AddYears($TimeStamp.year-1).AddMonths($TimeStamp.month-1).AddDays($TimeStamp.day-1);
  $dt = $dt.AddHours($TimeStamp.hour).AddMinutes($TimeStamp.minute).AddSeconds($TimeStamp.second);
  return $dt;
}

function Convert-To64Bit()
{
  param($high, $low);
  return ($high*[Math]::Pow(2,32))+$low;
}

Usage

By passing in the BIG-IP address, username, and password, the application displays the following values for my 6400 running BIG-IP v9.4.5.

PS C:\> .\GlobalTMMStats.ps1 bigip_address username password
TMM Statistics

Time Stamp                                             : 9/4/2008 3:07:30 PM
--------------------------------                       :
TMOS Processing Agent Process ID                       : 1843
TMOS Processing Agent CPU #                            : 1
TMOS Processing Agent Instance                         : 0
TMOS Processing Agent count                            : 1
Bytes - Client Side In                                 : 1592610157
Bytes - Client Side Out                                : 368247248
Packets - Client Side In                               : 10423120
Packets - Client Side Out                              : 1307746
Connections - Client Side Current                      : 62
Connections - Client Side Max                          : 1704
Connections - Client Side Total                        : 900245
Bytes - Server Side In                                 : 124809743
Bytes - Server Side Out                                : 372942115
Packets - Server Side In                               : 1738424
Packets - Server Side Out                              : 1383771
Connections - Server Side Current                      : 61
Packets - Server Side Max                              : 114
Packets - Server Side Total                            : 900244
TMOS Total CPU Cycles                                  : 6.85920895943003E+15
TMOS Idle CPU Cycles                                   : 6.84824103521506E+15
TMOS Yielded CPU Cycles                                : 0
Connections - Denials due To Maintenance               : 0
Connections - Denials Due To Address Limits            : 0
Connections - Denials Due To VS Limits                 : 0
Packets - Denials Due To SYN Attacks                   : 0
Packets - Denials Due To No Handlers                   : 25348
Packets - Denials Due to License Restrictions          : 0
Connections - Redirected To Other CMP Processing Agent : 0
Connection - Dropped Due To Memory Contstraints        : 0
Memory - TMM Total                                     : 931135488
Memory - TMM Used                                      : 25043732
Packets - Total Dropped                                : 0
Errors on Ingress                                      : 0
Errors on Egress                                       : 0
Requests - Total HTTP Requests                         : 80475

Conclusion

Running of the application shows some interesting statistics that are not otherwise exposed in any way.  Ever wanted to look at the number of Egress errors, or Denials due to SYN attacks?  Well, now you have an easy way to monitor these values and many many more.  While this application was written in PowerShell, it is very simple to implement this logic in the language of your choice.

A full version of this script can be found under the PsTMMStatistics entry in the iControl CodeShare.

 

Get the Flash Player to see this player.