Workflows are the fundamental mechanism by which Operations Manager gets things done. Workflows, in essence, are individual pieces of code (modules) that are chained together. For a primer on workflows and modules, I recommend the AuthorMPs site. Workflows remind me of Powershell pipelines or to a lesser extent, linux command piping. Individual commandlets (the equivalent of OpsMgr modules) can be mixed together in different ways to accomplish a more complex task (the equivalent of an OpsMgr workflow) by piping output from one to another. As long as the output from one commandlet can be processed by the next commandlet in the pipeline, you can chain them together.

One big difference between developing pipeline-aware commandlets and developing OpsMgr modules and workflows is the potential for debugging. With Powershell, you can use the full debugging power of Visual Studio (or whatever editor you use) by simply setting breakpoints in your commandlet and attaching to the host running powershell (usually powershell.exe). Workflow debugging is certainly possible but not nearly as easy. There are several reasons for this:

• If you’re not using custom-made modules, the running code is a black box. If the module does not already have the ability to output tracing information, there is no way to add it.
• You can’t just run a workflow from a debugger. OpsMgr is complete control of when it starts and ends, so you must do this indirectly.
• If you are using custom-made modules, you need to find the OpsMgr process that they get loaded into in order to debug them.

One of the most common problems in developing workflows is trying to figure why (or why not) a data item is making it through certain parts of the workflow such as condition detection modules. In order to avoid “trial and error” development, you need some way to debug the workflows. I’ll outline a few options below for troubleshooting this kind of scenario below.

Option 1: Attach a Debugger

If you’ve developed a custom module, you can step through the code by doing the following:

• Get OpsMgr to start running your workflow. For example if you are debugging a monitor, make sure the monitor is enabled and it’s datasource is receiving data.
• Open the code in your debugger (Visual Studio in this example)
• Attach to the OpsMgr process running your workflow. This is the tricky step. The process is MonitoringHost.exe, but there may be several. You want the one that is running managed code (see the Type column in screenshot below) with the credentials of the Operations Manager action account. If it is still not obvious which process to attach to, you’ll have to attach to each one and see if your DLL module is loaded. Use the module window (Ctrl+Alt+U) to list the DLLs once you attach to the process. Once you see your DLL, you’re read to debug!

On a side note, I (or one of our team members) may do a blog post in the future on creating managed code modules.

Option 2: Use the Operations Manager Trace Files

Honestly I would leave this as a last resort, but the trace files can be useful beyond workflow debugging (such as script debugging). See #8 on my top 10 list for instructions and comments on this.

Option 3: Use the F5 Workflow Tracer

Inspired by the workflow tracer that was #6 in my top 10 list, our team has built a debugging module that can be used to trace the output of any module in your workflows. We used the same configuration elements as the workflow tracer and added support for data item batching, robust acknowledgements, and other minor improvements.

The module can be placed after any module in your workflows, and reports on the data item that come out of that module. Here is an example from the F5 Monitoring Log:

Element: F5.PRO.PerformanceMonitorType (note: this tells me which workflow is being traced)
Stage: After COUNTER TYPE detection (note: this tells me which module is being traced)
Target: {4447e45e-9f65-e01d-2c17-24dee61fb1d9}

<?xml version="1.0" encoding="utf-16"?>
<DataItem type="System.Performance.LinkedData" time="2009-11-12T12:32:46Z" sourceHealthServiceId="fa056941-2b22-7d8e-a34d-7604e1347f99">
  <ObjectName>LTM Pool Member</ObjectName>
  <CounterName>Server - Current Connections</CounterName>
  <InstanceName />
  <IsNull>false</IsNull>
  <Value>0</Value>
  <ManagedEntityId>{29c7fc90-0f44-b70c-9f88-5a7ee6d13a40}</ManagedEntityId>
  <RuleId>{0e799c3d-b8d0-bce0-259b-f9025d507996}</RuleId>
</DataItem>

This module can be downloaded here. You’ll need to import the included MP into OpsMgr and place the included DLL in the Operations Manager program files folder. Instructions for installation and configuration are the same as the original workflow tracer, except with our MP and DLL.

Another barrier to debugging OpsMgr becomes apparent if you want to use this module to debug a sealed workflow (one that’s in a sealed MP and is not marked public). There is no way to simply insert a module of your own in such a workflow since you cannot change a sealed MP. The workaround goes like this:

• Create an override for the workflow that disables it
• Export the MP that the workflow is in to XML format (use the #1 tool on my top 10 list).
• Get the XML for the workflow from the exported pack and create a copy of it in your own, new, MP. If the workflow references other elements marked “internal” to the MP, you will have to create copies of those as well.
• Insert the debugging module wherever you’d like in your new workflow.
• Import your new pack

Simple as that ! :)

To help understand this process, I’ve created an override pack for our own F5 PRO-Enabled Management Pack that replaces one of our monitors with one that includes the debugging module between each module in the workflow. I realize the pack looks strange as formatted in our wiki – just copy and paste from the <xml> element on down to </ManagementPack>.

Option 4: Use the Operations Manager Workflow Analyzer

I was both elated and saddened by the brand new workflow analyzer included in the new authoring console. Elated because this tool rocks, and saddened because it really would have helped to have it earlier :). This tool does the following things:

• Shows you all the workflows registered in OpsMgr as well as their state
  • Technical note: Internally this is basically running the “Show Running Rules and Monitors” task that is already available via the UI

• Allows you trace any of the workflows
  • Technical note: Internally this is overriding the TraceEnabled property on the workflow and hooking into OpsMgr’s built in tracing capabilities. I’ve looked for a way to override this property through the UI, and it does not seem possible.

• Allows you do an ‘analysis’ of the workflow. This was not very interesting to me – just shows you some basic properties and running instances.

You can increase the amount of information printed out in the trace if you turn on verbose logging (see Option 2 above).

The main drawback of this tool is that it’s hooking into trace calls made by the OpsMgr modules so you’re not going to get this level of detail if you’re trying to debug a custom-built module. It may be possible to write to the same mechanism, but I have not looked into that yet.

Happy Debugging!

<< Back to Part I

During that last year and a half that I’ve been on the F5 Management Pack team, we’ve had to do quite a bit of Systems Center Operations Manager debugging. I think it is certainly possible to develop management packs without needing to delve into Operations Manager internals at all, but much of what our pack does is outside the norm. For example:

• We have a custom-built agent on the Big-IP LTM box that our management pack talks to using a proprietary protocol
• We’ve built custom OpsMgr workflow modules in order to meet our performance objectives
• We just released a PRO pack which is fairly cutting-edge technology so there is very little documentation to go on

In this series of posts, I’ll share some of the techniques we’ve used for debugging and developing management packs. So, without further adieu, I present part I – the top 10 most useful tools.

10. Runtime Health Explorer

Troubleshooting state change problems can be tricky. There are several ways to view health and monitor states of your discovered instances, but not all in one place. This tool does just that, and has a synchronization feature as well.

9. DIR

Yes, the lowly DIR command. Simple, yet powerful.

Try running this command:

c:\Program Files\System Center Operations Manager 2007\Health Service State\>dir *.PS1 /s /b

If you have scripts included in your management pack, they will be placed in a subfolder of this directory before they are executed. Looking at these scripts can help troubleshoot selectors and script caching. A great explanation of this technique can be found here.

8. Operations Manager Tracing Configuration Scripts

By turning verbose logging on, you can see an amazing amount of detail around what OpsMgr is doing. I have found this technique most useful when debugging scripts, especially ones using the new powershell module. For example, you may have an error in your script because you forgot to add a snapin and the normal task window will not show you what the problem is. The TracingGuidsScript.log file will show the error.

7. Module Explorer

It is critical to understand what modules are available to you when developing new workflows. This tool shows you all the modules available in your current OpsMgr environment.

6. Workflow Tracer

If you have a workflow with several ConditionDetection modules and you’re losing data items somewhere, this tool is extremely useful. This module can be inserted in any of your workflows (between each module in a workflow is the best for debugging) and it will log the data items that are passing through. This is also useful for understanding what the raw data items look like.

Part II of this series will be a further exploration and extension of this concept.

5. Lineage Explorer

Much thanks to SCOMNIVORE for three of the tools on this list, including this one. This tool is similar to the Module Explorer but allows you to view class and relationship type lineage as well as module lineage.

4. Reflector

Operations Manager is not open source, but there is an enormous amount of .NET code that can be decompiled and examined. One of the posts in this series (#3 I think) will cover this topic. However, I’ll leave you with this little snippet from Microsoft.EnterpriseManagement.HealthService.DLL:

public enum ModuleType
{
    DataSource,
    Condition,
    ReadAction,
    WriteAction
}

3. SQL Server Query Analyzer

A good understanding of the OperationsManager and OperationsManagerDW databases goes a long way when troubleshooting MP issues. There are also an enormous number useful OpsMgr SQL queries available out there. This guy covers a couple dozen in just one post and our own Julian Balog just posted a couple queries for troubleshooting statistics.

2. System Center Operations Manager 2007 R2 Authoring Resource Kit (Authoring Console)

I wasn’t incredibly impressed with the original authoring console, but the just-released new version looks great. It also includes several new tools for debugging. This is the best (only?) tool that targets the beginner to intermediate level MP author. I’ve used the console several times to author a module and then save it to XML to learn about the XML schema and how things are supposed to be put together.

1. MP to XML Dumper

This simple tool is #1 because the best way to learn and debug MPs in my experience is to look through other MPs, especially the built-in OpsMgr ones. This can also be done with a powershell one-liner from the OpsMgr powershell console:

Get-ManagementPack | Export-ManagementPack –Path “C:\packs\”

Honorable Mention:

• MP Viewer: This didn’t make the list because the authoring console has all the same features and much more. The one nice thing about this tool however is that you can open a XML or MP file directly without needing to connect to OpsMgr, resolve references, etc.

If you have a great tool that is not on this list, please post a comment. I’ll make it “Top 11” if there’s something really good that I missed!

>> Part II: WorkFlows

The new PRO-Enabled management pack enables integration with Virtual Machine Manager (VMM) via PRO-Tips. This integration enables scenarios in which statistics and health events collected from Big-IPs by the core management pack are used to trigger actions by VMM.

One example of this kind of scenario is to use VMM to start up virtual machines when a traffic spike starts overwhelming your existing pool of servers. The same virtual machines can be suspended or stopped when the traffic load returns to normal. We’ve shown how the management pack can be used to enable these scenarios in a series of data orchestration videos. The new PRO-Enabled management gives the VMM administrator control and auditing of the actions that VMM is instructed to take.

When the PRO pack is installed, a new folder installed into Systems Center Operations Manager:

The new views available are:

• F5 PRO Actions: Clicking on this view causes available PRO actions to show up in the actions pane. Currently the only action is “Discover PRO Pool Members”
• F5 PRO Alerts: Any PRO alerts that you enable will be shown here. These alerts cause VMM to generate PRO tips.
• F5 PRO Monitored Pool Members: This view shows all pool members that have PRO alerts enabled.

After the PRO Pack is installed, the PRO monitors need to be enabled and configured in order to start functioning. The configuration page for the “Add Server Resource” monitor is shown below:

By default, the configuration of the monitor is as follows:

• The monitor is disabled
• The Big-IP statistic that it monitors is LTM Pool Member Server Current Connections
• The monitor is in a healthy state when current connections are less than or equal to zero
• The monitor is in a critical state when the current connections are greater than zero

The monitor goes to a critical state when it detects incoming connections because the pool member that is being monitored has been set up on the Big-IP to receive connections only when the primary pool is overwhelmed. The data orchestration videos go into more detail on this setup.

When the monitor goes into a critical state it creates a PRO alert and causes VMM to generate a PRO Tip which can be viewed from the PRO Tip window:

The VMM administrator can then choose to implement the tip or dismiss the tip. Implementing the tip will cause new server resources to spin up. Some details of what the tip actually does can be configured from within Systems Center.

The VMM administrator can also set options that govern how all PRO tips are handled:

• PRO tips can be enabled/disabled by severity level
• Tips can be configured to automatically run by severity level

Once the action has taken place, the Jobs windows from within VMM can be used to view the audit trail of the actions that were taken by implementing the PRO tip:

Please visit the F5 Management Pack downloads page to download the new PRO pack. In the coming weeks we’ll have more in-depth coverage of what you can do with the PRO pack, so stay tuned!

We’ve had several folks ask about how the Management Pack scales and how to determine what kind of hardware is needed when monitoring large numbers of Big-IPs. This article will outline how to tune statistics collection in the Management Pack to best suite your needs and available hardware.

Keep in mind that Operations Manager and SQL Server have countless ways to optimize performance and that those options should be reviewed first as they potentially will make much more of a difference that just tuning the F5 Management Pack. One example is that your Data Warehouse should be running on a separate server that is optimized for SQL Server (such as having data and logs on separate disks).

Once your general infrastructure is optimized, it’s time to consider which performance metrics you’d like to gather. By default only the global (device level) stats are collected but you can turn on collection for CPUs, Nodes, Pool Members, etc. Please see the Performance Collecting and Monitoring topic for exactly how to turn on statistics collections.

In our testing we’ve found that a single Operations Manager management server running on a Dell 2950 can handle up to 250 statistics/sec from monitoring Big-IPs. I would recommend trying to shoot for a lower number than that so that you have some CPU cycles left for monitoring other types of objects in your network and handling non-OpsMgr tasks.

So let’s assume that we don’t want to go above 150 stats/sec for each management server. You can use the following table to calculate how many stats/sec you will generate by turning on the various categories of statistics:

As I mentioned only the Global stats are turned on by default and are collected every 2 minutes, so out-of-the-box stats/sec when monitoring 2 Big-IPs can be calculated  as such:

((2 Big-IP * 49 stats) / 120 seconds) = .8 stats/sec

Now let’s suppose you want to turn on stats for Pool Members, and each Big-IP has 20 pool members. The additional stat load would be the following:

(2 Big-IP * 20 Pool Members Each * 7 stats per pool member) / 120 seconds = 23 stats/sec

We now would be collecting a grand total of 23.8 stats/sec – still lots of room to grow! Armed with this methodology for calculating load, you can now balance your need for each kind of statistic against the load it will put  on the management server.

One way to determine for sure how many stats/sec are currently being generated is to view the various F5 counters that are installed in the Windows Performance Monitor.

For more detailed information on which stats are collected, please see the health model wiki page.

The Management Pack team is very excited to present this release which features an expanded set of PowerShell APIs and server tasks. The PowerShell API has been integrated with the Operations Manager PowerShell API as well as with the BIG-IP security model. These new features enable some very compelling scenarios, especially in the arena of Data Center Orchestration. You’ll hear more from us on that topic soon, but for now here’s an overview of the new features.

For more in-depth information on these topics including tutorial videos, visit the ManagementPack Home page and the Management Pack Video site.

BIG-IP Security Model Integration

We’ve added the ability to associate Windows NT accounts with BIG-IP accounts. Once you set up this mapping, any server task or PowerShell commandlet that needs to talk to a BIG-IP will log into the BIG-IP under the account that maps to whichever NT account is performing the operation.

As an example, let’s say I’m currently logged in as DOMAIN\joelh. I can map my account to the “admin” account using a new server task or using the following PowerShell command. Note that a secure dialog box will prompt me for the BIG-IP password:

Set-F5.DeviceConfigurationToken –DeviceIp 172.27.25.235 –UserName pooladmin

Now, when I run the “Disable Pool Member” server task, the task will only succeed if the “pooladmin” user has permissions on the BIG-IP to disable this particular pool member:

The security model also applies when using any of the F5 MP commandlets:

$token = Get-F5.DeviceConfigurationToken 172.27.25.235 // retrieves token that is mapped to current NT account
Disable-PoolMember –Token $token –Member 10.10.10.10:80

Operations Manager PowerShell Integration

This piece of integration essentially means that you can pass Operations Manager PowerShell objects into any of the the F5 Management Pack commandlets. This allows you to leverage the full power of the Operations Manager command shell when administering BIG-IPs. Here is a quick example that disables any pool members that have been put into Maintenance Mode:

$pmsNeedingDisable = @(Get-MonitoringObject -MonitoringClass $f5LTMPoolMemberClass -Criteria "EnabledState =`'Enabled`' and (InMaintenanceMode = 1)")
$pmsNeedingDisable | Disable-F5.LTMPoolMember -Token $token

An even quicker way to take advantage of this capability is by opening the Operations Manager Shell from within the context of a discovered object. An example of the power of this capability is shown below. I right-click an LTM pool in the Operations Manager diagram view and open the Operations Manager Shell from the context menu. I can then type a simple command to disable all pool members in the pool:

.  

Other Additions

There are many more additions in this release including

• New server tasks such as Create Pool Member and Force Pool Member Offline
• 10 new commandlets
• Lots of bug fixes and performance improvements

Visit the Management Pack Home page for the download.

F5 Management Pack v1.0.0.527 adds support for discovering and managing Viprion devices. This includes discovery of CPUs on each blade with an added Slot ID property in Operations Manager to differentiate them. Additional monitor states are also available in the Viprion health model.

There are many other fixes and additions in this release including:

• Improved support for Big-IPs running version TMOS version 10.x
• Improved reliability when running on Windows Server 2008
• Support for management of up to 20 devices across multiple management servers
• An updated big3d agent (10.0.1.600) that includes scalability improvements
• Relationship types defined in the F5 Management Pack are now public, enabling creation dependency monitors that reference F5 monitors
• Rules that generate alerts now have Alert Suppression enabled

You can find the free downloads as well as documentation, videos, and the support forums on the F5 Management Pack home page.