Event Notification with PERL and SSL/TLS

Some additional information:

 

 

TLSv1 seems to cause the trouble. If I create the SOAP::Transport::HTTP::Daemon object WITHOUT SSL_version => 'TLSv1', it seems to be more stable and it's possible to restart the Event Listener without problems. After I used TLSv1, the Event Subsystem on the LB get's somehow "stuck" and I had to delete the Notification Subscription and create a new one, otherwise the LB ceased to send Events. Restarting the httd daemon on the LB did not help.

 

 

Does the iControl Event Subsystem support TLSv1?

 

 

Enabling debug for the iControl system (b db iControl.Loglevel debug) did not help either. There are debug messages in the log, however nothing that's related to the SSL/TLS and/or connection problems.

 

 

BTW: What's the component/process that sends events to the Event Listener? Maybe I can debug that process..

 

 

Thank you!

 

 

Regards

 

Kurt Knochner

Hi Kurt, glad you are trying out the event notification interface. You asked a lot of questions so I'll try to start by explaining how eventd works and hopefully that will get you covered.

 

 

At the core of BIG-IP is our configuration engine mcpd. It contains a notification service for daemons on the device to notify them when the state of it's configuration changes (ie. an object is added/deleted in the GUI).

 

 

 

I wrote eventd as another one of those listening daemons. Right now, the only way to configure eventd is by building a client with the iControl Management.EventSubscription interface (which you probably figured out). The client then configures "listeners" which contain what type of notifications it wants to hear about.

 

 

 

The eventd sits in a event loop listening for events. When one occurs, it looks in it's configuration (stored on disk under /config/eventd.xml) for listeners that want to be notified of that specific state change. It then pushes the change into a queue for each listener.

 

 

 

Listeners can be configured to buffer responses based on various criteria (see the EventSubscription's create method.

 

 

 

When the threshold for a notification occurs (max messages, timeslice, etc), it will attempt to make a connection to the EventNotification interface on your listener. If the connection attempt fails, it retries a couple of times (also configurable). After the retry limit fails, it will put the listener in a "disabled" state. The only way to bring this back up is to re-enable it with the EventSubscription methods. This is probably what's happening to you.

 

 

 

As for a SSL connection, I honestly haven't tested that. I'm not sure why it wouldn't work and I'm not sure if it's a client thing (in my daemon) or a server thing (on the perl side). Definitely something I'll have to test out when I can get to it.

 

 

 

As for the user field, that was originally included as a future placeholder if we ever got the username included down in the configuration daemon for changes. That was on the roadmap at the time but still hasn't been implemented so for now, the username will be empty.

 

 

 

If you want to debug eventd, you should do the following

 

 

 

1) bigstart shutdown eventd

 

2) $ /sbin/eventd -d -f

 

 

 

That will run eventd in the foreground printing debug statements.

 

 

 

Hopefully this helps clear some things up.

 

 

 

-Joe

 

 

 

 

 

Hi Joe,

 

 

thank you for the fast answer!

 

 

So, eventd is the one that sends the Events to my Perl Listener? If so, I will try to debug that process, as shown in your post.

 

 

I've done more tests and it seems to be TLSv1. SSLv2/v3 seems to be O.K. Probably it's a problem with both my perl script (IO Library) as it crashes sometimes when TLSv1 is used (mybe due to the data coming from eventd). eventd also seems to lock up after I use TLSv1 on the client. I see no further TCP packets to my listener port after that, unless I remove the subscription and create a new one. I will try to debug eventd and report back. Can you please check how eventd handles TLSv1?

 

 

 

Regarding the username. Is there any way to get the name of the user who changed the config? Audit logging provides that information, so it must be available somewhere. However, the audit log is written by httpd, which obviously knows the user name. Maybe you could scan /var/log/audit within eventd (similar to alertd) and create Events for any audit log entry ;-))

 

 

Any idea how I can solve my problem, i.e. implement my own "audit/config change" tracker that also includes the user name?

 

 

Thank you!

 

 

Regards

 

Kurt

 

Hi Joe,

 

 

I've done some eventd debugging now. Output below. Unfortunately, I cannot see any reason for the problem in the debug output. Any idea what's going wrong?

 

 

[root@xxxxxxxx-C2:Standby] alertd /usr/bin/eventd -d -f

 

 

RuntimeOptions::queryAdminIP: Admin IP = xx.21.xx.11

 

Entering Notification Thread...

 

Received TRANSACTION @ 4ea6c6a2 (1319552674) (r=1/c=0/p=0)!

 

retrieval start_transaction

 

ClusterMonitor::handleOperation: clustered.environment = false

 

ClusterMonitor::handleOperation: s = ST_UK, e = false, p = unknown --> EV_NC

 

ClusterMonitor::handleOperation: a = ACT_READ, ns = ST_NC

 

ClusterMonitor::handleRead

 

Configuration::load()

 

Creating Consumer

 

Creating Consumer

 

ClusterMonitor::handleOperation: cluster.primary = false

 

ClusterMonitor::handleOperation: s = ST_NC, e = false, p = false --> EV_NC

 

ClusterMonitor::handleOperation: a = ACT_NONE, ns = ST_NC

 

ClusterMonitor::handleOperation: cluster.mgmtipaddr = ::

 

retrieval end_transaction

 

Received TRANSACTION @ 4ea6c6a8 (6) (r=0/c=0/p=1)!

 

MCPSerializer::serialize - operation 2908 (modify); object 2122 (db_variable)

 

MCPSerializer::addData(), db_variable_name : string

 

MCPSerializer::addData(), db_variable_value : string

 

MCPSerializer::addData(), db_variable_object_id : ulong

 

MCPSerializer::addData(), db_variable_transaction_id : ulong

 

adding event to consumer DF99021F-1974-742B-C185-77067226AA6

 

Processing consumer: 'DF99021F-1974-742B-C185-77067226AA6' + objects: 1; time (t0,t1,delta): (1319552650, 1319552680, 30)

 

adding event to consumer 2AA91888-1974-9995-C03A-461AB90526A

 

Processing consumer: '2AA91888-1974-9995-C03A-461AB90526A' + objects: 1; time (t0,t1,delta): (1319552636, 1319552680, 44)

 

timerExpired

 

timerExpired

 

 

==> tcpdump output (running in the background)

 

LB: 10.11.57.2

 

Listener (using TLSv1): 10.30.42.98.11011

 

 

 

16:24:40.113111 IP 10.11.57.2.58881 > 10.30.42.98.11011: S 515597348:515597348(0) win 5840 out slot1/tmm2 lis=

 

16:24:40.113508 IP 10.30.42.98.11011 > 10.11.57.2.58881: S 3804375753:3804375753(0) ack 515597349 win 5792 in slot1/tmm2 lis=

 

16:24:40.114328 IP 10.11.57.2.58881 > 10.30.42.98.11011: . ack 1 win 46 out slot1/tmm2 lis=

 

16:24:40.114550 IP 10.11.57.2.58881 > 10.30.42.98.11011: P 1:74(73) ack 1 win 46 out slot1/tmm2 lis=

 

16:24:40.114986 IP 10.30.42.98.11011 > 10.11.57.2.58881: . ack 74 win 46 in slot1/tmm2 lis=

 

16:24:40.115207 IP 10.30.42.98.11011 > 10.11.57.2.58881: F 1:1(0) ack 74 win 46 in slot1/tmm2 lis=

 

16:24:40.115375 IP 10.30.42.98.11011 > 10.11.57.2.58881: R 2:2(0) ack 74 win 46 in slot1/tmm2 lis=

 

 

==> my perl script dies after this !!! I will have to check the TLS options

 

 

Received TRANSACTION @ 4ea6c6a8 (0) (r=0/c=0/p=1)!

 

MCPSerializer::serialize - operation 2908 (modify); object 2122 (db_variable)

 

MCPSerializer::addData(), db_variable_name : string

 

MCPSerializer::addData(), db_variable_value : string

 

MCPSerializer::addData(), db_variable_object_id : ulong

 

MCPSerializer::addData(), db_variable_transaction_id : ulong

 

adding event to consumer DF99021F-1974-742B-C185-77067226AA6

 

adding event to consumer 2AA91888-1974-9995-C03A-461AB90526A

 

timerExpired

 

Saving Consumer list

 

Processing consumer: 'DF99021F-1974-742B-C185-77067226AA6' + objects: 1; time (t0,t1,delta): (1319552680, 1319552680, 0)

 

timerExpired

 

startSweepTimer: 4ea6c6a8 -> 4ea6c6b8

 

timerExpired

 

Saving Consumer list

 

Processing consumer: 'DF99021F-1974-742B-C185-77067226AA6' + objects: 1; time (t0,t1,delta): (1319552680, 1319552680, 0)

 

timerExpired

 

startSweepTimer: 4ea6c6a8 -> 4ea6c6b8

 

timerExpired

 

Saving Consumer list

 

timerExpired

 

Processing consumer: '2AA91888-1974-9995-C03A-461AB90526A' + objects: 2; time (t0,t1,delta): (1319552680, 1319552695, 15)

 

timerExpired

 

 

==> my perl died, so TCP Reset.

 

 

16:24:55.999344 IP 10.11.57.2.60965 > 10.30.42.98.11011: S 700834715:700834715(0) win 5840 out slot1/tmm2 lis=

 

16:24:55.999798 IP 10.30.42.98.11011 > 10.11.57.2.60965: R 0:0(0) ack 700834716 win 0 in slot1/tmm2 lis=

 

 

timerExpired

 

startSweepTimer: 4ea6c6b8 -> 4ea6c6c8

 

Signal 2 received

 

Destroying Consumer DF99021F-1974-742B-C185-77067226AA6

 

Destroying Consumer 2AA91888-1974-9995-C03A-461AB90526A

 

 

BTW: how can I paste formatted text into the editor window? It denied it with an error: "Your browser does not support pasting rich content". Browser: Firefox.

 

 

Thank you.

 

 

Regards

 

Kurt

UPDATE: I've done some further checks. Apparently the perl listener is part of the problem. Sometimes it dies directly after the SSL CLIENT HELLO (coming from the LB), however only if I start the SOAP Server with the option TLSv1, which is strange, as eventd offers TLSv1 anyway !?!? Seems to be a bug in the perl IO library.

 

 

So, one part of the problem is the perl script.

 

 

However, in debug mode I was unable to reproduce the second problem ("stuck" eventd). It will now allways retry to connect to the server, maybe due to the FIN/RST of the TCP connection after the script died !??! I don't get it. There's something different in debug mode.

 

 

Anyway, I will simply not start the SOAP server with the option TLSv1, as eventd is using that anyway and I will restrict access to my listener port by using TCP wrapper or iptables.

 

 

So: problem understood (partly) and the workaround is acceptable :-)

 

 

Regards

 

Kurt