Forum Discussion

Lerna_Ekmekciog's avatar
Lerna_Ekmekciog
Icon for Nimbostratus rankNimbostratus
Sep 15, 2006

iControl Server Thread Safe ?

I am load testing my custom SOAP service which is a java app that sends SOAP queries to iControl SOAP server. As part of this test I send concurrent requests to iControl server. An example of this is sending n concurrent requests for getting a pool and all its fields, meaning lb method, monitor, member, from the device. However I only get one response from iControl server with the pool and all its fields correctly populated in the SOAP message. All the rest of responses contain fields with no values. That is n-1 empty responses.

 

 

When I do ssldump on the communication between the server and my app I can see that the server only sees one request to get lb method and one request to get monitor association as opposed to n requests where n is the number of concurrent requests.

 

 

I'm trying to figure out where these n requests are being multiplexed.

 

 

Is the iControl SOAP server thread safe ?

 

 

Lerna

20 Replies

  • Joe, I understood what you were saying. But if your only benchmark of failure/success is the aftermath of 100 disable requests, you might not see the error (You can never guarantee that all 100 threads will hit the webservice at the exact same millisecond -- this is very improbable). So the end result will be a pool member which is disabled, and you will think that everything worked well.

     

     

    But this is your show, so proceed anyway you think is right.

     

     

    Vladimir
  • I guess I'm still confused. Each thread will issue 100s of sequential requests to different pool members (thread 1: 10.10.1.1-100, thread 2 10.10.2.1-100, etc). If I have 100 parallel threads each issuing out sequential requests to different pool members wouldn't you eventually have one overlap of calls where more than one thread is making a call in the same millisecond?

     

     

    And, it's not really my show since you brought up the issue.

     

     

    BTW, I got a rough app built that will spin up a pool with 10000 members, spin up 100 threads each sequentially disabling different pool members. At the end of the loop, all 10000 pool members were disabled. Maybe I'm totally missing something. Next step will be to put out timer with each request to determine when the requests go out.

     

     

    Another factor here could be that I'm running on v10.1 and this could be a platform specific problem. When I get my app flushed out, I'll pass the source along to you to test out.

     

     

    -Joe
  • Just to be clear, it works like this

     

     

    Create 10000 pool members (10.10.1-100,1-100)

     

     

    Thread 1

     

    Disable members 10.10.1.1-100 in sequence one-at-a-time. This will make 100 iControl calls, once for each different member

     

    Thread 2

     

    Disable members 10.10.2.1-100 in sequence.

     

    ...

     

    Thread 100

     

    Disable members 10.10.100.1-100 in sequence.

     

     

    This will result in 10000 iControl calls.

     

     

    Then the app will query the state for all 10000 members and if any of their states are not disabled then something broke.

     

     

    -Joe
  • I'm very curious as to the use case here. What is the benefit of issuing the same call against the same system from multiple threads? Given that it's a management service it is risky to ask it to support concurrency like this. Also, don't we have this already built right into the API? As Joe mentions above you can pass a list of 100 systems into the disable call and iControl will honor that.

     

     

    Sorry to derail the thread but I must be missing the design goal here and I'd like to understand it.

     

     

    -Matt
  • I developed an application that allows the user to toggle the state of a pool member (or pool members).

     

     

    The use case: a pool might have 10 pool members, and 9 or the 10 are enabled. Now, I want to enable the 10th pool member, but before I can do that, I need to disable the other 9 (this is one of our internal requirements). The call to disable the 9 pool members was handled by about 5-8 threads. What I was noticing is that when 2 of the threads sent the soap requests to the F5 at the same exact time (same millisecond), only one succeeded, and the other one was simply ignored (no exceptions, no changes).
  • Interesting, thanks for the explanation. Even still, it sounds like you really only need two calls here, with just two threads:

     

     

    1) Use one thread to call out to disable the 9 servers in the pool, passing in the list of all 9 pool members to disable_member (or whatever other call you want)

     

    2) Fire the second off immediately to enable member 10.

     

     

    Will this method not work here?

     

     

    -Matt
  • Unfortunately two calls won't work here due to the event-based assynchronous rpc calls that I have setup on the front-end.

     

     

    Again, I worked around the problem by synchronizing the methods that make the actual disable/enable calls, so I'm not experiencing any issues anymore. But this is an interesting issue for you guys to look into since this is an unexpected behavior.

     

     

    Thanks though!

     

    Vladimir
  • So, I ran some tests on 10000 pool members and spun up 100 threads each sequentially disabling 100 pool members that were previously enabled. After the calls were made, I looped through all of them to make sure that the status took and every single method call worked. Glad you have a solution and also glad that this is the first time I've heard this issue in the 9 years since we introduced iControl. Now, if I could only build an app that could reproduce the problem...

     

     

    -Joe