Forum Discussion

CraigM_17826's avatar
CraigM_17826
Icon for Altostratus rankAltostratus
Aug 04, 2009

Web managment interface drops connection after 36hrs or so

Hi everyone,

 

 

just wondering if anyone else has run into this issue or not. Due to our BIGIPs crashing randomly (due to what we now appears to be a possible memory leak issue) we decided to leave a web session to the unit's active, displaying the performance stats, specifically ram usage. The PC running the session to the two units displays on a central 52" LCD monitor so everyone in support can see the status.

 

 

What is happening is that after 36hrs or so the connection to one of the unit is lost. If I attempt a refresh it just states that an internal error occured and to check the logs. If I close the browser and reconnect it is fine.

 

 

I've looked at the logs on the offending unit and in the webui.log I found this

 

 

2009-08-04 15:25:12,255 ERROR [Thread-21] controller.F5WebController:processWebResponse - Exception thrown in processWebResponse: java.lang.NullPointerException

 

java.lang.NullPointerException

 

at web.common.formredirect_jsp._jspService(formredirect_jsp.java:123)

 

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

 

at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)

 

at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:432)

 

at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:356)

 

at com.f5.controller.F5WebController.processWebResponse(F5WebController.java:157)

 

at com.f5.controller.F5WebController.processWebRequest(F5WebController.java:123)

 

at com.f5.controller.ControlServlet.doGet(ControlServlet.java:172)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

 

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)

 

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)

 

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)

 

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

 

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

 

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

 

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

 

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

 

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

 

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

 

at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)

 

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)

 

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

 

at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)

 

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)

 

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)

 

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)

 

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

 

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

 

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)

 

at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)

 

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)

 

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)

 

at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)

 

at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261)

 

at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360)

 

at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:604)

 

at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:562)

 

at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:679)

 

at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)

 

at java.lang.Thread.run(Unknown Source)

 

 

Going by the date/time of this error, this occured when I tried to refresh the page.

 

 

Now this does not happen on the 2nd BigIP unit so it seems odd the first unit is doing this. The 1st unit is in standby mode.

 

 

Has anyone else had this happen?

 

 

We are running the latest 9.4 build of the firmare on both units.

 

 

tia

 

 

Craig

2 Replies

  • I expect F5 Support will be your best option for troubleshooting this as it would help to see the full logs/metrics in a tech.out when the failure occurs.

     

     

    Aaron
  • my two cents. * Caution: consult with support before making changes.

     

     

    We have seen very significant benefit by increasing the allocated host memory. So much that 3 years back, we pushed for a DB change to be included in new software versions instead of the old grub based change. The default linux alloc is way too low. It however depends on what platform and software version you are running. If you are running redundant 3600's for eg, you can allocate like 2Gigs to your host. Very very simple change.

     

     

    ** AGAIN CAUTION: Check with support. This is just for information

     

     

    9.4.6

     

     

    > do "b memory show all " and see reserved host vs tmm if you have the default ~300M

     

    > bigpipe db Provision.ExtraMB

     

    ( should show 0)

     

    > bigpipe db Provision.ExtraMB (whatever you want - current)

     

    > grubmem

     

     

    for older releases

     

     

    grub_open /mnt/tmp/install/boot/hda/grub/grub.conf

     

    vi /mnt/tmp/install/boot/hda/grub/grub.conf

     

    grub_close

     

    Note: You must not be in the /tmp parent directory when you run the grub_close

     

    command; the command will fail because the mounted file system is still busy.

     

    edit tmm

     

    /etc/bigstart/scripts/tmm