Forum Discussion

brad9iner_11512's avatar
brad9iner_11512
Icon for Altostratus rankAltostratus
Mar 20, 2019

Error processing HTTP request header

I'm running into an issue on my apache tomcat 8 servers when receiving traffic routed through the F5. We have enabled TLS on the application servers and have setup an SSL Server profile to send the traffic through over HTTPS. The pool is configured to send the traffic to the servers on ports 8443 and 8444. However we are getting the following error from the servers.

2019-03-20 05:26:32,938+0000 INFO org.apache.coyote.http11.Http11Processor service - Error  parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:462)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

I am not certain what to look at on the F5 for how it is formatting/encrypting the traffic before it sends it onto the servers. Any advice on the direction I should take this would be very much appreciated.

application delivery
BIG-IP
cloud
LTM

9 Replies

Sort By
  • DennisJann_1272's avatar
    DennisJann_1272
    Icon for Nimbostratus rankNimbostratus
    Mar 21, 2019

    It looks like this error occurs when you send an HTTPS request to a Tomcat server running an HTTP listener.

    I got a similar error message from a sample Tomcat Docker image (

    docker run -it --rm -p 8888:8080 tomcat:alpine
    ) when I sent it an HTTPS request (
    curl -Ivk https://localhost:8888/
    ).

    21-Mar-2019 00:36:25.559 INFO [http-nio-8080-exec-8] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
 java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:428)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

    You could confirm this by removing the serverssl profile from the VIP in question.

    You could also try connecting to ports 8443 and 8444 on your Tomcat hosts to verify that they are in fact configured to accept HTTPS connections.

    If neither of these resolves the issue, you could try sharing a sanitized version of your VIP configuration, along with confirmation that the Tomcat servers are accepting HTTPS connections.

    • brad9iner_11512's avatar
      brad9iner_11512
      Icon for Altostratus rankAltostratus
      Mar 21, 2019

      Here is the config we have setup on our tomcat servers:

      
`
      




  
  
  




      `

      Which is the same execpt 8444 for our other JVM. We have 2 nodes in this cluster with the same configs. I don't have direct access to the F5 our VIP is configured on. But I had my resource do some testing. When they disabled 8444 in the pool in question for both nodes we are able to get successful responses. But when they disable 8443 and leave 8444 enabled then that is when we start getting "empty reply from server" messages as well as the stack trace in the description.

    • brad9iner_11512's avatar
      brad9iner_11512
      Icon for Altostratus rankAltostratus
      Mar 21, 2019

      new update on this, looking further into the F5 with my resource that has access we do have an iRule applied

      when SERVER_CONNECTED {
    if {[LB::server port] != 8443 && [LB::server port] != 443} {
        SSL::disable
    }
   }

      What would I need to edit on that to make it also look at ports 8444 and 8445? I can't quite find the "or" equivalent for this kind of iRule.

    • DennisJann_1272's avatar
      DennisJann_1272
      Icon for Nimbostratus rankNimbostratus
      Mar 21, 2019

      Given your existing iRule, you would enclose the "or" conditions in ()'s like this:

      when SERVER_CONNECTED {
   if { ([LB::server port] != 8443 or [LB::server port] != 8444 or [LB::server port] != 8445) && [LB::server port] != 443 } {
      SSL::disable
   }
}

      I don't know your specific requirement, but I've generally used SSL::disable in the HTTP_REQUEST event like this:

      when HTTP_REQUEST {
   if { [HTTP::path] starts_with "/plaintext-path" } {
      SSL::disable
      pool http_pool
   } else {
      pool https_pool
   }
}

      Hope this helps.

  • DennisJann's avatar
    DennisJann
    Icon for Nimbostratus rankNimbostratus
    Mar 21, 2019

    It looks like this error occurs when you send an HTTPS request to a Tomcat server running an HTTP listener.

    I got a similar error message from a sample Tomcat Docker image (

    docker run -it --rm -p 8888:8080 tomcat:alpine
    ) when I sent it an HTTPS request (
    curl -Ivk https://localhost:8888/
    ).

    21-Mar-2019 00:36:25.559 INFO [http-nio-8080-exec-8] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
 java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:428)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

    You could confirm this by removing the serverssl profile from the VIP in question.

    You could also try connecting to ports 8443 and 8444 on your Tomcat hosts to verify that they are in fact configured to accept HTTPS connections.

    If neither of these resolves the issue, you could try sharing a sanitized version of your VIP configuration, along with confirmation that the Tomcat servers are accepting HTTPS connections.

    • brad9iner_11512's avatar
      brad9iner_11512
      Icon for Altostratus rankAltostratus
      Mar 21, 2019

      Here is the config we have setup on our tomcat servers:

      
`
      




  
  
  




      `

      Which is the same execpt 8444 for our other JVM. We have 2 nodes in this cluster with the same configs. I don't have direct access to the F5 our VIP is configured on. But I had my resource do some testing. When they disabled 8444 in the pool in question for both nodes we are able to get successful responses. But when they disable 8443 and leave 8444 enabled then that is when we start getting "empty reply from server" messages as well as the stack trace in the description.

    • brad9iner_11512's avatar
      brad9iner_11512
      Icon for Altostratus rankAltostratus
      Mar 21, 2019

      new update on this, looking further into the F5 with my resource that has access we do have an iRule applied

      when SERVER_CONNECTED {
    if {[LB::server port] != 8443 && [LB::server port] != 443} {
        SSL::disable
    }
   }

      What would I need to edit on that to make it also look at ports 8444 and 8445? I can't quite find the "or" equivalent for this kind of iRule.

    • DennisJann's avatar
      DennisJann
      Icon for Nimbostratus rankNimbostratus
      Mar 21, 2019

      Given your existing iRule, you would enclose the "or" conditions in ()'s like this:

      when SERVER_CONNECTED {
   if { ([LB::server port] != 8443 or [LB::server port] != 8444 or [LB::server port] != 8445) && [LB::server port] != 443 } {
      SSL::disable
   }
}

      I don't know your specific requirement, but I've generally used SSL::disable in the HTTP_REQUEST event like this:

      when HTTP_REQUEST {
   if { [HTTP::path] starts_with "/plaintext-path" } {
      SSL::disable
      pool http_pool
   } else {
      pool https_pool
   }
}

      Hope this helps.