Failure creating certificate acme challenge 404 error in BIG-IP F5 WAF
We have more than 600 government websites behind the BIG-IP system. We have done almost 60% of certificates created and offloaded.Suddenly we couldn't create any certificate and got the below error. This error not only for one website. Now we can't renew or create a new certificate. We use fanceg/letsencrypt -in GitHub to integrates Let's Encrypt with BigIP (GitHub - fanceg/letsencrypt-bigip). INFO: Using main config file /etc/dehydrated/configProcessing verugal.ds.gov.lk Signing domains... Generating private key... Generating signing request... Requesting new certificate order from CA... Received 1 authorizations URLs from the CA Handling authorization for verugal.ds.gov.lk 1 pending challenge(s) Deploying challenge tokens... Responding to challenge for verugal.ds.gov.lk authorization... Cleaning challenge tokens... Challenge validation has failed : ( ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01" ["status"] "invalid" ["error","type"] "urn:ietf:params:acme:error:unauthorized" ["error","detail"] "Invalid response fromhttp://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE1[43.224.124.166]: 404" ["error","status"] 403 ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Invalid response fromhttp://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE1[43.224.124.166]: 404","status":403} ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/12995442889/eoq1dQ" ["token"] "CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE" ["validationRecord",0,"url"] "http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE1" ["validationRecord",0,"hostname"] "verugal.ds.gov.lk" ["validationRecord",0,"port"] "80" ["validationRecord",0,"addressesResolved",0] "43.224.124.166" ["validationRecord",0,"addressesResolved"] ["43.224.124.166"] ["validationRecord",0,"addressUsed"] "43.224.124.166" ["validationRecord",0] {"url":"http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE","hostname":"verugal.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"} ["validationRecord"] [{"url":"http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE","hostname":"verugal.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"}] ["validated"] "2021-05-10T04:46:36Z") Processing vrc.bopepoddala.ds.gov.lk Signing domains... Generating private key... Generating signing request... Requesting new certificate order from CA... Received 1 authorizations URLs from the CA Handling authorization for vrc.bopepoddala.ds.gov.lk 1 pending challenge(s) Deploying challenge tokens... Responding to challenge for vrc.bopepoddala.ds.gov.lk authorization... Cleaning challenge tokens... Challenge validation has failed : ( ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01" ["status"] "invalid" ["error","type"] "urn:ietf:params:acme:error:unauthorized" ["error","detail"] "Invalid response fromhttp://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y[43.224.124.166]: 404" ["error","status"] 403 ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Invalid response fromhttp://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y[43.224.124.166]: 404","status":403} ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/12995448812/pq_1KA" ["token"] "v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y" ["validationRecord",0,"url"] "http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y" ["validationRecord",0,"hostname"] "vrc.bopepoddala.ds.gov.lk" ["validationRecord",0,"port"] "80" ["validationRecord",0,"addressesResolved",0] "43.224.124.166" ["validationRecord",0,"addressesResolved"] ["43.224.124.166"] ["validationRecord",0,"addressUsed"] "43.224.124.166" ["validationRecord",0] {"url":"http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y","hostname":"vrc.bopepoddala.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"} ["validationRecord"] [{"url":"http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y","hostname":"vrc.bopepoddala.ds.gov.lk","port":"80","addressesResolved":["43.224.124.166"],"addressUsed":"43.224.124.166"}] ["validated"] "2021-05-10T04:46:58Z") Can anyone help me out with this issue? Are there any process changes or updates in letsencrypt site or BIG-IP intigrations? Due to this lots of government websites affected!