Trust your CDN, but not completely
A content delivery network, or content distribution network (CDN), is a geographically distributed network of reverse proxy servers. The aim is to provide high availability and performance by distributing the service spatially relative to client browsers. According to Gartner, some of the top players in this space are Fastly, Akamai, Cloudflare and of course the hyperscalers all have an offering too.
The basic idea when deploying a CDN is to place as much of your application as possible into the edge platform provided by the CDN, while dynamic data that is often database driven would remain on an origin server. Many CDNs do much more, including caching, security, etc.
CDN's are commonly available with security features such as:
- Web Application Firewall (WAF)
- DoS protection
- Anti-bot protection.
F5 also has CDN functions in our Distributed Cloud via the acquisition of CDN provider Lilac, you can read more about the CDN features here.
In this article, we will explore why it is necessary to consider attacks vectors via the CDN in your threat model, and protect your origin servers against such attacks. Such a defense-in-depth approach will not be new to most security engineers.
Threat hunting & analysis
F5 Advanced WAF (AWAF) has rich logging and reporting that is built-in to the web UI. There are two handy places for engineers to see a summary of what is happening from a security perspective; the Event Correlation and Bot Traffic dashboards. Event Correlation in particular lets security engineers easily surface unusual traffic, without being overwhelmed by the sheer volume of security events that can be present on a busy WAF.
This makes the Event Correlation feature a great place to start when hunting for threats in your environment - this is where I first became aware of an unusual attack in my own test environment that was sourced via a very popular CDN/DoS provider (that shall remain nameless).
As mentioned earlier in this article, CDN functions are very commonly deployed in many enterprises. But it is unusual to see malicious traffic initiated from a CDN server to random Internet destinations - as they often have integrated security functions (& large SOC teams that are analysing outbound traffic from the CDN).
Lets analyse how F5 AWAF mitigated this attack.
Mitigation with IP Intelligence
IP Intelligence did not mitigate these CDN sourced attacks, because the source IPs were not captured in any dynamic blocklist - this indicates that the CDN host was probably only recently being used for malicious outbound traffic. It is also very common for CDN provider ranges to be explicitly allowed through a firewall policy. Many CDN providers prescribe this to their customers:
https://techdocs.akamai.com/origin-ip-acl/docs/update-your-origin-server
https://developer.fastly.com/reference/api/utils/public-ip-list/
The concept of a trusted CDN range places security architects in an interesting position; should a firewall policy dynamically block traffic from the CDN itself? Even when doing so may result in legitimate traffic being blocked? Should engineers just rely on whatever security functions the CDN provider offers?
Clearly, when the CDN is delivering both good and bad traffic, layer 4 security constructs are not useful.
Mitigation with Bot Defense
When reviewing the requests coming from the CDN ranges, analysis of the timing and distribution would indicate that they are automated requests. The ~20 malicious requests from various CDN IPs all arrived within a 6 second window. However, the requests passed thru the Bot Defense profile. This is because they all arrived within the 30 second grace period for a client browser to complete the Javascript challenge.
If any more such malicious requests arrived post the 30 second grace period, Bot Defense would start rejecting them. The Bot Defense profile could also be modified to have a shorter grace period, and/or to require the challenge to be completed before processing requests. This is usually overkill in peacetime scenarios.
Mitigation with WAF policy
The last line of defense is the WAF policy, and this is where the requests were ultimately rejected.
When drilling further into the event correlation, you can see that the block reason was Threat Campaign Detected. Expanding this further shows that the requests all match a fairly old vulnerability in ThinkPHP.
GET /index.php?s=//think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=1f3870be274f6c49b3e31a0c6728957f HTTP/1.1
Host: mirage.init4.org
Connection: Keep-Alive
X-Forwarded-For: 2a0b:8bc0:1:20d::, 172.71.182.78
CF-RAY: 7c669a1f86731c81-AMS
X-Forwarded-Proto: http
CF-Visitor: {"scheme":"http"}
CF-EW-Via: 15
CDN-Loop: cloudflare; subreqs=1
Accept: */*
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
CF-Connecting-IP: 2a0b:8bc0:1:20d::
cf-worker: varakoc3312864.workers.devThis vulnerability is known as CVE-2018-20062. A security researcher at F5 did a pretty good write-up on the subject:
Evidently, ThinkPHP vulnerabilites continue to bear fruit for hackers, because they are still at it, years later:
https://www.akamai.com/blog/security/thinkphp-exploit-actively-exploited-in-the-wild
Of course, we have Threat Campaign signatures to match this ThinkPHP vulnerability (and thousands of others), and they mitigated this real-world attack perfectly. Looking at the requests, the attackers were obviously just performing recon to see if they could find any ThinkPHP targets. If you look at the request headers, you will see that these requests are actually being sourced via IPv4 and IPv6 addresses 185.213.174.190 & 2a0b:8bc0:1:20d::. No doubt this client had uncovered some kind of misconfiguration/vulnerability on the CDN hosts.
Conclusion
The malicious traffic observed via CDN in this scenario reinforces the need to have defense in depth - relying on the security features in a CDN is just as dangerous as relying on a single control point on a firewall, a DoS provider, an origin server or anything else. Organisations should have as many control points as is operationally feasible, and consider all potential attack vectors (including their own infrastructure!) when composing those control points.