Wow, the IT security world was on fire. LinkedIn and Twitter feeds were almost just talking about that: The commonly used Apache Log4j 2 library has been compromised. 5 CVEs have been reported to date: CVE-2021-44228, CVE-2021-45046, CVE-2021-45105, CVE-2021-4104 and CVE-2021-44832.

Google has been providing their findings of ongoing investigations for both Google and Google Cloud products and services:

The Google Cybersecurity Action Team provides recommendations and solutions available to Google Cloud customers and security teams to manage the risk of the Apache Log4j 2 vulnerability. You will find in there concrete examples with Cloud Armor, Chronicle, Cloud IDS, Security Command Center, Container scanning, Binary Authorization, Cloud Logging and Apigee.

Impacted or not by those current events, there is some lessons learned that we could leverage to make sure that whatever languages used by the developers, there is tools in place to detect, protect and remediate such compromised dependencies. In the context of containers and Kubernetes, here are 4 easy ways to help with this:

Update dependencies

The second section of the recent article The past, present, and future of Kubernetes with Eric Brewer highlights a very important security aspect regarding to the dependencies of applications:

As the number of dependencies used in software development grows, the security risks multiply. Investing in software supply chain security is imperative, and a move towards managed services is actually safer than self-managed solutions.

99% of our vulnerabilities are not in the code you write in your application. They’re in a very deep tree of dependencies, some of which you may know about, some of which you may not know about.

If you are using GitHub, you could leverage GitHub’s security features like dependabot and CodeQL to help identify exposure in your codebase.

As an example, you could leverage dependabot on any GitHub repository just by creating this .github/dependabot.yml file to check updates on both your containers base images in your Dockerfile as well as your gradle packages (here is the list of all available packages ecosystem):

cat <<EOF > .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "daily"
  - package-ecosystem: "gradle"
    directory: "/"
    schedule:
      interval: "daily"
EOF

You could read and learn more here about the impact of Apache Log4j 2 vulnerabilities and the packages depending on it. In addition to understand how widespread are those vulnerabilities, you will also learn more about the initiatives such as Secure Open Source Rewards and Open Source Insights driven by Google.

Scan containers

Scanning your containers as early as possible, at least from within your Continuous Integration pipelines to block any critical vulnerabilities, is also very key to secure and trust your software supply chain.

If you scan a Java container having those Log4j 2 vulnerabilities by running this command gcloud artifacts docker images scan --additional-package-types=MAVEN, you will be able to get the following information:

CRITICAL            9.3         projects/goog-vulnz/notes/CVE-2021-44228    org.apache.logging.log4j:log4j-api    2.13.3        2.15.0
CRITICAL            5.1         projects/goog-vulnz/notes/CVE-2021-45046    org.apache.logging.log4j:log4j-api    2.13.3        2.16.0
HIGH                4.3         projects/goog-vulnz/notes/CVE-2021-45105    org.apache.logging.log4j:log4j-api    2.13.3        2.17.0
MEDIUM              6.0         projects/goog-vulnz/notes/CVE-2021-44832    org.apache.logging.log4j:log4j-api    2.13.3        2.17.1

In addition to that, you could use Binary Authorization by leveraging the vulnerability-based attestations feature too.

Another important aspect is to scan your containers actually running on your Kubernetes clusters, for example with GKE you could leverage the Container Thread Detection feature from Security Command Center. A new Active Scan: Log4j Vulnerable to RCE rule has been launched too.

Protect Ingress

Web Application Firewalls (WAF) are often the first option for protecting and mitigating vulnerabilities that require HTTP transport. Google Cloud Armor helps protect your applications and websites against denial of service (DDOS) and web attacks (WAF).

Google Cloud Armor customers can now deploy a new preconfigured WAF rule that will help detect and, optionally, block attempted exploits of CVE-2021-44228 and CVE-2021-45046:

gcloud compute security-policies rules create 12345 \
    --security-policy $securityPolicyName \
    --expression "evaluatePreconfiguredExpr('cve-canary')" \
    --action "deny-403" \
    --description "CVE-2021-44228 and CVE-2021-45046"

From there, you could look at the denied logs generated by this rule via Cloud Logging:

filter="resource.type=\"http_load_balancer\" "\
"jsonPayload.enforcedSecurityPolicy.name=\"${securityPolicyName}\" "\
"httpRequest.requestUrl=~\"jndi\" "\
"OR httpRequest.userAgent=~\"jndi\" "\
"OR httpRequest.referer=~\"jndi\""
gcloud logging read \
    --project $projectId \
    "$filter" \
    --format='table(httpRequest.requestUrl)'

Restrict Egress

Because the compromised Apache Log4j 2 fetches malicious variables over the network, filtering your workload’s egress connections is key to mitigate and protect your entire platform. To accomplish a defense in depth security setup, there is three layers of features you could leverage: Private GKE cluster with NAT Gateway, Kubernetes Network Policies and Istio Egress Gateway.

First, Private GKE clusters will guarantee that your nodes don’t have any public IPs which will avoid any egress communications. To define outbound internet access for certain private nodes you can use Cloud NAT.

Second, Kubernetes Network Policies will allow you to restrict egress from your applications. For example, the following example illustrates how to deny all egress in the namespace your-namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: your-namespace
spec:
  podSelector: {}
  policyTypes:
  - Egress

With GKE Dataplane V2 you could even leverage the logging capabilities to get the deny logs, I illustrated this in my blog ebpf and cilium, to bring more security and more networking capabilities in gke.

Note: if you want to set DNS based policies, you will need to leverage Cilium Network Policy or the FQDNNetworkPolicies.

Third, Egress Gateway with your service mesh, directing all the egress traffic through it, and allocating public IPs to the egress gateway nodes allows the application nodes to access external services in a controlled way. The most important security aspect for a service mesh is probably ingress traffic like we saw earlier in this blog article. You definitely must prevent attackers from penetrating the cluster through ingress APIs. Having said that, securing the traffic leaving the mesh is also very important. Once your cluster is compromised, and you must be prepared for that scenario, you want to reduce the damage as much as possible and prevent the attackers from using the cluster for further attacks on external services and legacy systems outside of the cluster. To achieve that goal, you need secure control of egress traffic.

Looking for a concrete implementation with the 3 of them together with GKE? Here is the high-level guide and its tutorial companion.

Complementary and further resources

Hope you enjoyed that one to improve your security posture, sail safe out there!