kubernetes network policies, how to secure the communications between your pods

Update on October 17th, 2019, this blog article has been promoted and published on the official Microsoft’s Open Source blog there: Tutorial: Calico Network Policies with Azure Kubernetes Service.

On May 2019, Network Policies on AKS was announced GA:

A user-defined network policy feature in AKS enables secure network segmentation within Kubernetes. This feature also allows cluster operators to control which pods can communicate with each other and with resources outside the cluster. Network policy is generally available through the Azure native policy plug-in or through the community project Calico.

I encourage you to give a read of this article too: Integrating Azure CNI and Calico: A technical deep dive where you will see all the concepts explained on a Networking perspective with AKS. Furthermore here is the Kubernetes tutorial, the Calico tutorial and the AKS tutorial you could give a try to practice with those concepts.

Some gotchas here:

• By default, any pods could communicate with any other pods across namespaces within a Kubernetes cluster, it’s by design.
  • But Kubernetes Network Policies will guarantee the “Just Enough Access” principle of your Security posture
• By default, there is no default plugin pre-installed with Kubernetes to actually apply such Network Policies
  • You need to install this plugin, otherwise your Network Policies won’t have any effect.
• With AKS, you have the option between azure or calico as your Network Policy plugin
  • You could only define this at the cluster creation, update is not yet supported
  • With calico you could either use kubenet or azure as your CNI but for Azure CNI it’s only with azure (not kubenet) as Network Policy plugin.
  • It’s not yet supported for Windows nodes
• Both azure and calico Network Policy plugin are open source:
  • azure: https://github.com/Azure/azure-container-networking 
  • calico: https://github.com/projectcalico/calico
    • AKS currently, as we speak, supports Calico version 3.5.0

Because I love practicing while learning, here is a scenario where I was able to build 4 different Network Policies for this setup below:

• WEB is exposed and accessible publicly from the Internet
• WEB talks to API
• API talks to DB
• Not any other exposures nor communications, just this.

Let’s have a look to share with you my learnings there!

First we need to provision a cluster with Network Policies enabled, in my case I will go ahead with Calico:

az aks create \
    --network-policy calico

To start and illustrate this we need to deploy those Pods:

ns=yournamespace
kubectl create ns $ns
kubectl config set-context --current --namespace $ns
kubectl apply \
    -f https://raw.githubusercontent.com/mathieu-benoit/k8s-netpol/master/db-api-web-deployments.yaml

You now have 3 Pods and 3 Services:

kubectl get pod,svc -n $ns

We could check that WEB is publicly accessible:

curl $(kubectl get svc web -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Our first test is to see that any pods could communicate with each others even externally, let’s run few successful commands:

kubectl run curl-$RANDOM \
    --image=radial/busyboxplus:curl \
    --rm \
    -it \
    --generator=run-pod/v1
# curl www.microsoft.com
# curl http://db:15984
# exit

Let’s apply the first Network Policy that should be for any namespace, Deny all ingress and egress!

kubectl apply \
    -f https://raw.githubusercontent.com/mathieu-benoit/k8s-netpol/master/deny-all-netpol.yaml

We could check that WEB isn’t anymore publicly accessible:

curl --connect-timeout 2 $(kubectl get svc web -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Let’s also re-run the two previous tests which should fail now:

kubectl run curl-$RANDOM \
    --image=radial/busyboxplus:curl \
    --rm \
    -it \
    --generator=run-pod/v1
# curl --connect-timeout 2 www.microsoft.com
# curl --connect-timeout 2 http://db:15984
# exit

Actually no one could communicate from/to that namespace at this stage, that’s what we want. Now let’s be more granular and illustrate the “Least Access” and “Just Enough Access” Security Principles.

First, we want DB be accessible only from API on port 5984 and doesn’t have access to anything:

kubectl apply \
    -f https://raw.githubusercontent.com/mathieu-benoit/k8s-netpol/master/db-netpol.yaml

Let’s validate that DB doesn’t have access to anything:

kubectl run curl-$RANDOM \
    --image=radial/busyboxplus:curl \
    --labels app=db \
    --rm \
    -it \
    --generator=run-pod/v1 \
    -n $ns
# curl --connect-timeout 2 http://web:80
# curl --connect-timeout 2 www.microsoft.com
# exit

We now want API having access only to DB on port 5984 and be accessible only from WEB on port 8080:

kubectl apply \
    -f https://raw.githubusercontent.com/mathieu-benoit/k8s-netpol/master/api-netpol.yaml

Actually we need also to do an extra action here by adding a Label on the kube-system Namespace (NetworkPolicies are all about Labels ;)):

kubectl label ns kube-system name=kube-system

Let’s validate that API has access to DB but doesn’t have access to WEB nor Internet:

kubectl run curl-$RANDOM \
    --image=radial/busyboxplus:curl \
    --labels app=api \
    --rm \
    -it \
    --generator=run-pod/v1

# curl http://db:15984
# curl --connect-timeout 2 http://web:80
# curl --connect-timeout 2 www.microsoft.com
# exit

And finally we want WEB having access only to API on port 3000 and be accessible only from Internet on port 80:

kubectl apply \
    -f https://raw.githubusercontent.com/mathieu-benoit/k8s-netpol/master/web-netpol.yaml

Let’s validate that WEB has access to API but doesn’t have access to DB or Internet:

kubectl run curl-$RANDOM \
    --image=radial/busyboxplus:curl \
    --labels app=web \
    --rm \
    -it \
    --generator=run-pod/v1
# curl http://api:8080
# curl --connect-timeout 2 www.microsoft.com
# curl --connect-timeout 2 http://db:15984
# exit

We could check that WEB is publicly accessible again:

curl $(kubectl get svc web -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Here you are! We have secured communications for our 3 Pods: WEB, API and DB by defining the very strict minimal requirements on that regard, nothing less and nothing more.

Some gotchas:

• It’s all about Labels on Pods and Namespaces
• It’s not about Services nor the ports exposed there, it’s about Pods’ ports
• You could use podSelector and namespaceSelector
• In one NetworkPolicy, you could combine multiple to: and multiple from:, therefore they will be applied as AND rules
• Again, the scope is per Namespace,  but via the namespaceSelector for Ingress or Egress you could reference external Namespaces
  • You could use GlobalNetworkPolicy with Calico to apply rules across Namespaces
• To be able to reach out to another Pod via its Service name exposure you need to add an Egress rule for the DNS resolver (with the label k8s-app=kube-dns) in the kube-system Namespace. We saw that we need to add a label name=kube-system on the kube-system Namespace.
• Network Policy Engine is doing the union of all the rules, Rule1 OR Rule2 OR…
• Default rules are for Ingress, as soon as you are adding Egress you need to specify this in the policyTypes: section

Resources:

• Secure traffic between pods using network policies in Azure Kubernetes Service (AKS) | Azure Friday
• Securing Cluster Networking with Network Policies - Ahmet Balkan, Google
  • https://github.com/ahmetb/kubernetes-network-policy-recipes
• An Introduction to Kubernetes Network Policies for Security People
• Kubernetes Network Policies Viewer

Hope you enjoyed this blog article and this learning process and hope you will be able to leverage this for your own context and needs.

Cheers!