Sail Sharp: .NET on Kubernetes

In February 2021, I got this opportunity to deliver this talk Sail Sharp, .NET Core & Kubernetes for the .NET Meetup in Quebec city (it was in French). I illustrated the best practices to prepare any .NET applications for Kubernetes. I was using the cartservice app (in dotnet) from the very popular Online Boutique sample apps.

Since then, I have been one of the top contributors to the Online Boutique repository. I contributed to the golang, python, dotnet, java and nodejs apps. I learned a lot. Some of my contributions, among others, were about optimizing and securing the container images for all these apps.

Here is the high-level timeline of my contributions related to the cartservice app:

As a side note, I started my journey with the .NET Framework version 3.0 (only on Windows at that time) back in 2006! Since then, I have been amazed about the evolution of the .NET ecosystem. And to be honest all these contributions gave me a reason to stay up-to-date and have a lot of fun while learning more about containers and Kubernetes! :)

Wow! Quite a ride, isn’t it?

Today, in this blog post, I will highlight 8 tips to optimize and secure your .NET containers based on what I have learned based on all of that:

  1. Multi-stage build
  2. Optimized bundled application
  3. Small base image
  4. Immutable base image
  5. Update dependencies
  6. .dockerignore
  7. Unprivilege/non-root container
  8. Read-only container filesystem

If you want to see the final Dockerfile and the Deployment manifest to deploy a secure and optimized .NET application in Kubernetes, feel free to directly jump to the end of this blog post.

Disclaimer: Whereas some of the concepts could be applicable to Windows containers, this blog post is only covering Linux containers. Furthermore, I’m not taking into account the multi-platform container support, I’m just supporting amd64 and not arm64 as an example.

Create a folder where we will drop all the files needed for this blog post:

mkdir my-sample-app

Create a minimal and simple ASP.NET app we will use for this blog post:

cat <<EOF > my-sample-app/Program.cs
using Microsoft.AspNetCore.Builder;
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/", () => "Hello, World!");
app.Run();
EOF
cat <<EOF > my-sample-app/my-sample-app.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
  </PropertyGroup>
</Project>
EOF

1. Multi-stage build

Create our first Dockerfile with a multi-stage build. That’s not the final one, until then, please bear with me:

cat <<EOF > my-sample-app/Dockerfile
FROM mcr.microsoft.com/dotnet/sdk:7.0 as builder
WORKDIR /app
COPY my-sample-app.csproj .
RUN dotnet restore my-sample-app.csproj -r linux-x64
COPY . .
RUN dotnet publish my-sample-app.csproj -r linux-x64 -c release -o /my-sample-app --no-restore

FROM mcr.microsoft.com/dotnet/runtime:7.0
WORKDIR /app
COPY --from=builder /my-sample-app .
ENTRYPOINT ["/app/my-sample-app"]
EOF

This Dockerfile uses multi-stage build, which optimizes the final size of the image by layering the build and leaving only required artifacts.

Let’s build this container image locally:

docker build -t my-sample-app my-sample-app/

We can see that the size of the container image is 288MB on disk locally.

You can locally run the container and test that it is working successfully:

docker run -d -p 80:80 my-sample-app
curl localhost:80

2. Optimized bundled application

When using dotnet publish, we can use different features to optimize the size of the bundled application:

With -p:TrimMode=full, the size of the container image is now 136MB on disk locally, but the container is then not working in my case because of this warning when doing dotnet publish:

warning IL2026: Using member 'Microsoft.AspNetCore.Builder.EndpointRouteBuilderExtensions.MapGet(IEndpointRouteBuilder, String, Delegate)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. This API may perform reflection on the supplied delegate and its parameters. These types may be trimmed if not directly referenced.

3. Small base image

To reduce the surface of attack or to avoid dealing with security vulnerabilities debt, using the smallest base image is a must.

You can find all the dotnet container images available here:

In my case, I choose to use the alpine one: dotnet/runtime-deps:7.0-alpine3.17. For that, we need to update the Dockerfile with -r linux-musl-x64 for both commands: dotnet restore and dotnet publish.

We can see that the size of the container image is now 43.2MB on disk locally. Impressive! Isn’t it?

Note: if this app was compatible with -p:TrimMode=full, the size of the container image would have been now 31.5MB on disk locally.

Below is the illustration of the sizes of the different base images:

SIZE
REPOSITORY                              TAG                                     SIZE
mcr.microsoft.com/dotnet/sdk            7.0.202                                 775MB
mcr.microsoft.com/dotnet/runtime-deps   7.0.4                                   117MB
mcr.microsoft.com/dotnet/runtime-deps   7.0.4-cbl-mariner2.0-distroless         26.4MB
mcr.microsoft.com/dotnet/runtime-deps   7.0.4-alpine3.17                        12.1MB
mcr.microsoft.com/dotnet/runtime-deps   8.0.0-preview.2-jammy-chiseled          13MB

Here, like you can see, I decided to take the smallest container image dotnet/runtime-deps:7.0.4-alpine3.17: 12.1MB.

But what about dotnet/runtime-deps:7.0.4-cbl-mariner2.0-distroless (26.4MB) and dotnet/runtime-deps:8.0.0-preview.2-jammy-chiseled (13MB)?

Good question, glad you asked!

They are very attractive because they are bringing the concept of distroless. They are container images that do not contain the complete or full-blown OS with system utilities installed. You can read more about CBL-Mariner 2.0 here, and more about Chiseled Ubuntu Containers here. Both are not yet ready for production.

Note: Chainguard is also working on having their own distroless images for dotnet. Something to keep in mind too!

This blog post Image sizes miss the point explains really well why the distroless ones are very attractive:

To reduce debt, reduce image complexity not size.

By using a tool like syft, we could see that the distroless ones are less complex than the alpine one, with less dependencies, reducing the debt and surface of risks. See results below.

For mcr.microsoft.com/dotnet/runtime-deps:7.0.4-cbl-mariner2.0-distroless:

[13 packages]
NAME                         VERSION               TYPE 
distroless-packages-minimal  0.1-3.cm2             rpm   
e2fsprogs-libs               1.46.5-3.cm2          rpm   
filesystem                   1.1-12.cm2            rpm   
glibc                        2.35-3.cm2            rpm   
krb5                         1.19.4-1.cm2          rpm   
libgcc                       11.2.0-4.cm2          rpm   
libstdc++                    11.2.0-4.cm2          rpm   
mariner-release              2.0-36.cm2            rpm   
openssl                      1.1.1k-21.cm2         rpm   
openssl-libs                 1.1.1k-21.cm2         rpm   
prebuilt-ca-certificates     2547388:2.0.0-10.cm2  rpm   
tzdata                       2022g-1.cm2           rpm   
zlib                         1.2.12-2.cm2          rpm

For mcr.microsoft.com/dotnet/runtime-deps:7.0.4-alpine3.17:

[25 packages]
NAME                    VERSION                TYPE   
alpine-baselayout       3.4.0-r0               apk     
alpine-baselayout-data  3.4.0-r0               apk     
alpine-keys             2.4-r1                 apk     
apk-tools               2.12.10-r1             apk     
busybox                 1.35.0                 binary  
busybox                 1.35.0-r29             apk     
busybox-binsh           1.35.0-r29             apk     
ca-certificates         20220614-r4            apk     
ca-certificates-bundle  20220614-r4            apk     
keyutils-libs           1.6.3-r1               apk     
krb5-conf               1.0-r2                 apk     
krb5-libs               1.20.1-r0              apk     
libc-utils              0.7.2-r3               apk     
libcom_err              1.46.6-r0              apk     
libcrypto3              3.0.8-r3               apk     
libgcc                  12.2.1_git20220924-r4  apk     
libintl                 0.21.1-r1              apk     
libssl3                 3.0.8-r3               apk     
libstdc++               12.2.1_git20220924-r4  apk     
libverto                0.3.2-r1               apk     
musl                    1.2.3-r4               apk     
musl-utils              1.2.3-r4               apk     
scanelf                 1.3.5-r1               apk     
ssl_client              1.35.0-r29             apk     
zlib                    1.2.13-r0              apk

Note: very important note that this container image contains busybox with wget included, which could help someone who made it into the running container to download malicious files, etc.

For mcr.microsoft.com/dotnet/runtime-deps:8.0.0-preview.2-jammy-chiseled-amd64:

[0 packages]
No packages discovered

Note: Another important point is the fact that alpine is based on musl, on the other hand, the distroless ones are based on glibc. This blog post: Why I Will Never Use Alpine Linux Ever Again highlights some known issues with alpine/musl. Good to keep in mind too.

Furthermore, and for your information, I gave trivy a try for these three container images, here is the summary of the scans:

  • For mcr.microsoft.com/dotnet/runtime-deps:7.0.4-alpine3.17:
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
  • For mcr.microsoft.com/dotnet/runtime-deps:7.0.4-cbl-mariner2.0-distroless:
Total: 6 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 4, CRITICAL: 1)
  • For mcr.microsoft.com/dotnet/runtime-deps:8.0.0-preview.2-jammy-chiseled-amd64:
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

With all of that being said, I will still continue to use the mcr.microsoft.com/dotnet/runtime-deps:7.0.4-alpine3.17 one. I will keep an eye on mcr.microsoft.com/dotnet/runtime-deps:8.0.0-preview.2-jammy-chiseled-amd64, that’s for sure, it seems to be very promising!

4. Immutable base image

Use a specific tag or version for your base image, not latest is important for traceability. But a tag or version is mutable, which means that you can’t guarantee which content of the container you are using. Using a digest will guarantee that, a digest is immutable.

Update the Dockerfile with these two base images:

  • mcr.microsoft.com/dotnet/sdk:7.0@sha256:f712881bafadf0e56250ece1da28ba2baedd03fb3dd49a67f209f9d0cf928e81
  • mcr.microsoft.com/dotnet/runtime-deps:7.0-alpine3.17-amd64@sha256:941c0748b773dd13f2930cded91d01f62d357a785550c25eabe3d53d7997ae4b

Note: it’s also highly encouraged that you store these two base images in your own private container registry and update your Dockerfile to point to them. You will guarantee their provenance, you will be able to scan them, etc.

5. Update dependencies

An important aspect is to keep your dependencies up-to-date in order to fix CVEs, catch new features, etc. One way to help you with that, in an automated fashion, is to leverage tools like Renovate or Dependabot if you are using GitHub.

Here is an example of how you can configure Dependabot to keep your container base images as well as your .NET packages up-to-date:

cat <<EOF > .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/my-sample-app"
    schedule:
      interval: "daily"
  - package-ecosystem: "nuget"
    directory: "/my-sample-app"
    schedule:
      interval: "daily"

6. .dockerignore

Use a .dockerignore file to ignore files that do not need to be added to the image.

Here is an example of how your .dockerignore could look like:

cat <<EOF > my-sample-app/.dockerignore
**/*.sh
**/*.bat
**/bin/
**/obj/
**/out/
Dockerfile*
EOF

7. Unprivilege/non-root container

For security purposes, always ensure that your images run as non-root by defining USER in your Dockerfile.

ASP.NET Core apps listen on port 80 by default. The problem is that port 80 is a privileged port that requires root permission. For making our container unprivilege, we will configure its port to 8080:

EXPOSE 8080
ENV ASPNETCORE_URLS=http://*:8080
USER 1000

You can now run this container with -u 1000 on port 8080:

docker run -d -p 80:8080 -u 1000 my-sample-app
curl localhost:80

8. Read-only container filesystem

To make the container in read-only mode on filesystem, DOTNET_EnableDiagnostics needs to be turned off. DOTNET_EnableDiagnostics allows is used for debugging, profiling, and other diagnostics.

ENV DOTNET_EnableDiagnostics=0

You can now run this container with --read-only:

docker run -d -p 80:8080 -u 1000 --read-only my-sample-app
curl localhost:80

That’s a wrap!

Congrats!

With these 8 tips illustrated throughout this blog post, we:

  • Reduced the surface of attack of the container image (alpine was chosen, more distroless options are coming, stay tuned!)
  • Illustrated tips to improve the day-2 operations in order to keep our dependencies up-to-date
  • Made the container running as unprivilege/non-root in read-only on filesystem

Here is the final Dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:7.0@sha256:f712881bafadf0e56250ece1da28ba2baedd03fb3dd49a67f209f9d0cf928e81 as builder
WORKDIR /app
COPY my-sample-app.csproj .
RUN dotnet restore my-sample-app.csproj -r linux-musl-x64
COPY . .
RUN dotnet publish my-sample-app.csproj -r linux-musl-x64 -c release -o /my-sample-app --no-restore --self-contained true -p:PublishSingleFile=true -p:PublishTrimmed=true -p:TrimMode=partial

FROM mcr.microsoft.com/dotnet/runtime-deps:7.0-alpine3.17-amd64@sha256:941c0748b773dd13f2930cded91d01f62d357a785550c25eabe3d53d7997ae4b
WORKDIR /app
COPY --from=builder /my-sample-app .
EXPOSE 8080
ENV ASPNETCORE_URLS=http://*:8080
ENV DOTNET_EnableDiagnostics=0
USER 1000
ENTRYPOINT ["/app/my-sample-app"]

And if you want to deploy this container image in a secure manner in Kubernetes, here is the associated Deployment resource:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-sample-app
  labels:
    app: my-sample-app
spec:
  selector:
    matchLabels:
      app: my-sample-app
  template:
    metadata:
      labels:
        app: my-sample-app
    spec:
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: my-sample-app
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
          image: my-sample-app:latest
          ports:
            - containerPort: 8080

You are now ready to Sail Sharp! Hope you enjoyed that one! Cheers!