Container Security Best Practices

Introduction

Containers have revolutionized how we build, ship, and run applications. But with great power comes great responsibility—especially when it comes to security. A single misconfigured container or vulnerable image can become the gateway for attackers to compromise your entire infrastructure.

In this comprehensive guide, I'll walk you through the essential best practices for securing containerized applications, from the initial image build to runtime protection in production.

Understanding the Container Security Landscape

Container security isn't just about scanning images for vulnerabilities. It's a multi-layered approach that spans the entire container lifecycle:

Build → Store → Deploy → Runtime
  ↓       ↓        ↓        ↓
SAST   Registry  K8s       Runtime
Scan   Scan     Policies  Protection

Each stage requires specific security controls and best practices. Let's dive into each one.

1. Secure Container Images

Start with Minimal Base Images

The smaller your base image, the smaller your attack surface. Always prefer minimal base images:

Good Choices:

Example: Multi-stage Build with Distroless

# Build stage
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp

# Production stage
FROM gcr.io/distroless/static-debian12
COPY --from=builder /app/myapp /
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/myapp"]

Scan Images for Vulnerabilities

Integrate vulnerability scanning into your CI/CD pipeline:

Popular Tools:

Example: Trivy in GitHub Actions

name: Container Scan
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Build image
        run: docker build -t myapp:latest .
      
      - name: Run Trivy scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:latest
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
      
      - name: Upload results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

Keep Images Updated

Vulnerabilities are discovered daily. Establish a process to regularly rebuild and redeploy images:

2. Container Registry Security

Private Registries with Access Controls

Never use public registries for proprietary images. Use private registries with proper access controls:

Options:

Image Signing and Verification

Ensure image integrity with cryptographic signatures:

Using Docker Content Trust (DCT):

# Enable DCT
export DOCKER_CONTENT_TRUST=1

# Push signed image
docker push myregistry.io/myapp:v1.0.0

# Only signed images can be pulled
docker pull myregistry.io/myapp:v1.0.0

Using Cosign (Sigstore):

# Generate key pair
cosign generate-key-pair

# Sign image
cosign sign --key cosign.key myregistry.io/myapp:v1.0.0

# Verify signature
cosign verify --key cosign.pub myregistry.io/myapp:v1.0.0

Scan Registry Images Continuously

Don't just scan during build—continuously scan images in your registry:

# Example: Trivy scheduled scan
apiVersion: batch/v1
kind: CronJob
metadata:
  name: registry-scan
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: trivy
            image: aquasec/trivy:latest
            args:
            - image
            - --severity
            - CRITICAL,HIGH
            - myregistry.io/myapp:latest

3. Runtime Security

Run as Non-Root User

Never run containers as root. Create a dedicated user:

FROM node:18-alpine

# Create app user
RUN addgroup -g 1001 appgroup && \
    adduser -D -u 1001 -G appgroup appuser

# Set working directory
WORKDIR /app

# Copy application
COPY --chown=appuser:appgroup . .

# Install dependencies
RUN npm ci --only=production

# Switch to non-root user
USER appuser

EXPOSE 3000
CMD ["node", "server.js"]

Use Read-Only Root Filesystem

Make the container's root filesystem read-only to prevent tampering:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /app/cache
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

Implement Resource Limits

Prevent resource exhaustion attacks with limits:

apiVersion: v1
kind: Pod
metadata:
  name: resource-limited
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "256Mi"
        cpu: "200m"

Drop Unnecessary Capabilities

Containers don't need all Linux capabilities. Drop them:

apiVersion: v1
kind: Pod
metadata:
  name: minimal-capabilities
spec:
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE  # Only if needed for ports < 1024

4. Kubernetes Security Policies

Pod Security Standards

Enforce security policies at the cluster level:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Network Policies

Implement zero-trust networking with Network Policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

Use OPA/Gatekeeper for Policy Enforcement

Implement custom policies with Open Policy Agent:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockHostNamespace
metadata:
  name: block-host-namespace
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces:
    - production

5. Secrets Management

Never Hardcode Secrets

Bad:

ENV DATABASE_PASSWORD=mysecretpassword123

Good - Use Kubernetes Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: bXlzZWNyZXRwYXNzd29yZDEyMw==  # base64 encoded
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: DATABASE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

External Secrets Management

For production, use dedicated secrets management:

Options:

Example: External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: secret/database
      property: password

6. Runtime Threat Detection

Deploy Runtime Security Tools

Monitor container behaviour in real-time:

Popular Tools:

Example: Falco Rule

- rule: Unauthorized Process in Container
  desc: Detect unexpected processes in production containers
  condition: >
    spawned_process and
    container and
    container.image.repository = "myapp" and
    not proc.name in (node, npm)
  output: >
    Unexpected process started in container
    (user=%user.name command=%proc.cmdline container=%container.name)
  priority: WARNING

Implement Admission Controllers

Validate and mutate pod specifications before deployment:

Example: Admission Webhook

func (v *Validator) ValidatePod(pod *corev1.Pod) error {
    // Ensure non-root user
    if pod.Spec.SecurityContext == nil || 
       pod.Spec.SecurityContext.RunAsUser == nil ||
       *pod.Spec.SecurityContext.RunAsUser == 0 {
        return fmt.Errorf("pod must not run as root")
    }
    
    // Ensure resource limits
    for _, container := range pod.Spec.Containers {
        if container.Resources.Limits == nil {
            return fmt.Errorf("container must have resource limits")
        }
    }
    
    return nil
}

7. Compliance and Auditing

Enable Audit Logging

Track all API server activity:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: ""
    resources: ["pods", "secrets"]
  namespaces: ["production"]

Regular Security Audits

Conduct periodic security assessments:

1. Weekly: Vulnerability scan reports review

2. Monthly: Policy compliance checks

3. Quarterly: Penetration testing

4. Annually: Comprehensive security audit

CIS Benchmarks

Follow CIS Docker and Kubernetes benchmarks:

# Install kube-bench
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml

# Review results
kubectl logs job/kube-bench

8. Supply Chain Security

SBOM Generation

Generate Software Bill of Materials for all images:

# Using Syft
syft myapp:latest -o cyclonedx-json > sbom.json

# Scan SBOM with Grype
grype sbom:./sbom.json

Verify Base Image Provenance

Use only trusted base images from verified publishers:

Implement Software Supply Chain Security

Use tools like SLSA (Supply-chain Levels for Software Artifacts):

# GitHub Actions with SLSA
name: SLSA Build
on: push

jobs:
  build:
    permissions:
      id-token: write
      contents: read
    uses: slsa-framework/slsa-github-generator/.github/workflows/builder_go_slsa3.yml@v1.9.0
    with:
      go-version: 1.21

9. Emergency Response Plan

Incident Response Playbook

Have a plan for security incidents:

1. Detection: Automated alerts trigger investigation

2. Containment: Isolate affected containers immediately

3. Eradication: Patch vulnerabilities, rebuild images

4. Recovery: Deploy secure versions

5. Lessons Learned: Update policies and procedures

Container Forensics

Preserve evidence for investigation:

# Create snapshot of running container
docker commit suspicious-container forensics-snapshot:latest

# Export filesystem
docker export suspicious-container > container-filesystem.tar

# Analyse with forensics tools

10. Security Checklist

Use this checklist for every container deployment:

Build Time:

Registry:

Runtime:

Kubernetes:

Conclusion

Container security is not a one-time task—it's an ongoing process that requires vigilance at every stage of the container lifecycle. By implementing these best practices, you significantly reduce your attack surface and create multiple layers of defence.

Key Takeaways

1. Start with secure foundations: Use minimal base images and scan for vulnerabilities

2. Defence in depth: Implement security controls at every layer

3. Principle of least privilege: Drop unnecessary capabilities and run as non-root

4. Automate security: Integrate scanning and policy enforcement into CI/CD

5. Monitor continuously: Deploy runtime security tools and audit regularly

Next Steps

1. Audit current containers: Run Trivy/Grype on all production images

2. Implement least one runtime security tool: Start with Falco (it's free!)

3. Establish security policies: Define and enforce Pod Security Standards

4. Create response plan: Document incident response procedures

5. Train the team: Ensure everyone understands container security basics

Remember: Security is everyone's responsibility. Build security into your culture, not just your containers.

---

*Questions about container security? Need help securing your containerized infrastructure? [Let's connect](#contact).*