ouranos/docs/kb/Docker won't start inside Incus container.md

Docker won't start inside Incus container

Issue

Running Docker inside Incus has worked for years, but a recent Ubuntu package update caused it to fail.

Symptoms

Docker containers won't start with the following error:

docker compose up
Attaching to neo4j
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied

The issue is AppArmor on Incus containers. The host has AppArmor, and Incus applies an AppArmor profile to containers with security.nesting=true that blocks Docker from writing to /proc/sys/net/ipv4/ip_unprivileged_port_start.

Solution (Automated)

The fix requires both host-side and container-side changes. These are now automated in our infrastructure:

1. Terraform - Host-side fix

In terraform/containers.tf, all containers with security.nesting=true now include:

config = {
  "security.nesting" = true
  "raw.lxc"          = "lxc.apparmor.profile=unconfined"
}

This tells Incus not to load any AppArmor profile for the container.

2. Ansible - Container-side fix

In ansible/docker/deploy.yml, Docker deployment now creates a systemd override:

- name: Create AppArmor workaround for Incus nested Docker
  ansible.builtin.copy:
    content: |
      [Service]
      Environment=container="setmeandforgetme"
    dest: /etc/systemd/system/docker.service.d/apparmor-workaround.conf

This tells Docker to skip loading its own AppArmor profile.

Manual Workaround

If you need to fix this manually (e.g., before running Terraform/Ansible):

Step 1: Force unconfined mode from the Incus host

# On the HOST (pan.helu.ca), not in the container
incus config set <container-name> raw.lxc "lxc.apparmor.profile=unconfined" --project agathos
incus restart <container-name> --project agathos

Step 2: Disable AppArmor for Docker inside the container

# Inside the container
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/apparmor-workaround.conf <<EOF
[Service]
Environment=container="setmeandforgetme"
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

Reference: ktz.blog

Verification

Tested on Miranda (2025-12-28):

# Before fix - fails with permission denied
$ ssh miranda.incus "docker run hello-world"
docker: Error response from daemon: failed to create task for container: ... permission denied

# After applying both fixes
$ ssh miranda.incus "docker run hello-world"
Hello from Docker!

# Port binding also works
$ ssh miranda.incus "docker run -d -p 8080:80 nginx"
# Container starts successfully

Security Considerations

Setting lxc.apparmor.profile=unconfined only disables the AppArmor profile that Incus applies to the container. The host's AppArmor daemon continues running and protecting the host itself.

Security layers with this fix:

• Host AppArmor ✅ (still active)
• Incus container isolation ✅ (namespaces, cgroups)
• Container AppArmor ❌ (disabled with unconfined)
• Docker container isolation ✅ (namespaces, cgroups)

For sandbox/dev environments, this tradeoff is acceptable since:

• The Incus container is already isolated from the host
• We're not running untrusted workloads
• Production uses VMs + Docker without Incus nesting

Explanation

What happened is that a recent update on the host (probably the incus and/or apparmor packages that landed in Ubuntu 24.04) started feeding the container a new AppArmor profile that contains this rule (or one very much like it):

deny @{PROC}/sys/net/ipv4/ip_unprivileged_port_start rw,

That rule is not present in the profile that ships with plain Docker, but it is present in the profile that Incus now attaches to every container that has security.nesting=true (the flag you need to run Docker inside Incus).

Because the rule is a deny, it overrides any later allow, so Docker's own profile (which allows the write) is ignored and the kernel returns permission denied the first time Docker/runc tries to write the value that tells the kernel which ports an unprivileged user may bind to.

So the container itself starts fine, but as soon as Docker tries to start any of its own containers, the AppArmor policy that Incus attached to the nested container blocks the write and the whole Docker container creation aborts.

The two workarounds remove the enforcing profile:

1. raw.lxc = lxc.apparmor.profile=unconfined — Tells Incus "don't load any AppArmor profile for this container at all", so the offending rule is never applied.

2. Environment=container="setmeandforgetme" — Is the magic string Docker's systemd unit looks for. When it sees that variable it skips loading the Docker-default AppArmor profile. The value literally does not matter; the variable only has to exist.

Either way you end up with no AppArmor policy on the nested Docker container, so the write to ip_unprivileged_port_start succeeds and your containers start again.

In short: Recent Incus added a deny rule that clashes with Docker's need to tweak that sysctl; disabling the profile (host-side or container-side) is the quickest fix until the profiles are updated to allow the operation. Because the rule is a deny, it overrides any later allow, so Docker’s own profile (which allows the write) is ignored and the kernel returns: