Docker containers on overlay networks

Sometimes I want to join docker services onto overlay networks. If I want the service to be accessible over an overlay network, there are at least 2 options:

1. configure the docker host on the overlay network and expose the service port on the host node
2. configure the container on the overlay network

Each approach has advantages and disadvantages depending on the environment and service. Personally, I like most of my services to have their own unique identity and access on my overlay network. This makes them easier to move, and keeps more of their network and security config with the container itself.

At high level:

• create a new Dockerfile with FROM original image
• in Dockerfile, download and install overlay network binaries and supervisord
  • where possible, manually hardcode/specify random port for overlay network to listen on
  • ensure corresponding port is exposed on the host to avoid relays
• clear the entrypoint and set it to launch supervisord.
• Then get the launch command from the original image and create a supervisord config from that command and the overlay network start command,
• in docker-compose file mount the overlay network config into the right locations, expose overlay network ports
• passthrough tun device and add extra net capabilities as required

This varies based on how the original image was created, but the process of basically creating a new image from the old, including overlay network setup and a process manager, and then mounting those configs in feels quite robust.

For example, light images which only include the single binary don’t lend themselves well to this approach (like the otel-collector), so in those cases, I just created a new image from a preferred base (e.g. alpine), pluck the binary from the original and the remainder of the process stays the same.

tips and things to remember §

• Persist the overlay network config storage to avoid having to completely reauth & rejoin
• Some overlay networks will change the default network namespace, this will break setups that depend on docker compose network DNS
• alpine apk repos have some old pacakges, remember to switch to enable edge repo for recent tailscale versions
  • “RUN sed -i ‘s|v3.\d*|edge|’ /etc/apk/repositories

universal docker mod approach §

Tailscale released a universal docker mod. Most containers that use the s6 container overlay (e.g. basically all linuxsever.io images) can use universal docker mods. Where available, this is a far far quicker approach.

Unfortunately it’s not well supported anymore already. The official branch doesn’t support custom login servers to use headscale, so I use this branch instead https://github.com/tailscale-dev/docker-mod/pull/5.