Deploying Private Tailscale Service with Headscale (Enabling Embedded DERP)

The Plan First

Use Docker Compose to orchestrate Headscale + Headplane (a WebUI that looks a lot like Tailscale’s official one), use Caddy to request TLS certificates (NGINX is also fine, but I’m lazy; Caddy can handle automatic application and renewal of TLS certificates for me) and reverse proxy to the Docker container.

I directly used the DERP service built into Headscale, which is much simpler than some extra DERP setups found online 😎

What you need is a domain name and a server with Docker installed (if you need to buy a server, you can click the link to purchase Tencent Cloud Lighthouse).

Later in the article, examples of Caddyfile, compose.yml, Headscale, and Headplane configuration files will be given. You only need to make simple modifications (domain name) to use them directly.

A Few More Words

Actually, I set up Headscale a few months ago, but documentation for self-hosting DERP is relatively scarce (because foreigners can just use the DERP nodes provided by Tailscale, and those who really need to self-host DERP are basically users in mainland China, which should be a relatively niche group). At that time, I hadn’t configured the built-in DERP of Headscale properly, so previously I only had Headscale itself, and DERP was still going through Tailscale’s official nodes.

Until recently, when my broadband package was about to expire, I asked China Telecom customer service, and the current public IPv4 will likely be reclaimed. So I started researching solutions for after losing the public IP:

• Use public IPv6
• Use Tailscale (with self-hosted DERP)

Coincidentally, while fiddling with IPv6 back in my hometown these past few days, I found it quite troublesome: first, the DDNS that comes with RouterOS adds the egress public IPv4 (but actually dialing only gets an internal address), which causes domain resolution to get an inaccessible A record; secondly, IPv6 cannot use L2TP/IPSec (enabling L2TP with RouterOS is quite simple) to set up a VPN to go home, and IKEv2 certificates are a bit troublesome.

Then I found that Tailscale can actually encrypt all traffic (using public Wi-Fi with a VPN layer to encrypt all traffic is my most frequent use case for public IPv4). So I fiddled with Headscale again and got self-hosted DERP working. Now, connecting back home to Unicom from Guangzhou Telecom won’t take a detour through Tailscale’s Los Angeles node DERP.

Preparation

Directory Structure

First, create the directory. I put all configurations in the /app/headscale directory. The structure is as follows:

.
├── compose.yml
├── config
│   └── config.yaml
├── headplane
│   └── config.yaml
├── lib
└── run

If you follow my directory structure, you can use the following command to create it:

sudo mkdir -p /app/headscale/{config,headplane,lib,run}

Docker Compose

The versions used here are headscale/headscale:0.26.0 (at the time of writing, 0.26.1 has been released, you can also use the new version) and ghcr.io/tale/headplane:0.6.0.

Modify the official example compose.yaml and save it to /app/headscale/compose.yml:

services:
  headscale:
    image: headscale/headscale:0.26.0
    restart: unless-stopped
    container_name: headscale
    ports:
      - "0.0.0.0:8080:8080" # Port used by headscale HTTP
      - "0.0.0.0:9090:9090" # TODO: This port might not be needed, I'll confirm when I have time
      - "0.0.0.0:3478:3478" # Port used by DERP
    volumes:
      - /app/headscale/config:/etc/headscale
      - /app/headscale/lib:/var/lib/headscale
      - /app/headscale/run:/var/run/headscale
    command: serve
    labels:
      # Label needed by Headplane
      me.tale.headplane.target: headscale

  headplane:
    image: ghcr.io/tale/headplane:0.6.0
    container_name: headplane
    restart: unless-stopped
    ports:
      - '3000:3000'
    volumes:
      - '/app/headscale/headplane/config.yaml:/etc/headplane/config.yaml'
      # Headscale's config file also needs to be mounted, and the path should be consistent
      - '/app/headscale/config/config.yaml:/etc/headscale/config.yaml'

      # Directory where Headplane's data files are stored
      - '/app/headscale/lib:/var/lib/headplane'

      # Read-only mount of Docker socket
      - '/var/run/docker.sock:/var/run/docker.sock:ro'

Headscale Configuration

First, copy the official configuration file (https://github.com/juanfont/headscale/blob/main/config-example.yaml) to /app/headscale/config/config.yaml, then find and modify the following parts:

# Server address, change to your domain name, and protocol to https (Caddy will handle it)
server_url: https://your-domain

# Listen address, change from 127.0.0.1 to 0.0.0.0
listen_addr: 0.0.0.0:8080

# Then the derp section:
derp:
  server:
    # Set to true to enable headscale's built-in derp
    enabled: true

    # region_id can keep the default 999
    region_id: 999
    # region_code and region_name can be anything, suggest something unique to confirm correct usage of self-hosted DERP later
    region_code: "headscale-gz"
    region_name: "Guangzhou"

    # Ensure this is true, so it will be automatically pushed to nodes
    automatically_add_embedded_derp_region: true

    # Change ipv4 to your server's IP address, also fill in ipv6 if available
    ipv4:
    ipv6:
    # This concludes the derp.server part to be modified
    # Next is the derp.urls part
  url:
    # The example config provides a tailscale official derpmap. If you don't want to use tailscale's derp, delete or comment it out.
    # - https://controlplane.tailscale.com/derpmap/default

Headplane Configuration

Also, copy the official example configuration https://github.com/tale/headplane/blob/main/config.example.yaml to /app/headscale/headplane/config.yaml, then modify the following parts:

server:
  # For encrypting cookies, randomly generate 32 characters
  cookie_secret:
  # Secure cookies. If true, Headplane must be accessed via HTTPS. Since I don't plan to expose Headplane to the public network, and later it will be accessed via internal IP + http after connecting to the server with Tailscale (as this server will also become an internal device), change it to false.
  cookie_secure: false

headscale:
  # URL of the Headscale instance. Since it's orchestrated with Docker Compose, you can directly access it using the container name `headscale`, and the port is 8080 as configured just now.
  url: "http://headscale:8080"

Launch!

The configuration files are ready. Next, switch the working directory to /app/headscale and execute:

docker compose up -d

After a few seconds, execute docker compose logs to check the output. If everything is fine, there should be no errors, which means it started successfully!

Configure Firewall and Caddy

You need to allow TCP 443 and 3478 ports in your security group.

(If Caddy is not installed, you can refer to the official documentation https://caddyserver.com/docs/install for installation via package manager)

Next, configure Caddy so that Headscale can be accessed from outside. First, point your domain to your server, then add a section to /etc/caddy/Caddyfile (generally this file, if not, check which file follows --config by executing ps ax | grep caddy):

<your-domain> {
	reverse_proxy :8080
}

Then execute sudo service caddy reload to reload Caddy, and confirm no errors via sudo service caddy status.

Done

If none of the previous steps reported errors, the deployment is complete. To join this server on Linux, use this command (Tailscale must be installed first, installation instructions: https://tailscale.com/kb/1031/install-linux):

tailscale up --login-server https://<your-domain> # You can add parameters as needed

In graphical clients (iOS/macOS/Android/Windows, etc.), use Add Account Using Alternate Server to add an account, and fill in https://<your-domain> for the address.

Both methods will pop up a Machine Key. Copy this key into Headplane under Add Device -> Register Machine Key.

Oh, by the way, before adding the first device, you cannot access Headplane through the internal IP. At this time, use SSH port forwarding. Open a new terminal window and execute:

ssh -L 3000:127.0.0.1:3000 <your-server-IP-or-domain> -N
# Simple explanation of 3000:127.0.0.1:3000
# The first 3000 is the local port; the middle 127.0.0.1 is the address to forward to after SSH connecting to the server, i.e., forwarding to the server itself; the second 3000 is the port to forward.
# Put together: Forward connections you make to local port 3000 (first 3000) to your server, and have your server access port 3000 (second 3000) of 127.0.0.1 (i.e., the server itself).

Then access http://127.0.0.1:3000/admin with a browser to open Headplane. The first time you open Headplane, it will ask for an API Key, and it kindly tells you that you can get the key by executing headscale apikeys create on the server. But since we are running in Docker, the command needs a slight change:

docker exec -it headscale headscale apikeys create

A brief explanation for your future reference when executing other headscale commands:

docker exec -it is for executing commands interactively inside a container. The first headscale immediately following is the container name, and the subsequent headscale apikeys create is the command to be executed. So you can also list created API Keys like this:

docker exec -it headscale headscale apikeys list

Getting sidetracked. Back to the port forwarding: after you’ve configured your machine, just Ctrl + C to end the window doing the port forwarding. If you’ve already added this server running Headscale to your Machines, then you won’t need port forwarding anymore. You can access it directly after connecting to Tailscale: http://<server-hostname-or-internal-IP, usually 100.64.x.x>:3000/admin

Oh, one more thing: confirm the self-hosted DERP. When you have added two machines that are not in the same internal network and they access each other, you can check on one of the machines:

• If it’s an Android device, click on the device you just accessed. There is a dashboard icon in the top right corner. Clicking it will perform a PING. You can then see either Direct connection or Relayed connection(HEADSCALE-GZ), etc. If the name in parentheses is the DERP region_code you named earlier, it means it’s connected to the self-hosted DERP server.
• If it’s Linux, execute tailscale status. If it’s macOS, execute /Applications/Tailscale.app/Contents/MacOS/Tailscale status. If you see direct, it’s a direct connection; if you see relay followed by your custom DERP name, it’s connected to the self-hosted DERP.
• iOS doesn’t seem to show connection status; Windows has not been tested, feel free to explore.