Build a Private Network with Headscale on ZCP

This tutorial builds a private network on ZCP: a VPC with a network tier that has no public exposure by default, plus a self-hosted Headscale server that gives you WireGuard-based mesh access into it. It’s the foundation for the rest of this series: private storage and private employee desktops both live inside the tier you build here, reachable only through the mesh, never over a public IP.

By the end you have:

• A private VPC and network tier
• A custom network ACL locking that tier down to only the traffic it needs
• A self-hosted Headscale server, deployed from the Marketplace
• A subnet router bridging the public internet and the private tier
• Your own device connected to the mesh, reaching into the private tier

Plan for about 30 minutes.

Note

The slugs in this tutorial (region yul-1, project default-9, plans, and so on) are examples from one account. Yours will differ. Every step shows the list command that prints the right value for your account and region. Always use those, don’t copy the examples verbatim.

Before you start

• A ZSoftly Public Cloud account. Sign up first if you do not have one.
• A terminal with an SSH client.
• jq installed. The build and teardown scripts in this tutorial require it (apt install jq or brew install jq).
• The Tailscale client installed on your own machine, to prove connectivity at the end.

Note

No domain or TLS certificate is required. The Headplane template runs over plain HTTP by default, and that’s the baseline this tutorial uses. Encryption for the admin UI comes from an SSH tunnel instead of TLS, not from skipping encryption entirely. See “What the script builds” below.

Step 1: Install the CLI

# macOS and Linux
curl -fsSL https://raw.githubusercontent.com/zsoftly/zcp-cli/main/scripts/install.sh | bash

# Windows (PowerShell)
irm https://raw.githubusercontent.com/zsoftly/zcp-cli/main/scripts/install.ps1 | iex

Confirm it works with zcp version.

Step 2: Authenticate

1. In the portal, open Profile → API Tokens and create a token. Copy it.
2. Create a CLI profile and answer its prompts:

zcp profile add default

It prompts for the Bearer token you copied, then a default region and default project (see the note below to look those up first if you don’t already know them). There’s no API URL prompt. That’s a fixed default unless you override it with --api-url-override. Then verify:

zcp auth validate

Note

Every command that touches a region-specific resource needs a region and a project. The profile you just created already carries defaults from the prompts above. Look them up first if you don’t know them yet:

zcp region list                # find your region, e.g. yul-1
zcp project list                # find your project slug, e.g. default-9

Override the profile’s defaults for one command with --region/--project flags, or for the rest of your shell session with export ZCP_REGION=.../export ZCP_PROJECT=....

Step 3: Find your resources

Find the Headplane template (it bundles the Headscale control server and a web UI):

zcp template list | grep -i headplane

Note the SLUG (for example zmi-headplane-070-ubuntu2404-100-1). Template slugs vary by region and version.

You also need a compute plan, a network plan, a VPC router plan, and a storage category:

zcp plan vm                 # compute plans, e.g. ca2sm
zcp plan network            # network plans, e.g. pnet-yul
zcp plan router             # VPC router plan, e.g. virtual-private-cloud-vpc-1
zcp storage-category list   # e.g. pro-nvme, ssd-storage, premium-ssd

Step 4: Add your SSH key

ssh-keygen -t ed25519 -C "you@example.com"   # skip if you already have one

zcp ssh-key import --name my-key --key-file ~/.ssh/id_ed25519.pub

zcp ssh-key list

Note

The key name must be 20 characters or fewer, and the public key itself must be unique on your account.

Run the script

Building the VPC, private tier, ACL, Headplane, and subnet router is one script: zcp/build-private-network.sh from the zsoftly/tools repository. It runs through the same phases explained in the next section, in order, and prints each resource as it creates it.

Note

This script and the teardown script further down need a bash shell. That’s native on macOS and Linux. On Windows, run it from WSL or Git Bash.

bash <(curl -fsSL https://raw.githubusercontent.com/zsoftly/tools/main/zcp/build-private-network.sh) \
  --ssh-key my-key --name my-workspace

This tutorial uses my-workspace as the --name prefix throughout its command examples. The screenshots below are from an earlier test run with a different prefix, so the resource names you see in them won’t match the command examples exactly. Your own resource names will differ from both, based on whatever --name you choose.

The script picks up ZCP_REGION and ZCP_PROJECT from your shell if you exported them in Step 2. Pass --region/--project instead if you didn’t.

Flag	Purpose	Default / requirement
--ssh-key	Key name from Step 4, used for both VMs	Required
--name	Prefix for every resource name it creates	workspace
--region	zcp region slug	Required (flag or env var)
--project	zcp project slug	Required (flag or env var)
-y/--yes	Skip the confirmation prompt below	Off

The Headplane template, both compute plans, the network plan, the VPC router plan, and both storage categories are auto-discovered from your account. The script prints what it picked at the top of its own output, so you can see the choice before anything is created.

The subnet router template isn’t looked up. It defaults to a fixed slug (ubuntu-2404-lts-1). Pass --router-template if your account doesn’t have that template. Your public IP (--my-ip), the VPC network base (--network-address), and the billing cycle (--billing-cycle) aren’t account lookups either. They default to an auto-detected or fixed value that you can override.

Pass any flag explicitly to pin a specific value instead. Run the script with --help for the full list.

==> Preflight checks
[OK] zcp CLI authenticated, region=yul-1 project=default-9
[INFO] Detecting your public IP...
[INFO] Admin-port access scoped to: <your-ip>/32
[INFO] Resolved resources:
    Headplane template   : zmi-headplane-070-ubuntu2404-100-1
    Headplane plan       : ci2ls
    Router template      : ubuntu-2404-lts-1
    Router plan          : ci2ls
    Network plan         : pnet-yul
    VPC router plan      : virtual-private-cloud-vpc-1
    VPC storage category : pro-nvme
    VM storage category  : pro-nvme

This creates a VPC, two VMs (plans above), and networking on your account now.
Billing starts as soon as each resource is created.
Type 'yes' to continue:

Unless you passed -y/--yes, the script stops here and waits for you to type yes. Everything above this point is read-only lookups, nothing has been created yet.

The script then takes several minutes. It waits for both VMs to boot, waits for Headplane’s first-boot provisioning to finish, and waits for SSH before configuring anything over it.

What the script builds

VPC and private tier

The private tier itself has no public IP. Only Headplane gets an internet-facing application port.

The script creates a VPC (my-workspace) and a network tier inside it (my-workspace-tier), with no public IP on the tier itself. Both VMs the script deploys get their own public IP, and the script needs it to configure each one over SSH regardless. Beyond that, each VM needs it for a different reason. Headplane needs it because it is the Headscale server: the router and your own device both reach it over that address. The router needs it too, but only during setup, to reach Headplane before it enrolls in the mesh. Headplane is the only one of the two with an internet-facing application port, opened deliberately through firewall and port-forward rules.

Note

ZCP gives the VPC a source-NAT IP for outbound traffic automatically at creation. The script doesn’t allocate one itself, and you shouldn’t either with zcp ip allocate. Doing so creates a redundant, billable extra IP. If it happens, zcp ip release <slug> cleans it up.

The custom network ACL

The ACL makes the tier private, not the VPC by itself.

The tier’s default ACL permits everything. The script replaces it with one that only allows what the tier needs, then applies that ACL to the tier. A VPC alone doesn’t guarantee isolation. The ACL does.

Note

The second CIDR the script allows, 100.64.0.0/10, is Headscale’s mesh address range. This isn’t something looked up after Headplane exists. It’s Tailscale and Headscale’s documented default IP range for every device on the mesh (RFC 6598, the “Shared Address Space” block). Every default install uses it, unless someone deliberately reconfigures it. The script can add these rules before Headplane is deployed, because the range itself doesn’t depend on anything else it builds.

Caution

Allowing only the tier’s own CIDR (10.20.1.0/24) is not enough. Reaching the subnet router’s own tier IP through the mesh works with just that rule, because that traffic terminates directly at the router’s WireGuard tunnel endpoint, before the tier ACL is evaluated.

Traffic to any other VM on the tier is forwarded by the router first. We confirmed directly with a packet capture that Tailscale’s default source NAT rewrites this forwarded traffic to the router’s own tier address, not the original mesh client’s address. The mesh-CIDR rule (ingress and egress) stays in this ACL because an earlier live test found reaching other tier VMs fails without it. Egress rules are required too, since this platform’s network ACLs are stateless: an ingress-only rule doesn’t cover return traffic.

Verify the rules landed (see zcp acl rules under Inspect what was created, below):

Headplane

Headplane opens exactly one application port to the internet, deliberately: the mesh control endpoint. The admin UI never touches the public internet at all.

The script deploys the Headplane marketplace template (it bundles the Headscale control server and a web UI) on its own public network.

By default, the template’s first-boot script points the Headscale configuration at the VM’s private IP. External devices need the public IP instead, so the script SSHes in, rewrites that config, and restarts the stack.

The script opens port 8080 (Headscale’s mesh control endpoint) on the firewall and creates the matching port-forward rule. A firewall rule alone permits traffic at the network level. On this kind of network it does not get you reachability by itself. A port-forward rule maps the public IP’s port to the VM’s private IP. Both are required. 8080 has to stay open broadly (0.0.0.0/0): any remote device that will ever join the mesh needs to reach it from wherever it is, by design.

Caution

Port 3000, the admin UI, is never opened on the firewall at all, not even scoped to your own IP. The API key it protects controls the entire mesh: minting preauth keys, approving routes, seeing every connected device. Sending that key over plain HTTP on a public port is a real credential-theft risk, not a theoretical one, so the script never exposes it that way. Reach it only through an SSH tunnel over the port 22 rule already in place:

ssh -L 3000:localhost:3000 ubuntu@<headplane-public-ip>

Leave that connected, then open http://localhost:3000/admin/login in your own browser. The script’s final summary prints this exact command with your real IP filled in.

Caution

Marketplace App templates like this one get a default SSH firewall rule at deploy time, open to any address (0.0.0.0/0, both TCP and UDP port 22), an explicit rule visible in zcp firewall list --ip <ip-slug> (find the slug with zcp ip list). Not something you created, and not scoped to you. The script adds a replacement rule scoped to your own IP and confirms it exists before deleting the open one. That way, a failure partway through never leaves the VM unreachable over SSH.

The script runs this same routine on both VMs it creates, not just Headplane: add and confirm a rule scoped to your own IP, then remove any open 0.0.0.0/0 rule found. The subnet router, deployed from a plain OS image, may not have the default open rule Marketplace App templates get, but it still gets the scoped rule. It’s the actual gateway into your private tier, so it needs to be locked down too, whatever its starting state.

First boot on the Headplane VM generates a unique cookie secret, starts the stack, creates a default Headscale user, and mints an API key, written to /etc/headplane/credentials.txt on the VM. The script reads it over SSH and prints it in its final summary. That’s a one-time read by convention, not a technical limit: if you need the key again later, SSH in and read /etc/headplane/credentials.txt directly.

Note

If the Headplane UI rejects this key (“API key was not found in the Headscale database”), mint a fresh one directly on the VM and use that instead:

sudo docker exec headscale headscale apikeys create --expiration 90d

With the tunnel open, sign in at http://localhost:3000/admin/login with the API key.

The subnet router

The subnet router is the only path between the internet and the private tier.

The script deploys a small VM with two network interfaces: its own public network, to reach Headscale for registration, plus the private tier, attached after creation.

The platform hot-adds the second network interface, but the operating system doesn’t bring it up automatically. The script writes a netplan file for the new interface and applies it, then reads back the address the tier’s DHCP assigned.

It installs the Tailscale client on the router (the same client Headscale uses, pointed at a custom control server) and enables IP forwarding before registering. tailscale up prints a warning about this (“IP forwarding is disabled, subnet routing/exit nodes will not work”). It does not block on it. Skip this step and you end up with a route that’s approved but never forwards traffic.

The script mints a preauth key on the Headplane server and uses it to register the router, advertising the tier’s CIDR as a route.

Note

Registering needs the numeric Headscale user ID, not the username string. --user default fails with a parse error. You may also see a warning about “UDP GRO forwarding” being suboptimally configured. That’s a performance tuning suggestion, not an error, and it doesn’t block registration.

Approving an advertised route on the Headscale side is not automatic, so the script does this for you: it looks up the router’s node ID and approves the tier CIDR against it. Without that approval, list-routes shows the route as Available but never Approved or Serving. It can take a few seconds for Serving (Primary) to catch up even after approval. That’s normal.

You can also see this from the Headplane UI: the router shows Connected with a Subnets badge once it’s advertising the route.

This VM is the door into the private tier. Later tutorials’ desktops and storage sit behind it without needing their own public IPs.

Inspect what was created

Everything the script built is a normal ZCP resource. List it the same way you’d list anything else:

zcp vpc list
zcp network list
zcp instance list
zcp acl rules my-workspace my-workspace-acl

ID            NAME                        STATE    PRIVATE IP  PUBLIC IP      REGION
a1b2c3d4-...  my-workspace-headscale      Running  10.0.0.214  198.51.100.10  YUL-1
e5f6a7b8-...  my-workspace-subnet-router  Running  10.0.0.76   198.51.100.11  YUL-1

acl rules takes the VPC’s slug, not its name. They’re the same on a clean account, but a name can get an auto-suffixed slug (my-workspace-1) if something else already used it. If the command above doesn’t resolve, get the real slug from zcp vpc list first.

Connect from your own machine

The build script’s own final summary already mints a fresh preauth key for your device. It prints a ready one-liner that installs Tailscale, if it isn’t already installed, and registers it against your Headscale server in one step. This is the same vpn/install.sh script you’d use to onboard any other endpoint. Copy that line from your terminal output. It looks like this (key genericized, yours is a real value):

HEADSCALE_URL="http://198.51.100.10:8080" \
  bash <(curl -fsSL https://raw.githubusercontent.com/zsoftly/tools/main/vpn/install.sh) "your-name" --key "hskey-auth-EXAMPLE..."

Paste it into a terminal on your own machine and run it. On macOS you may need to approve a network extension once in System Settings → Privacy & Security before it finishes connecting. On Windows without WSL or Git Bash, use vpn/install.ps1 from the same zsoftly/tools repository instead. The PowerShell invocation pattern is in that repo’s README.

Verify:

tailscale status
ping <router-tier-ip>

Direct, same-region connections typically respond in a few milliseconds. A connection relayed through a DERP server (common across distant networks) can take several hundred milliseconds, especially on the first packet while the path negotiates. Both are normal. Network path affects latency far more than VM size does.

Caution

If a node shows offline in tailscale status, with a health check message about being unable to reach the coordination server, restart tailscaled on the affected node. This can happen even when nothing about the network setup is wrong:

sudo systemctl restart tailscaled

This has been observed on both the subnet router and plain clients.

Verify isolation

Reaching the subnet router’s own tier IP in the previous section proves the router itself has no public exposure. It does not prove the tier as a whole is isolated. That traffic terminates directly at the router’s WireGuard endpoint. Reaching any other VM on the tier goes through the router’s forwarding instead, a different path.

For real proof, deploy a second VM the same way the script deployed the subnet router: its own public IP for setup, then attached to the tier. That public IP is only there so you can configure it, not part of what’s being tested:

zcp instance create --name isolation-check \
  --template ubuntu-2404-lts-1 --plan ci2ls --billing-cycle hourly \
  --network-plan pnet-yul --storage-category pro-nvme --ssh-key my-key --wait

zcp instance add-network isolation-check --network my-workspace-tier

add-network also takes the tier’s slug, not its name, the same caveat as above. Get it from zcp network list if the name alone doesn’t resolve.

SSH in over its own public IP and bring up the hot-added tier NIC the same way the script did for the router (see “The subnet router” above for why this manual step is needed):

ip -br link show   # find the new interface, typically ens8

sudo tee /etc/netplan/60-tier-nic.yaml <<'EOF'
network:
  version: 2
  ethernets:
    ens8:
      dhcp4: true
EOF
sudo netplan apply

ip -4 -br addr show ens8   # note the address it gets, e.g. 10.20.1.201

From your already-connected device:

ping <isolation-check-tier-ip>

That succeeds, through the mesh and the router’s forwarding, not just to the router’s own address. Now try reaching the same tier IP from anywhere that never joined the mesh: your own home network, a different machine, anywhere on the public internet. It fails, every time. That address is private, with no public IP and no port-forward rule anywhere in this design. It was never internet-reachable in the first place, mesh or no mesh. That is the actual proof of isolation, not the mesh itself.

Like every --network-plan deploy in this tutorial, isolation-check also created its own standalone network and pinned source-NAT IP. instance delete won’t remove it, and the teardown script only manages resources under --name my-workspace, so it won’t touch this one either. Capture that network’s ID before deleting the VM, since the association disappears once the VM is gone:

zcp ip list -o json | jq -r '.[] | select(.vm=="isolation-check") | .network_id // empty' | head -1

Then delete the VM (zcp instance delete isolation-check). It’s not part of the working setup, just a way to see this for yourself. Find the network that ID belongs to and remove it from the CMP web portal, the same way described in the Clean up caution below.

Clean up

Hourly billing runs while resources exist. The teardown script removes the subnet router, the Headplane VM, the private tier, and the VPC, for a given --name prefix.

bash <(curl -fsSL https://raw.githubusercontent.com/zsoftly/tools/main/zcp/destroy-private-network.sh) \
  --name my-workspace

It picks up ZCP_REGION/ZCP_PROJECT from your shell the same way the build script does. Pass --region/--project instead if you didn’t export them.

Caution

This does not always remove everything on its own. Each VM’s deploy implicitly creates its own standalone network, separate from the VPC and tier, and deleting the VM never removes that network or the source-NAT IP pinned to it. The script detects this and warns you with the network’s ID, but it cannot delete it automatically. There’s no zcp command that resolves that ID to something deletable. It also runs with no confirmation prompt of its own: every delete it issues happens immediately.

A confirmed leftover always makes the script exit non-zero, so check $? after running it. A “can’t verify” warning only does the same if it actually deleted something this run. If it deleted nothing because everything was already gone, that warning means “nothing here to check,” not “you have a leftover,” and the script exits 0. Either way, check its output for a leftover-network or cannot-verify warning before you consider cleanup done. If one appears, confirm what’s left with zcp ip list. It carries the network ID you need, while zcp network list doesn’t expose one to match against, which is also why the script can’t delete this automatically. A leftover is a network and its pinned IP, not an instance or a VPC, so zcp instance list/zcp vpc list won’t show it. Remove the flagged network from the CMP web portal (search by the network ID the script prints).

Recap

1. Install the CLI, authenticate, find your account’s resource slugs, and import an SSH key (Steps 1-4).
2. Run build-private-network.sh --ssh-key <name> --name my-workspace. It creates the VPC, private tier, locked-down ACL, Headplane, and subnet router, then prints a connect command for your own device.
3. Copy that command from the script’s output, run it on your own machine, then verify with tailscale status and a ping into the tier.
4. Run destroy-private-network.sh --name <prefix> when you’re done, to remove everything and stop billing.

Next steps

The next parts of this series (private shared storage, then Ubuntu employee desktops, both reusing the tier and mesh you built) are still in progress. In the meantime:

• CLI reference: every command and flag
• Tutorials overview: the full list of available tutorials