Escaping Carrier-Grade NAT with Wireguard

Living in a semi-rural/exurban area, I have few choices for internet (save me Starlink!), and currently use a small semi-local WISP (wireless ISP); as is becoming more common to save IPv4 addresses they have us behind Carrier-grade NAT (CGN, CG-NAT) (and don’t support IPv6), so there’s no way to expose a service to the internet. I recently became interested in Chia, and started plotting and farming using the spare disk space on my home Plex media server (an older server-class Xeon with a bunch of hard drives). However, I was having trouble finding and keeping peers, and kept losing sync, since my node could only connect out. This motivated me to finally look into setting up a VPN tunnel out through a small cloud VPS, which turned out to be really easy with wireguard.

Note: while this is focused on Chia, these instructions work equally well for any other service you may want to host at home - HTTP, Plex, etc. The only difference is the port(s) forwarded from the cloud server with iptables.

Warning: these instructions can be a bit technical, and assume some knowledge of UNIX/Linux command line ¹. For an easier (and maybe even cheaper!) option, some dedicated VPN services, such as PureVPN, can provide port forwarding and I’ve been told work well with Chia.

What you’ll need

• A home server that you can install wireguard on. I’m using Ubuntu 20.04. A Raspberry Pi with Raspbian would make a good cheap server.
• A cloud server with a public IPv4 address. I like Vultr since they have a Chicago datacenter, are reasonably priced, and have great support. WARNING recently cloud providers have been taking extreme measures to stop Chia farmers destroying their SSDs, including blocking traffic to and from port 8444. I’ll update this list as I hear about provider restrictions:
  • Hetzner has blocked port 8444 making it unusable for VPN [2021-05-19]
  • Scaleway has banned plotting only [2021-05-20]

Set up the server

Check your cloud provider’s documentation for how to set up your server. At minimum there’s a couple things you will want to do to protect the security of your VPN:

1. Set up a sudo user
2. Disable root login (set PermitRootLogin no in /etc/ssh/sshd_config)
3. Update the OS

Also recommended:

1. Set up key-based authentication and disable password authentication
2. Install fail2ban, to block IP addresses that fail login attempts: sudo apt install fail2ban

Finally, to allow the server to route traffic through the VPN you’ll need to enable IP forwarding:

$ sudo sysctl -w net.ipv4.ip_forward=1

And set the following in /etc/sysctl.conf to persist the setting after reboot:

net.ipv4.ip_forward=1

Install Wireguard

See the wireguard instructions for your OS. For Ubuntu:

$ sudo apt install wireguard

Generate Keys

On each server run the following to generate the server’s private and public keys:

$ sudo -i
# cd /etc/wireguard
# umask 077
# wg genkey | tee privatekey | wg pubkey > publickey

Now the private and public keys will be in /etc/wireguard/privatekey and /etc/wireguard/publickey, respectively.

Home server configuration

We’re going to configure the home server to connect to wireguard on the cloud server and route all traffic to the internet through it as well as accept traffic routed from the internet through it (local traffic won’t be affected).

As root, create and edit the file /etc/wireguard/wg0.conf:

$ sudo vi /etc/wireguard/wg0.conf

Here’s the full contents, with angle brackets indicating placeholders:

[Interface]
PrivateKey = <contents of /etc/wireguard/privatekey>
Address = 10.10.92.1/32

[Peer]
PublicKey = <contents of /etc/wireguard/publickey on the cloud server>
Endpoint = <cloud server public IP address>:51822
AllowedIPs = 0.0.0.0/0
PersistentKeepAlive = 25

• Interface.PrivateKey: this is the private key we generated above for the home server. Hint: in vi type :r /etc/wireguard/privatekey to slurp in the file contents.
• Interface.Address: an arbitrary private address for wireguard to use. Make sure it’s not on a subnet otherwise addressable from the server. (You can probably just use the addresses in these examples.)
• Peer.PublicKey: the public key we generated above on the cloud server.
• Peer.Endpoint: The public IP address of our cloud server, and the port we’ll have it listen on.
• Peer.AllowedIPs: 0.0.0.0/0 means to route all internet traffic to the cloud server, and accept any traffic from the internet that gets routed from the cloud server (the iptables config on the cloud server will limit what gets routed through).
• Peer.PersistentKeepAlive: this makes wireguard send a packet out to the cloud server every 25 seconds even if there’s no traffic. This is needed to maintain the connection through statefeul NATs (like our dreaded CG-NAT), since the cloud server can’t connect back to the home server.

Cloud server configuration

We’ll configure the cloud server to listen on the port we configured the home server to connect to (51822), and establish iptables firewall rules to direct traffic received on the Chia port 8444 through the VPN to the home server, as well as to make any traffic from the home server look like it’s actually coming from the cloud server.

As root, create and edit the file /etc/wireguard/wg0.conf:

$ sudo vi /etc/wireguard/wg0.conf

Here’s the full contents, with angle brackets indicating placeholders:

[Interface]
PrivateKey = <contents of /etc/wireguard/privatekey>
Address = 10.10.92.2/32
ListenPort = 51822

PreUp = iptables -t nat -A PREROUTING -d <cloud public IP> -p tcp --dport 8444 -j DNAT --to-destination 10.10.92.1
PreUp = iptables -t nat -A POSTROUTING -s 10.10.92.1/32 -j SNAT --to-source <cloud public IP>
PostDown = iptables -t nat -D PREROUTING -d <cloud public IP> -p tcp --dport 8444 -j DNAT --to-destination 10.10.92.1
PostDown = iptables -t nat -D POSTROUTING -s 10.10.92.1/32 -j SNAT --to-source <cloud public IP>

[Peer]
PublicKey = <contents of /etc/wireguard/publickey on the home server>
AllowedIPs = 10.10.92.1/32

• Interface.PrivateKey: this is the private key we generated above for the cloud server.
• Interface.Address: an arbitrary private address for wireguard to use, on the same subnet as the home server’s Interface.Address.
• Interface.ListenPort: the port the server will listen on for connections from the client (home server). This must match the port specified in Peer.Endpoint in the home server’s config.
• Interface.PreUp: Here we add NAT forwarding rules to iptables to forward traffic coming in on the Chia port 8444 through wireguard to the home server. The second entry makes any traffic coming out of the home server to appear like its origin is the cloud server.
• Interface.PostDown: these remove the rules from iptables when wireguard is stopped. The should match the PreUp rules, only specifying -D (delete) instead of -A (add).
• Peer.PublicKey: the public key we generated above on the home server.
• Peer.AllowedIPs: the Interface.Address of the home server.

Starting Wireguard

Start wireguard on each server with the /etc/wireguard/wg0.conf config. On Ubuntu:

$ sudo systemctl start wg-quick@wg0

You can check that it started successfully with:

$ sudo systemctl status wg-quick@wg0

And check that traffic is moving with:

$ sudo wg show

It should show a “latest handshake” and “transfer” amount on both servers.

Verifying it worked

To test that traffic is being routed correctly from the cloud server to your home server, you can use telnet on some other machine (even on your home network - remember, as far as everyone else is concerned, your home server is now indistinguishable from your cloud server):

$ telnet <cloud public IP> 8444
Trying <cloud public IP>...
Connected to <cloud public IP>.
Escape character is '^]'.

On your home server, you can make sure outbound traffic is going through wireguard with traceroute, e.g.:

$ traceroute node.chia.net
traceroute to node.chia.net (54.185.56.181), 30 hops max, 60 byte packets
 1  10.10.92.2 (10.10.92.2)  45.708 ms  45.689 ms  45.669 ms
 2  * * *
 3  104.156.235.33 (104.156.235.33)  75.545 ms  75.507 ms  75.508 ms
...

We see the first hop is to 10.10.92.2, which is the internal wireguard IP address for the cloud server, which shows it’s working!

You could also fire up a Chia client on some other machine, and manually connect to your server. If it connects and shows ports 8444/8444 then it’s working.

Happy farming! You can reach me on Keybase with questions. If you happen to strike it rich on Chia, and feel this helped, you can send me a tip: xch1ahlrrfs7034afqs72gplrzj3uu4gpyhvsxmzz4u59r544u53rqmqjlyplx.