Will Syncing Sink Chia?

Update 2021-05-21: just as I was publishing this, the Chia devs released 1.1.6, which includes huge syncing improvements, both for the node and the wallet. In tests, a new wallet went from syncing at 6000 blocks/hr to 40,000 blocks/hr. This means a new wallet can now sync in just a few hours. Amusingly it also fixes the wallet corruption problem I encountered, which frustrated me enough that I wrote this article.

There’s been a lot of grousing on r/Chia and Keybase chat and issues on GitHub lately about syncing problems. Generally these fall into three categories:

1. My node never or takes a long time to sync
2. My wallet never or takes a long time to sync
3. GIB SYNC PLS

Node sync

If your node is unsynced, you can’t farm. If you can’t farm, you can’t win any XCH. So this is a major problem for people. By and large I think this is an issue with people who don’t have port 8444 exposed. Either they haven’t bothered to try, did some minimal steps but haven’t checked (or don’t really know how to check), or are unable to due to CGNAT or some other upstream firewall. See my article on escaping CGNAT for setting up a VPN if you’re in the latter situation. With my VPN I have no problem getting or maintaining peers, typically maintaining 50-60 (with target_peer_count: 60) within 15 minutes of a node restart.

However, I’m starting from a recently-synced node. The blockchain DB is currently about three GiB, which can understandably take a long time to transfer if one were starting from height 0. Peer quality will be a huge factor; so long as the “peak height” is going up, the node will by synced eventually. Maybe someone else has the motivation to test this out and monitor it; I currently don’t.

There’s also tinfoil-hat speculation on bad actors seeding “bad” nodes to keep competition off the net. I tend to go with Hanlon’s Razor: “never attribute to malice that which is adequately explained by stupidity.” While I do occasionally see nodes connected with low block height, generally they’re synced or disconnected next time I check.

But never discount malice…

Wallet sync

In order to see your available funds, or send funds, you need a synced wallet (farming does not require a synced wallet). This is different from (but often conflated with) having a synced node.¹ Each wallet maintains a separate database, which has to be built block-by-block from the blockchain database (sent from the full node). I hadn’t encountered any issues with this, until recently when my wallet database got corrupted, showing status as “synced” but showing a transaction as “pending”, while chia explorer showed it had completed hours earlier. The general advice when this happens is “delete wallet database and re-sync”. Little did I know what I was in for…²

Syncing a wallet from height 0 to 300000 took me about 48 hours, at a rate of around 6000 blocks per hour.³

At this rate, in one year it will take almost two weeks to sync a new wallet - this is a major issue for everyone. Best practice is to send farming rewards to a cold wallet; want to transfer XCH out of there? Sure, get a node synced and wait a couple weeks for the wallet to sync (during which time an attacker could pull the key off your machine).⁴ Create a new Chia wallet and participate in the future of “green” money (and wait a couple weeks before you can use it).

While I haven’t looked at the code yet (famous last words), I imagine (read: hope) this can be improved, maybe by working directly off the main blockchain database. Maybe you’ll just have to let an exchange handle your wallet for you, for a price.

The bigger issue for me was the wallet getting corrupted, requiring a complete re-sync. After that experience I’ve started making daily backups of my blockchain and wallet databases. Backups are made by rsync to a mirrored ZFS pool (my media collection), which keeps nightly snapshots so I don’t have to worry about the backup becoming corrupted. This in turn will get backed up offsite at my regular schedule. I tried using the databases directly from ZFS, but the performance was terrible. Probably how it was setup, maybe the drive speed, but in any case doing a nightly rsync from my SSD system disk was faster than trying to debug that further, and syncing one day from the network (with good peers) will take under an hour, which I can live with.

PS: If you made it this far, you already know the answer: no. Well, maybe. But probably not. In the meantime, stay vigilant, stay defensive (with backups), and keep your farm, ugh, farming.