How to Prepare for the Polygon Archive Node Limit

Due to the recent growth of the Polygon ecosystem, Polygon archive nodes operating on the GETH client are expected to pass the 16TB storage threshold. 

If you have not already upgraded your storage size, Polygon’s archive node limit may cause your archive node to fail. 

This post will walk you through why this is happening and how to prepare for the upcoming Polygon archive node limit.

When will the Polygon archive node limit be reached?

The Polygon archive node limit is expected to be reached the week of 5/23. 

The exact date and time that the 16TB threshold will be crossed depends on two things: 

1. How long the archive node has been running

2. How 16TB is calculated

Nodes that see more re-orgs use more disk space and 16TB can either be 16,000 GB or 16384GB, and this difference is roughly filled in a week.

Why increase the storage limit?

The Polygon chain has experienced a tremendous amount of growth since its initial launch in June 2020. As a result, archive node operators, which have to maintain a copy of Polygon’s entire blockchain history, will have to store all of that data on their archive node.

Although Polygon is one of the first chains to pass 16TB of data, as other chain ecosystems continue to grow they will soon face the same challenge, so we recommend reading on to know how to prepare even if you’re not actively using Polygon.

What does the 16TB storage limit mean for my Polygon archive node?

For most cloud hosting providers, a 16TB storage limit is common. Unless you’ve prepared massive disk space and infrastructure for your Polygon archive nodes, you will likely no longer be able to serve data after this limit is reached.

This means all of your archive requests will begin to fail.

Why do I need Archive data?

Archive data allows you to access the entire blockchain history, rather than just getting historical Polygon transactions. This is extremely important for displaying data like:

1. Wallet transaction history 

2. Trends and analytics 

3. Indexed data 

4. And anything else that looks at historical data

4 Ways to Prepare for the Polygon Archive Node Limit

There are a couple of options here depending on if you want to optimize for short term, medium term, or long term. We’ll walk through each one. 

1. Delete and Re-sync Nodes (Short Term)

Deleting and re-syncing your nodes will clear away data that is no longer a part of the canonical chain, including things like uncle blocks. 

It is relatively easy to delete and re-sync Polygon nodes, but is only a temporary stop-gap solution since newer data will continue to fill up the newly freed storage space. 

2. Use Logical Volume Management (LVM) or RAID0 to Manage More Storage Space (Medium Term)

LVM (and RAID0) allows you to allocate storage from multiple disks into a single filesystem to bypass any single disk’s maximum space limitations. 

This is not trivial work to complete, but it will allow node operators to scale up disk partitions on demand while continuing to use their existing EC2 instance types and EBS volume types.

Alternatively, AWS has released a new EBS volume type: io2 block express that can support 64TBs max disk space. Though, it is limited to a new r5b EC2 instance type and is generally considered “overkill” for solving these kinds of disk space issues.

3. Switch to Erigon for Your Polygon Node Client (Long Term)

Erigon is a node client option that natively supports Bor consensus which is compatible with the Polygon/MATIC blockchain. For comparison, the state data for Polygon’s Proof of Stake Mainnet is as small as 4.5TB when using Erigon.

While setting up an Erigon client for Polygon is a longer-term solution, it allows for more manageable state data.

4. Use Alchemy as your Polygon Node Infrastructure Provider

The Alchemy team works day and night to ensure that our users have a smooth, reliable experience building with Polygon APIs, and on any chain that we support. 

We handle all upgrades and breaking changes so you don’t have to. And the best part, it’s free! Get set up with Alchemy in minutes by signing up here. 

For more updates, tips, and launches follow us on twitter @AlchemyPlatform.