What is a blockchain indexer?
Written by Sankrit K
Reviewed by Logan Ross
An indexer is a tool that extracts transaction data from a blockchain node, transforms it into a machine and human-readable form, and loads it into a database or another service for easy querying.
Blockchains have a fundamental problem - their data is not searchable. In other words, on-chain data cannot be queried by default.
Being able to query on-chain data is crucial for developers to build decentralized applications (dApps) that can provide useful services to users. Without the ability to search and retrieve data from the blockchain, it becomes difficult to create efficient and user-friendly dapps that can interact with on-chain data.
This is where indexers come in.
This article will provide a complete overview of indexers, how they work, and some examples.
What is an indexer?
An indexer helps query and analyze data stored on a blockchain by providing a fast and efficient way to retrieve the required information. It may also offer additional features such as indexing smart contract events, filtering data based on specific criteria, and enabling complex queries.
It plays a critical role in building blockchain applications and services by providing easy access to on-chain data.
On-chain data is stored on a decentralized network of nodes and is accessible to anyone on the blockchain. However, accessing this data can be challenging due to the nature of blockchain technology.
By extracting, transforming, and loading on-chain data into a database or another service, an indexer provides a fast and efficient way to retrieve the required information. This allows dapps to perform tasks like tracking transactions, monitoring smart contracts, and analyzing network activity.
What are the components of an indexer?
The components of an indexer in the context of blockchain can vary depending on the specific implementation, but typically include data source, database, indexing engine, search API, and a user interface:
1. Data Source
The indexer needs to connect to the blockchain network and extract data from the blockchain. This can be done by running a full node or by using a third-party service that provides an API to access blockchain data.
2. Database
The indexer needs to store the extracted data in a database for efficient searching and querying. The database can be a traditional relational database or a NoSQL database, depending on the specific requirements of the indexer.
3. Indexing Engine
The indexer needs an engine that can efficiently process blockchain data and extract relevant information. The indexing engine needs to be designed to handle the specific data structures used in the blockchain, such as blocks, transactions, and addresses.
4. Search API
The indexer needs to provide a search API that allows users to search the blockchain data using various parameters, such as addresses, transaction IDs, or block numbers.
5. User Interface
The indexer may also include a user interface that allows users to interact with the blockchain data through a web-based interface or a command-line interface.
How does indexing work?
Indexers work by making blockchain data searchable.
As blockchain is a distributed database where all the data is stored in a linear sequence of blocks, indexers are like keys that allow a user to find and access specific data within those blocks quickly.
The following example of a land ownership contract demonstrates how indexing works.
contract LandOwnership {
struct Parcel {
uint256 parcelNumber;
string ownerName;
string legalDescription;
uint256 assessedValue;
}
mapping (uint256 => Parcel) parcels;
function addParcel(uint256 parcelNumber, string memory ownerName, string memory legalDescription, uint256 assessedValue) public {
parcels[parcelNumber] = Parcel(parcelNumber, ownerName, legalDescription, assessedValue);
}
function getParcel(uint256 parcelNumber) public view returns (string memory, string memory, uint256) {
return (parcels[parcelNumber].ownerName, parcels[parcelNumber].legalDescription, parcels[parcelNumber].assessedValue);
}
}An indexer can be used to store and retrieve information about different parcels of land on a blockchain.
Say that a smart contract manages the ownership of different plots of land, and that as developer wants to look up information about each plot by simply typing in what he wants (like a search engine) instead of having to manually scrape through all the data.
Blockchain does not allow developers to do that inherently. They need an indexer.
An indexer stores and retrieves information about land ownership quickly by creating an index of all the different plots of land. For example, if a user wants to know who owns a specific plot of land, he can query the indexer and get the answer in seconds.
Or the user could retrieve information about a particular plot using a unique identifier, such as its parcel number. This is illustrated in the code below (written in Solidity).
In this example, the Parcel struct defines the information that is to be stored for each parcel of land. Parcel mapping uses the parcel number as the index to store and retrieve information about each parcel.
The addParcel function adds new parcels to the mapping, and the getParcel function retrieves information about a particular parcel using its parcel number.
Now, the user can easily look up information about any parcel of land on the blockchain by its unique identifier, making it much easier to manage and track ownership of different parcels.
What problems do indexers solve?
Some of the key problems that indexers solve are data access, query performance, data analytics, and dapp development.
Data Access
Blockchain data is stored in a distributed and decentralized manner, which makes it challenging to access and retrieve data quickly. Tools like Alchemy Subgraphs, Helius, and Goldsky provide a way to extract relevant data from the blockchain and create an index that can be easily searched and queried.
Query Performance
As blockchain data grows, querying becomes increasingly slow and resource-intensive. Tools like Etherscan solve this problem by creating a searchable index of blockchain data that can be queried quickly and efficiently.
Data Analytics
Blockchain data is often used for data analytics, but analyzing large volumes of data directly from the blockchain can be slow and resource-intensive. Indexing tools like Dune, Nansen, and Messari provide a way to extract and index blockchain data, making it easier to analyze and visualize data.
Dapp Development
Decentralized applications (DApps) often require fast and efficient access to blockchain data. Tools like Etherscan, Hardhat, and Helius provide a way for dapp developers to access on-chain data and build applications that interact with the blockchain more efficiently.
Indexing use cases
A few specific use cases of indexing are quick data retrieval, non-permanent data storage, latency reduction, infrastructure control, and parallelization of data.
Quick retrieval of data
By creating a unique indexer, data can be retrieved quickly without relying on node requests. For example, an index can be created to fetch the transaction history of an address.
Non-permanent data storage
An indexer can store mempool history for developing arbitrage strategies, simulate transactions and their results, and maintain the gas estimation history of the local development environment.
Latency reduction
Indexing removes latency by providing a fast and efficient way to search and retrieve data from large and complex data sources.
Control over infrastructure
By creating an indexer, the developer has complete control over the infrastructure and doesn't have to depend on external services for the dApp’s data needs.
Parallelization of data
An indexer can save a lot of time when working with large datasets by parallelizing data and reducing the time needed to process it from weeks to hours.
Examples of Indexers
Alchemy provides a list of indexing tools for dApp development. Here are some examples.
Alchemy Subgraphs
Alchemy Subgraphs launched after the acquisition of Satsuma, indexing protocol for blockchains such as Ethereum. Alchemy Subgraphs enable developers to create and distribute open APIs known as "subgraphs," which can be used to query blockchain data more quickly and effectively.
Alchemy Subgraphs use GraphQL, meaning that teams building with subgraphs currently using other providers (like The Graph) can switch to Alchemy without needing to rewrite any code.
NFTScan
NFTScan is a powerful tool that offers a range of services to help users stay on top of their non-fungible token (NFT) assets.
It serves as a multichain and indexing protocol for NFTs, allowing users to access comprehensive data infrastructure and real-time information across multiple blockchains, including Ethereum, Solana, BNB Chain, Polygon, Arbitrum, and Optimism.
With NFTScan's user-friendly API and analytics tools, developers and users alike can easily track and manage their NFT assets, gaining valuable insights into their value, ownership, and activity.
Chainbase
Chainbase automatically indexes and queries blockchain data with the help of blockchain data by utilizing enriched on-chain data and streaming computing technologies.
This simplifies the process of creating intuitive UI/UX dapps to allow non-crypto natives to try it out while making sense of data patterns.
It supports a range of chains, including Aptos, Arbitrum, Avalanche, BNB Chain, Ethereum, Fantom, Optimism, Polygon, and Sui, and offers resources and guides about indexing tools and archive nodes.
How to choose an indexer for your dApp?
Choosing a proper indexer for a dApp is a crucial decision that impacts the dApp's performance and user experience.
While Alchemy's list of indexing tools can be a helpful resource for discovering potential options, there are various factors to consider when making this decision.
Here are some tips to help when choosing the best indexer for a dApp:
1. Compatibility
An indexer must be compatible with a dapp's blockchain network. Some indexing tools may only work with specific blockchain networks, such as Ethereum or Multichain. Therefore, it is important to choose an indexer that supports the blockchain network the dApp uses.
2. Functionality
An indexer should provide fast and reliable access to blockchain data to ensure smooth performance and a seamless user experience. Additionally, some indexers may offer unique features, such as support for different data formats, query languages, and APIs. It is important to consider the functionality that aligns with the DApp's specific requirements.
3. Reputation
Selecting a reputable indexing tool can provide additional benefits, such as access to reliable support, documentation, and a community of developers. Therefore, it is crucial to choose an indexer that has a good reputation and has been used successfully by other dApps.
4. Cost
Indexing tools may offer different pricing models, such as usage-based or subscription-based. Therefore, it is important to consider the cost of the indexer and choose one that aligns with the dapp's budget.
Conclusion
Indexers have become an essential component of the blockchain ecosystem as they help developers to access and analyze onchain data in a more efficient and user-friendly way.
In the past, there have been some issues with indexers, such as centralization concerns and potential data inaccuracies, which have been addressed by decentralized indexing protocols and data verification mechanisms.
Recently, there have been several exciting developments surrounding indexers, including the emergence of new decentralized indexing protocols, which offer greater privacy, security, and scalability. Additionally, indexers are being integrated with other technologies, like decentralized oracles and smart contracts, to enable more advanced use cases and applications.
As more dapps are developed and adopted, the need for efficient and reliable onchain data indexing and querying will only continue to grow. Therefore, it is likely that web3 will see further advancements in indexing technologies and protocols, as well as greater collaboration and integration with other blockchain technologies, in the coming years.
Related overviews
Learn About The Tokenization Initiatives and Why Banking Institutions are Tokenizing Their Assets
Learn About Enterprise Grade Blockchain Infrastructure to Support Your Businesses Specific Needs
Learn About The Advantages, Disadvantages, and Use Cases of Private and Public Blockchains
