The Importance of Blockchain Data and the Rise of Indexers

Data is the core of Blockchain technology and the foundation for the development of decentralized applications (dApp). Although current discussions are mainly focused on data availability (DA), data accessibility is equally important yet often overlooked.

In the modular Blockchain era, DA solutions have become an indispensable part. These solutions ensure that all participants can access transaction data, achieve real-time verification, and maintain network integrity. However, the DA layer is more like a billboard than a database, meaning that data is not permanently stored but rather deleted over time.

In contrast, data accessibility focuses on the ability to retrieve historical data, which is crucial for dApp development and Blockchain analysis. Although less discussed, it is equally important as data availability. Both play different but complementary roles in the Blockchain ecosystem, and comprehensive data management must address both issues simultaneously to support robust and efficient Blockchain applications.

Traditional Methods of Blockchain Data Retrieval

Since the birth of Blockchain, it has completely changed the infrastructure and promoted the creation of dApps in fields such as gaming, finance, and social networks. However, building these dApps requires access to a large amount of Blockchain data, which is both difficult and costly.

For dApp developers, one option is to host and run their own archive RPC nodes. These nodes store all historical Blockchain data from the beginning, allowing for full access to the data. However, maintaining archive nodes is costly, and their query capabilities are limited, making it difficult to query data in the format that developers need. While running cheaper nodes is an option, these nodes have limited data retrieval capabilities, which may hinder the operation of dApps.

Another approach is to use commercial RPC node providers. These providers are responsible for the costs and management of the nodes and provide data through RPC endpoints. Public RPC endpoints are free but have rate limits that may affect the user experience of dApps. Private RPC endpoints offer better performance by reducing congestion, but even simple data retrieval requires a lot of back-and-forth communication, making them inefficient for complex data queries. Furthermore, private RPC endpoints are often difficult to scale and lack cross-network compatibility.

Blockchain Indexer: A Better Alternative

The blockchain indexer plays a key role in organizing data on the chain and sending it to the database for easy querying, which is why it is often referred to as the "Google of the blockchain." They index blockchain data and make it available through APIs such as GraphQL using a SQL-like query language (. The indexer provides a unified interface for querying data, allowing developers to quickly and accurately retrieve the information they need using a standardized query language, greatly simplifying the process.

Different types of indexers optimize data retrieval in various ways:

1. Full Node Indexer: Runs a full Blockchain node and directly extracts data, ensuring data completeness and accuracy, but requires a large amount of storage and processing power.

2. Lightweight Indexer: Relies on full nodes to fetch specific data as needed, reducing storage requirements but may increase query time.

3. Dedicated Indexer: Optimize retrieval for specific use cases for certain types of data or specific blockchains, such as NFT data or DeFi transactions.

4. Aggregator Indexer: Extracts data from multiple blockchains and sources, including off-chain information, providing a unified query interface, which is particularly useful for multi-chain dApps.

The Ethereum alone requires 3TB of storage space, which continues to increase as the Blockchain grows. The Indexer protocol deploys multiple indexers to efficiently index and rapidly query a large amount of data, which cannot be achieved by RPC.

The indexer also allows for complex queries, easy data filtering, and post-analysis extraction. Some indexers can aggregate multi-source data, avoiding the need to deploy multiple APIs in multi-chain dApps. By being distributed across multiple nodes, the indexer provides enhanced security and performance, while RPC providers may experience interruptions and downtime due to their centralized nature.

Overall, compared to RPC node providers, indexers improve the efficiency and reliability of data retrieval while reducing the cost of deploying a single node. This makes Blockchain indexer protocols the preferred choice for dApp developers.

![Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-16396b955382c2c74010c264affdca46.webp(

Indexer Use Case

Building dApps requires retrieving and reading Blockchain data to operate services. This includes various dApps such as DeFi, NFT platforms, games, and even social networks, as these platforms need to read data first in order to execute other transactions.

DeFi

DeFi protocols require different information to quote specific prices, rates, fees, etc. Automated Market Maker )AMM( needs price and liquidity information from the liquidity pool to calculate swap rates, while lending protocols need utilization rates to determine lending rates and the debt ratio for liquidation. It is essential to input information into the dApp before calculating the rates executed by users.

Game

GameFi requires rapid indexing and access to data to ensure a smooth gaming experience for users. Only through lightning-fast data retrieval and execution can Web3 games compete in performance with Web2 games, thereby attracting more users. These games need data such as land ownership, in-game token balances, and in-game operations. By using indexers, they can better ensure a stable data flow and stable uptime to guarantee a perfect gaming experience.

NFT

NFT markets and lending platforms need to index data to access various information, such as NFT metadata, ownership and transfer data, royalty information, etc. Quickly indexing such data can avoid browsing each NFT one by one to find ownership or NFT attribute data.

Whether it's a DeFi AMM that requires price and liquidity information, or a SocialFi application that needs to update new user posts, the ability to quickly retrieve data is crucial for the normal operation of dApps. With the help of an indexer, they can efficiently and accurately retrieve data, thus providing a smooth user experience.

Analysis

The indexer provides a method to extract specific data from the raw Blockchain data ), including smart contract events in each Block (. This offers the opportunity for more specific data analysis, thereby providing comprehensive insights.

For example, perpetual trading protocols can identify which tokens have high trading volumes and which tokens incur fees, thereby determining whether to list these tokens as perpetual contracts on their platform. DEX developers can create dashboards for their products to gain insights into which liquidity pools offer the highest returns or the strongest liquidity. They can also create public dashboards that allow developers to freely and flexibly query any type of data to be displayed on the charts.

Due to the availability of multiple blockchain indexers, identifying the differences between indexing protocols is crucial to ensure that developers choose the indexer that best suits their needs.

Blockchain Indexer Overview

The Graph

The Graph is the first indexing protocol launched on Ethereum that makes it easy to query previously inaccessible transaction data. It uses subgraph definitions and filters to collect subsets of data from the Blockchain, such as all transactions related to a certain DEX USDC/ETH pool.

Using index proof, indexers stake the native token GRT for indexing and querying services, and delegators can choose to stake their tokens here. Curators can access high-quality subgraphs to help indexers determine which subgraphs to index data for in order to earn the best query fees. In the transition to greater decentralization, The Graph will ultimately cease its hosting services and require subgraphs to upgrade to its network, while also providing upgraded indexers.

Its infrastructure brings the average cost per million queries to $40, which is much lower than the cost of self-hosted nodes. By using file data sources, it also supports parallel indexing of both on-chain and off-chain data for efficient data retrieval.

The Graph's indexer rewards have steadily increased over the past few quarters. This is partly attributed to the increase in query volume, but it is also related to the growth in token prices, as they plan to integrate AI-assisted queries in the future.

![The Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-53dbb4fd659cf6a7184990c886901658.webp(

Subsquid

Subsquid is a peer-to-peer, horizontally scalable decentralized data lake that efficiently aggregates large amounts of on-chain and off-chain data, and protects it through zero-knowledge proofs. As a decentralized worker network, each node is responsible for storing data from a specific Block subset, speeding up the data retrieval process by quickly identifying the nodes that hold the needed data.

Subsquid also supports real-time indexing, allowing for indexing before the block is finalized. It supports storing data in formats chosen by developers, facilitating easier analysis using tools like BigQuery, Parquet, or CSV. Additionally, subgraphs can be deployed on the Subsquid network without migrating to the Squid SDK, enabling no-code deployment.

Despite still being in the testnet phase, Subsquid has achieved impressive statistics, with over 80,000 testnet users, deployed over 60,000 Squid indexers, and more than 20,000 verified developers on the network. Recently, Subsquid launched the mainnet of its data lake.

In addition to indexing, the Subsquid Network data lake can also replace RPC in use cases such as analytics, ZK/TEE co-processors, AI agents, and Oracles.

SubQuery

SubQuery is a decentralized middleware infrastructure network that provides RPC and indexing data services. It initially supported the Polkadot and Substrate networks and has now expanded to include over 200 chains. Its operation is similar to The Graph, which uses indexing proofs, where indexers index data and provide query requests, and delegators stake their shares to the indexers. However, it introduces consumers to submit purchase orders to indicate that the income of the indexers is guaranteed, rather than managed by administrators.

It will introduce SubQuery data nodes that support sharding to prevent continuous synchronization of new data between each node, thereby optimizing query efficiency and moving towards greater decentralization. Users can choose to pay a computing fee of about 1 SQT token for every 1000 requests, or set custom fees for the indexer through the protocol.

Although SubQuery launched its token earlier this year, the issuance rewards for nodes and delegators have also increased in dollar value on a month-to-month basis, which represents a continuous increase in the number of query services offered on its platform. Since the TGE, the total amount of staked SQT has increased from 6 million to 125 million, highlighting the growth of network participation.

Covalent

Covalent is a decentralized indexing network, created by block sample producers )BSP( network nodes through batch exporting to create copies of blockchain data, and publish proofs on the Covalent L1 blockchain. This data is then refined by block result producers )BRP( nodes according to established rules to filter out the required data.

Through a unified API, developers can easily extract relevant Blockchain data in a consistent request and response format, without the need to write custom complex queries to access the data. The CQT token, settled on a certain platform, can be used as a means of payment to extract these pre-configured datasets from network operators.

The rewards of Covalent seem to show an overall upward trend from the first quarter of 2023 to the first quarter of 2024, partly due to the rise in the price of Covalent token CQT.

![The Development of Web3 Data Access: Introduction to Indexers and Related Projects])https://img-cdn.gateio.im/webp-social/moments-52ee29205aa307720198994a5f3de61f.webp(

Considerations for Choosing an Indexer

Customizability of Data

Some indexers ) like Covalent ( are general-purpose indexers that provide standard pre-configured datasets via API. While they may be fast, they do not offer the flexibility for developers who need custom datasets. By using the indexer framework, it allows for more custom data processing to meet application-specific requirements.

Security

Index data must be secure; otherwise, dApps built on these indexers are also vulnerable to attacks. For example, if transactions and wallet balances can be manipulated, the dApp may lose liquidity, thereby affecting its users. Although all indexers adopt some form of security by staking tokens, other indexer solutions may use proofs to further enhance security.

Subsquid provides options using optimistic and zero-knowledge proofs, while Covalent has also released proofs that include block hashes. The Graph offers an optimistic challenge window for indexer queries.