Improving IDLs Discoverability: Accelerating Solana Development, Integration and Composability

Introduction

Solana has garnered the reputation as the fastest and most scalable blockchain on the the market, leveraging its unique architecture to achieve high throughput and low latency. However, associated with a massive growth of this scale, is an important component of the ecosystem that requires constant attention: Developer Experience (DX). Improving the developer experience is always in the spotlight as developers and builders are the lifeline of Solana.
IDLs discovery and implementation for existing infrastructure appears to be a problem many developers in Solana presently struggle with. Two main concerns are predominantly echoed:

• Reading data on the network is difficult and even more so for non-anchor programs
• Obscurity in CPI with programs as security concerns

In this article, we will explore what Interface Definition Language (IDL) is all about, and how they are a vital part of improving the development experience on Solana. This article aims to provide a comprehensive understanding of the role IDLs play across the Solana development landscape, and why further understanding into upgrading its discoverability will bolster and accelerate development experience on Solana.

Interface Description Language(IDL)

An Interface Definition Language (IDL) in Solana refers to the public interface of a program. It represents the structure of programs accounts, instructions as well as error codes. They are usually in a JSON and Typescript file format and they are used to generate client-side code enabling clients to easily interact with a Solana program. They are automatically generated for programs written in anchor. IDLs can be compared to ABI files for EVM, which gives developers easy understanding of program structures.

Why are IDLs very useful

While the applications of IDLs can be enumerated to varying use cases, IDLs are very useful for the main reasons:

• They make program interaction human-readable: Through IDLs, we can demystify the concatenation of transactions and accounts as just a bunch of bytes and provide developers a medium to easily parse instructions or accounts to make sense of what the program is doing. This attribute of IDLs enables indie and professional developers to get smooth integration on explorers, consumer UIs, and data platforms (like Dune).Below is an example of a parsed transaction instruction on Solana:

Here is an example of how Solana explorer display public IDLs:

• Providing Standard Interfacing: By leveraging a standard interface, tools and frameworks can take advantage of this to generate instructions to interact with a program more easily, therefore enabling easier composability between program interfaces, making building client-side code far simpler (A good example would be the case of embedded wallet providers that whitelist specific ix discriminant). Below, we have an example of two Hello World frontends that create a greeting account, one using Native Solana and the other using Anchor:

Using Native Solana:

const GREETING_SEED = 'hello'
const greetedPubkey = await PublicKey.createWithSeed(
    payer.publicKey,
    GREETING_SEED,
    programId
)

// Check if the greeting account has already been created
const greetedAccount = await connection.getAccountInfo(greetedPubkey)
if (greetedAccount === null) {
    console.log('Creating account', greetedPubkey.toBase58(), 'to say hello to')
    const lamports = await connection.getMinimumBalanceForRentExemption(
        GREETING_SIZE
    )

    const transaction = new Transaction().add(
        SystemProgram.createAccountWithSeed({
            fromPubkey: payer.publicKey,
            basePubkey: payer.publicKey,
            seed: GREETING_SEED,
            newAccountPubkey: greetedPubkey,
            lamports,
            space: GREETING_SIZE,
            programId,
        })
    )
    await sendAndConfirmTransaction(connection, transaction, [payer])
}
// ...
export async function sayHello(): Promise<void> {
    console.log('Saying hello to', greetedPubkey.toBase58())
    const instruction = new TransactionInstruction({
        keys: [{ pubkey: greetedPubkey, isSigner: false, isWritable: true }],
        programId,
        data: Buffer.alloc(0), // All instructions are hellos
    })
    await sendAndConfirmTransaction(
        connection,
        new Transaction().add(instruction),
        [payer]
    )
}

Source: https://github.com/solana-labs/example-helloworld/blob/master/src/client/hello_world.ts

Using Anchor:

const anchor = require('@project-serum/anchor')

async function greet(message: string) {
    anchor.setProvider(anchor.Provider.local())
    // Here we load the HelloAnchor example IDL
    const program = anchor.workspace.HelloAnchor
    let greetKey = Keypair.generate()
    await program.methods
        .initialize(message)
        .accounts({
            newAccount: greetKey.publicKey,
            signer: wallet.publicKey,
            systemProgram: web3.SystemProgram.programId,
        })
        .signers([wallet, greetKey])
        .rpc()
}

greet('Hello World')

By utilizing IDLs, Anchor can abstract the process so that it is much easier to read, quicker to deploy, and more composable for CPI interactions. To know more about CPIs and why they are so crucial to Solana development, refer to this article Cross-Program Invocations and PDAs- the combination of two powerful mechanism on Anchor

• Dependency-free SDKs: IDLs provide dependency-free SDKs for long-tail of Solana programs with declare_program!()

Creating IDLs

With Anchor, IDLs are automatically generated when you build because they are so integral to the entire anchor programming landscape. From the anchor project directory, enter the following in your terminal:

anchor build --idl <IDL_OUTPUT_DIRECTORY>

However by default, idl.json will be saved in ./target/idl folder

For Solana programs written in native Rust, IDLs can be generated using tooling like Metaplex Shank or Codama Kinobi

Publishing IDLs

Anchor also makes it easy to publish IDLs on-chain. To do this, you simply need to enter the following in your terminal:

anchor idl init --filepath <FILEPATH> <PROGRAM_ID> --provider.cluster <CLUSTER> --provider.wallet <WALLET>

Make sure the wallet being used is the program authority and has SOL for the on-chain transaction! After the IDL is loaded, you should be able to see the IDL on Solana Explorer at https://explorer.solana.com/address/YOUR_PROGRAM_ID/anchor-program.

The Anchor idl command has several other handy tools that enables developers to fetch, update, or freeze an IDL. For information on each of those commands, you can enter the following in your terminal:

anchor idl -h

Interfacing IDLs on the Client Side

Programs defined in your IDL are mapped to correspond to functions provided by the generated Typescript module, which is integrated into the front end of our application. The general structure of this module is outlined below:

import { Program, AnchorProvider, setProvider } from '@project-serum/anchor'
import { Connection, KeyPair } from '@solana/web3.js'
import { PROGRAM_ID, IDL } from './your-program-dir'
// where IDL is the .json created by anchor build
// and PROGRAM_ID is your on-chain program ID

export const yourFunction = async () => {
    const wallet = KeyPair.generate()
    const connection = new Connection(QN_ENDPOINT)
    const provider = new AnchorProvider(connection, wallet, {})
    setProvider(provider)
    const program = new Program(IDL, PROGRAM_ID)
    // ... your code
    // e.g. await program.methods.yourMethod(YOUR_PARAMETERS).accounts({YOUR_ACCOUNTS}).rpc();
}

Due to the program instructions and account embedded in the IDL, the code editor can automatically detect callable programs via program.methods... A type guard is put in place to ensure all the necessary accounts are passed into the instruction. See Intro to client-side Anchor development

Improving IDL Discoverability

The majority of programs use Anchor today, and for non-Anchor frameworks, codama coupled with verified builds can fill the gaps. These IDLs provide efficient reading and a starting point for calling programs with declare_program!() but they are currently not discoverable. It is not yet ascertained whether IDLs can cover all possibilities on programs however the majority can be covered.

Current solutions being proposed to improve discoverability have to satisfy the following requirements:

• Must allow a generalized reading from the network.

• The time taken to understand how to implement CPI into other programs should be significantly reduced.

• Program discovery should be as frictionless as possible.

Conclusion

We have explored IDLs and their numerous benefits. We have also examined IDLs in Anchor and non-Anchor frameworks and how developers can utilize them to improve program composability.

However, existing infrastructure does not have versioned IDLs based on deployment time, which makes it difficult for developers concerned with consistent data integration to understand what IDL to use at what slot.

Improving IDL discoverability has huge benefits; developers can easily integrate CPI into programs, which expands composability within the general developer landscape, unlocking new use cases that may not have been thought of.

Discoverable IDLs also provide more insight into security deficiencies as even seasoned developers on Solana tend to secure their program through obscurity which can often prove to be a bug rather than a feature.

Over the coming years, IDLs are beginning to take a place within the spotlight as targets for more investment and development to enable building on Solana a more framework and language agnostic process, therefore accelerating developer experience across all development landscapes on Solana.

Additional Resources/Further Reading

• Anchor
• Metaplex shank
• Intro to client-side Anchor development