Day in the Life of a Substrate Extrinsic

By Elliot Cameron, Crytocurrency Infrastructure Engineer

The Polkadot ecosystem pioneered Substrate, a framework for building blockchains of nearly any kind. While impressively flexible, Substrate also enables protocols to share source code and tooling, helping new chains to develop and ship quickly because so much existing infrastructure can be reused. Flexibility and reusability do not usually mix well, making Substrate an impressive solution.

Substrate manages this flexible reusability by defining and building around a few core concepts, like runtimes, pallets, extrinsics, and storage, to name a few. The most common way that developers interact with these core concepts is through a Rust SDK called FRAME.

In this post, we’ll do a deep dive into one of those core, engineering concepts in Substrate: the extrinsic. We’ll briefly touch on some of the others along the way.

What is an Extrinsic?

As the name implies, “extrinsic” is a broad term referring to anything that comes from outside the blockchain—anything not derived from the existing chain state alone. Extrinsics are included in blocks to form new states on the chain. Many non-Substrate chains call these “transactions,” but Substrate aptly chose a generic term because the concept can refer to anything that can be included in a block.

While extrinsics come in a few types (you can read more about them here), the type that users interact with most is, unsurprisingly, the “signed transaction” extrinsic. Other extrinsic types are generated automatically by validators as they build blocks, but “signed transaction” extrinsics come directly from users of the system–for example, you.

The Lifecycle of an Extrinsic

Such extrinsics start out as intentions in your mind. Do you want to use your tokens, stake your tokens, contribute your tokens to a crowdloan? Your choices likely depend on which Substrate chain you’re using. Let’s say you want to stake your tokens. Now what? You need to tell the chain! That involves three steps:

1. encoding the action you want to take,
2. signing that action with your secret key,
3. submitting the extrinsic to a node on the chain.

These three steps must be performed by you, the user. After that, the extrinsic continues its journey like this:

1. The node gossips the extrinsic to other nodes and validators.
2. Eventually a validator includes your extrinsic in a new block.
3. Some chains have a final step called “finalization” which guarantees the block that included your extrinsic cannot be revoked. At that point, you can be completely certain that your intent has been realized.

Not all chains work the same way, but this is a common lifecycle, including the one found in Polkadot.

Again, in this lifecycle, the first three steps must be performed by you, the user. Let’s explore each step.

Encoding Your Intent

When telling a Substrate chain what you want to do, you interact with several core concepts that exist for all Substrate-based chains: calls, pallets, and runtimes.

Diagram 1: Structure of Substrate Runtime

A “call” is like invoking a function in programming. Functions take some number of inputs/arguments (that number might be zero) and perform an action. An example call might be “transfer(from, to, amount)”. This takes three arguments and sends an amount from one address to another.

A pallet is a grouping of calls and other stuff (like storage, events, etc.). For example, the “staking” pallet has storage, events, and calls for managing delegated stake. Another pallet might deal exclusively with staking. Pallets allow Substrate developers to share building blocks that might be used in many chains.

A runtime is a grouping of pallets and other information about the functionality supported by the Substrate chain. The runtime is the final description of what a chain can do and how it works. However, because chains tend to evolve over time, Substrate makes it easy for chains to change their runtime, allowing forkless upgrades as developers add new ideas, and remove old ones.

To summarize in reverse: At a specific moment in time, a Substrate chain has a runtime. That runtime has pallets, and each pallet supports calls. Some Substrate chains (Polkadot, for example) have many runtimes throughout their history, but only one of them is active at a time.

Diagram 2: A Substrate Runtime Changing Over Time

Once you lock in your intent you need to get the latest runtime metadata being used by the chain, ask what pallets it supports, and find the call in those pallets corresponding to your intent. In our example of staking your tokens, on Polkadot that would correspond to the “bond” call in the “staking” pallet, commonly written as “staking.bond”. You can see this visually in https://polkadot.js.org/apps. Go to the Developer menu and click Extrinsics. Then you can see the various pallets and the “calls” available in each of them.

The staking.bond “call” takes three arguments: (1) the controller address (which validator you want to bond against), (2) the amount you want to bond, and (3) where you want your staking rewards to go. Once you’ve chosen these three parameters, you have a fully-formed “call.”

Finally, you must encode this “call data.” You can see in the screenshot that Polkadot.js automatically shows your encoded form (in this case, it’s the text that starts with 0x0700…). This is a hexadecimal representation of the actual binary encoding using the SCALE codec. This is the encoding of your intent. This encoding is specific to the runtime that you used to generate it and will likely be interpreted differently if used on another chain/runtime.

SCALE Codec and Runtime Metadata

The SCALE codec was developed specifically for Substrate and is not only used to encode “calls.” It is also used to encode unsigned extrinsics, storage, and more.

The SCALE codec was designed with some very specific goals in mind. Data encoded by SCALE must be

• Sent between thousands of nodes in little time,
• Stored by thousands of nodes for long periods of time, and
• Used by small WASM-based runtimes with limited resources.

With these constraints, the SCALE codec was designed to produce dense encodings of data while also allowing for the encoding and decoding source code to be both efficient enough to run in resource-constrained contexts and small enough to be included in every Substrate runtime.

One of the effects of this design is that SCALE is not a self-describing codec. Data encoded to SCALE with a specific schema can only be decoded when that same schema is already known by the decoder.

SCALE offers some nice tools to help with this in the Rust ecosystem. Programs can very easily derive trait implementations for encoding and decoding most Rust data types. This makes working with SCALE in that context quite easy.

The larger challenge is communicating the knowledge that Rust has about those types to the broader world. To accomplish this, FRAME includes a full description of all types in each runtime’s metadata. Metadata accompanies each WASM runtime as simple data, not code. Part of that metadata is a description of all the types, “calls,” events, and storage elements that the runtime supports.

To see the latest metadata encoded as JSON, you can use a tool like subwasm. For example, after installing subwasm, run the following shell command.

subwasm metadata --chain polkadot --json

This produces a huge JSON output that includes all the information in the latest Polkadot runtime’s metadata.

The snippet below shows an example of how the metadata describes one the runtime’s calls, transfer_all.

{
  "name": "transfer_all",
  "fields": [
    {
      "name": "dest",
      "type": 188,
      "typeName": "<T::Lookup as StaticLookup>::Source"
    },
    {
      "name": "keep_alive",
      "type": 60,
      "typeName": "bool"
    }
  ],
}

From this little snippet we can learn that the “transfer_all” call has two arguments, dest and keep_alive. “dest” has a type that has something to do with a Source and keep_alive is a bool. If you lookup type 188 in the metadata, you’ll see how Source is defined.

With this kind of structure, any programming language can read the metadata and encode SCALE data that would be understood by the runtime or decode data generated by the runtime.

Signing Your “Call Data”

Once you’ve SCALE encoded the “call data” you want to perform on the chain, you need to sign it. Signing involves two conceptual steps: constructing a signer payload and producing a signature.

Nodes rely on the runtime’s metadata to agree on encoding and decoding of a signer payload. This payload is the input to the cryptographic signing function that produces the final signature.

Unsurprisingly, the signer payload uses the SCALE codec, and is not merely the same “call data” that you used to encode your intent. It also includes information that prevents replay attacks or reusing the same “call” on a different chain. This “extra information” is made up of signed extensions which are defined in the runtime and described in the metadata.

Using subwasm, you can see the signed extensions needed on Polkadot’s latest runtime like this:

subwasm metadata --chain polkadot --json | jq '.V14.extrinsic'

On Polkadot, these signed extensions include things like,

• the extrinsic nonce for the signing address,
• the full hash of the genesis block for Polkadot,
• the mortality information (specifying when the signature expires),
• and the fee.

Many of these are critical elements that must be included in the signing process to keep the chain safe. The nonce, for example, prevents replay attacks, the genesis hash prevents extrinsics from being reused on another chain, and mortality makes extrinsics expire after a given amount of time if they aren’t included in a block by then.

Security Note: Keep in mind that on some chains there is an “existential deposit” for an account. If an account’s balance drops below that amount, all of its on-chain state will be cleared, including its nonce. This means it’s theoretically possible to replay extrinsics against such accounts. Most extrinsics also expire due to “mortality,” mitigating this risk. “Immortal” extrinsics are possible, but highly discouraged for this reason.

The precise binary layout of the signer payload incorporates all of this information and, once SCALE-encoded, might have a shape like this:

<extrinsic version><call><signed extensions><additional signed>

<additional signed> here is a bit confusing. For each signed extension, the signer payload also has some “additional” data that goes with it. So if there were 10 signed extensions, then there would be 10 entries in <signed extensions> followed by another 10 entries in <additional signed>. Confusingly, some of these entries have zero-byte representations in the final SCALE-encoded result and none of them are cryptographically signed in this state. The name indicates that they are part of the input to signing.

The Polkadot.js web UI does not show you the SCALE-encoded signer payload. It creates that on the fly when you click “Submit Transaction.”

Cryptographically signing this payload depends on the chain but is usually straightforward. Most Substrate chains prefer the sr25519 signing scheme, but not all. Moonbeam, for example, uses the same ECDSA-based signing scheme as Ethereum.

The signer payload is first hashed (in some cases, twice) and handed to the appropriate signing function to get the final signature. This signature is then combined with the address and signed extensions to form the final signed extrinsic, which is, naturally, SCALE-encoded (e.g. here).

You now have the data needed to submit your extrinsic to the chain.

Submitting to the Chain

After that, the final step is easy. Connect to a node on the chain and submit an author.submitExtrinsic RPC (remote procedure call). You can even do this through Polkadot.js.

Once received, the node will gossip your extrinsic to the validators who will, hopefully soon, include your extrinsic on a block!

Using a tool like subscan, you can see some example extrinsics that were included on a randomly selected Polkadot block.

Calling it a Day

Your extrinsic is now immortalized in the history of the blockchain and now you know how part of that process works!

At Unit 410, we work with many different blockchain projects and often have to support technologies that aren’t yet mature. Adding support for these projects can be time consuming and difficult. However, understanding Substrate’s design for extrinsics has been abnormally rewarding. Leveraging Substrate has allowed us to onboard some of those projects with minimal changes and without sacrificing any confidence in the quality of our work. We’re excited about the categorical improvements Substrate brings to the blockchain ecosystem and look forward to working with it whenever we get the chance!

If you are interested in learning more about us, feel free to reach out to us: Security, Job Board.

Appendix: Helpful Tools

Some tools we’ve found that make Substrate-oriented development pleasant are listed below:

• Polkadot.js and its various JavaScript libraries. Despite the name, this is the most widely used SDK for building on top of Substrate-based chains, not just Polkadot.
• Desub is a Rust library for parsing call data, signer payloads, and signed extrinsics. It is very flexible as it relies only on knowledge of the SCALE codec and some Substrate/FRAME primitives. It works completely offline and requires that you give it the metadata at runtime. We’ve made several contributions to this library ourselves!
• Subwasm is helpful when exploring runtimes and metadata.