How-to: Monitoring Arbitrum One

By Nick Haskins (Sr. Infrastructure Engineer) and Brad Larson (Sr. Software Engineer)

We’re excited to share that Unit 410 recently became a defensive validator on Arbitrum One! Arbitrum One is a layer-2 Ethereum rollup that’s able to more scalably process ethereum transactions. Unit 410 is one of the most trusted engineering teams in crypto and brings years of experience launching, securing and monitoring novel networks like Arbitrum.

Because Arbitrum One relies on a single sequencer to process transactions, the presence of high-quality defensive validators protects the integrity of the network. Critical to safely operating on novel networks is the ability to confidently monitor the chain, which we’ll explore in this post.

To ensure infrastructure remains healthy, redundant sources of data are key to monitoring, including logs and metrics. Where these sources of data are internal, additional external or on-chain data is also required to confirm things are healthy.

However, monitoring a defensive validator is a bit different than a validator in a typical staking operation. Typically, you can monitor for blocks produced or signed. On Arbitrum, if everything is working correctly, a defensive validator will do nothing! How does one monitor nothing?

What Should Be Monitored?

What Should Be Monitored?

Since Arbitrum is an ETH layer 2 rollup, the defensive validator will need to interact with the Arbitrum One chain(L2) as well as the base Ethereum chain(L1). The L2 chain is where the defensive validator is allowlisted (beta only) and looking for invalid assertions on-chain. The L1 chain is where a defensive validator takes action and stakes if and when it finds an invalid assertion.

Since there will be interactions with two chains, let’s go over a few examples of what to monitor on each of them.

Ethereum Mainnet - L1 Chain

  • Are you a validator?
    • When a defensive validator finds an invalid assertion, it needs to take action. It can only take action if it is recognized as a Validator on the network. Simply running a validator does not mean you are allowlisted (beta only) or running in the defensive mode. Similarly, if the defensive validator makes a challenge and loses, it would be considered a zombie. You want to be alerted if you are not a validator, or if you are a zombie.
  • Is ETH staked?
    • When the defensive validator sees an invalid assertion, it will automatically create a challenge, propose a correct block and stake its ETH. If another validator has already created a challenge, and you agree with their block, you will automatically stake on their block.

Arbitrum One - L2 Chain

  • Is the validator synced?
    • Since a defensive validator needs to be monitoring for invalid assertions, you will need to ensure it’s currently synced. If it is out of sync, it will be delayed in finding the invalid assertion and this could be an indication of an underlying failure (hardware, network, etc).
  • Is the validator validating blocks?
    • In order to detect an invalid assertion, a defensive validator must validate each block and ensure it matches what is expected.

How To Monitor?

How To Monitor?

Networks generally provide a number of ways to query metrics (RPC, smart contracts, Prometheus metrics, etc). Arbitrum provides a variety of smart contracts on L1 and L2 as well as JSON RPC methods that can be used to get the information a defensive validator’s monitors need. In order to use them, it will also need an L1 and L2 node to query.

Pre-reqs

  • Typescript
  • Web3 SDK
  • Fetch
  • L1 & L2 Nodes

For the purpose of this example the monitors will be built in Typescript using Fetch and the Web3 SDK.

Last Validated Block Monitor

In order to find what Arbitrum RPC methods are available, you first look at the implementation. One valuable RPC is the latestValidatedBlock which you can use to determine if we’re stuck, or processing as normal.

For this monitor you are going to create a JSON-RPC 2.0 request using fetch to query the arb_latestValidatedBlock method like so:

fetch(
  new Request(VALIDATOR_ENDPOINT_URL, {
    method: 'POST',
    body: JSON.stringify({
      jsonrpc: '2.0',
      method: 'arb_latestValidatedBlock',
      params: [],
      id: 1,
    }),
    headers: {
      'Content-Type': 'application/json'
    },
  })
).then((response) => response.json())
.then((json) => {
  log.warn(json.result)
})

{
 "jsonrpc": "2.0",
 "id": 1,
 "result":"0x77d503489a074253b4ed377f8c37b94fa58d8d1d325495a9e322f3989823e0ec"
}

This is going to warn for every block it validates though, which isn’t very useful. One of the ways you can make this useful is to alert when the validator hasn’t validated a new block in n minutes like so:

...
.then((json) => {
  if (stateLastValidatedBlock != json.result) {
    stateLastValidatedBlockTimestamp = Now();
    stateLastValidatedBlock = json.result;
  } else if (( Now() - stateLastValidatedBlockTimestamp) > 10m ) {
    log.warn("The validator hasn't validated a block in over 10 minutes")
  }
})

Is Syncing Monitor

Is Syncing Monitor

For the remainder of the code examples you will be using Web3.js.

const apiL1 = new Web3(new Web3.providers.HttpProvider(L1_URL));
const apiL2 = new Web3(new Web3.providers.HttpProvider(L2_URL));

This will be a pretty straightforward example using the isSyncing method, which is a predefined ETH contract in Web3 that Arbitrum supports.

api.eth.isSyncing().then((syncing) => {
    //syncing is an Object|Bool=false
    const isSyncing = syncing === false ? false : true;
    if (isSyncing) {
      log.warn(`${name}: L1 data isSyncing : ${isSyncing}`);
    }
  })
  .catch((error) => {
    log.error(`${name}: Error occurred while checking isSyncingL1: ${error}`);
  });

Since you are using a predefined method, all of the heavy lifting is done for you, but what if you want to use a contract not defined in Web3.eth? While it’s a little more complicated, it can still be done as demonstrated below.

Is Validator Monitor

Now let’s go over using custom contracts not defined in Web3. You will use the web3.eth.Contract method which takes an AbiItem[] and an address to send the contract to. Wait, what’s an AbiItem?

Smart contracts have an Application Binary Interface that is used to communicate with them. An AbiItem is simply the ABI of the smart contract you are calling. Since one will not know the ABI for isValidator, you need to find that first and then build an ABI for it.

If you go to the ArbRollup address on EtherScan, you will find a contract and a method called isValidator. If you click on the method, it will expand showing the inputs and outputs. The isValidator method takes an input of address and returns a bool. With that information you can construct the following JSON representation of the isValidator ABI.

isValidator - ABI

[
  {
    "name": "isValidator",
    "type": "function",
    "inputs": [
      {
        "internalType": "address",
        "name": "",
        "type": "address"
      }
    ],
    "outputs": [
      {
        "internalType": "bool",
        "name": "",
        "type": "bool"
      }
    ]
  }
]

Now that you have the isValidator ABI, you can create the contract using it and the Arb Rollup address.

const ArbRollUpAddress = '0x45e5cAea8768F42B385A366D3551Ad1e0cbFAb17';
const abi = readFileSync('isValidatorAbi.json', 'utf-8');
const abiJson = JSON.parse(contractJson);
const isValidatorContract = api.eth.Contract(abiJson as AbiItem[], ArbRollUpAddress);

All that’s left is building the batch request and handling the result. In this case, you are going to log.warn() if you are not a validator.

isValidatorContract.methods
  .isValidator(address)
  .call()
  .then((value: boolean) => {
    log.info(`${name}: isValidator: ${value} `);
    if (value == false) {
      log.warn(`${name}: isValidator: ${value} `);
    }
  })
  .catch((error: Error) => {
    log.error(`${name}: Error occurred while checking isValidator: ${error}`);
    return;
  });

The .isValidator(address) line of code is the method name defined in your ABI with the expected inputs. You will see this change in the following example.

Amount Staked Monitor

Amount Staked Monitor

When a defensive validator finds an invalid assertion, it will automatically stake against another peer with the correct assertion. Let’s write a monitor for when ETH is staked.

You will once again get the ABI for amountStaked from the ArbRollup address on EtherScan. From there you will construct the ABI.

amountStaked - ABI

 [
   {
     "type": "function",
     "name": "amountStaked",
     "inputs": [
       {
         "internalType": "address",
         "name": "",
         "type": "address"
       }
     ],
     "outputs": [
       {
         "internalType": "uint256",
         "name": "",
         "type": "uint256"
       }
     ]
   }
 ]

Now you just need to create the contract

 const ArbRollUpAddress = '0x45e5cAea8768F42B385A366D3551Ad1e0cbFAb17';
 const abi = readFileSync('amountStakedAbi.json', 'utf-8');
 const abiJson = JSON.parse(abi);
 const amountStakedContract = api.eth.Contract(abiJson as AbiItem[], ArbRollUpAddress);

and send the request.

 amountStakedContract.methods
   .amountStaked(address)
   .call()
   .then((amountStaked: number) => {
     if (amountStaked > 0) {
       log.warn(`${name}: Amount Staked: ${amountStaked} `);
     }
   })
   .catch((error: Error) => {
     log.error(`${name}: Error occurred while checking getAmountStaked: ${error}`);
   })

Whats Left?

If you recall, you only ever log.warn() when you find something you wanted to be notified of. This is generally where you would add your internal or third-party Incident Management software to create an alert for you to investigate. You might also look into throttling these alerts to avoid duplication and spamming.

Also, only a handful of monitors were implemented. There are plenty of contracts you can take advantage of to add additional monitoring. Listed below are some of the contracts you might find useful.

Arbitrum Rollup - L1

  • amountStaked
  • isZombie
  • isPaused
  • getCurrentChallenge
  • lastStakeBlock
  • isStakedOnLatestConfirmed

Arbitrum Challenge Manager - L1

  • totalChallengesCreated

Arbitrum Info - L2

  • arbOsVersion
  • arbChainId

Conclusion

Once you find all of the information you need, building out contracts and RPC calls becomes straightforward. Finding this information can be the hardest part and usually posting in the particular network’s real-time chat used by the network is a great way to get pointed in the right direction.

While it would be great if all of these metrics were located in one place, at least these metrics are exposed somewhere so that validator operators can sleep better at night.

Find this interesting? We are currently hiring for multiple positions in various locations and remote.