Hyperlane

Run Validators

Run Hyperlane validator and relayer agents for the BitSong crescendo-1 testnet to enable cross-chain message delivery.

This guide walks you through running Hyperlane validator and relayer agents that power cross-chain message delivery between BitSong and Base Sepolia. It follows the official Hyperlane validator documentation, adapted for the BitSong testnet.

Two deployment configurations are covered:

  • Local (Testnet) — Hex private keys with localStorage checkpoint storage. Simplest to set up, suitable for testing.
  • AWS (Production) — AWS KMS signing keys with S3 checkpoint storage. Hardware-backed security, recommended for production.

Prerequisites

Before running agents, ensure you have completed the previous steps:

  1. Agent keys generated and exported as environment variables
  2. Hyperlane bridge setup completed (all 7 on-chain transactions)
  3. Docker installed and running (see install Docker below)
  4. Foundry installed (provides cast)
  5. A funded Cosmos signer account on BitSong (see agent keys guide)

Install Docker

The Hyperlane agents run as Docker containers. Install Docker Engine on your server if you don't have it already:

Terminal
# Remove any old Docker packages
sudo apt-get remove -y docker docker-engine docker.io containerd runc 2>/dev/null

# Install prerequisites
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin

After installation, add your user to the docker group so you can run containers without sudo:

Terminal
sudo usermod -aG docker $USER
newgrp docker

Verify Docker is working:

Terminal
docker --version
# Docker version 28.x.x, build ...

docker run --rm hello-world
# Hello from Docker!
If you're on a different Linux distribution, follow the official Docker Engine installation guide for your OS.

System Requirements

ResourceMinimumRecommended (production)
CPU2 cores2 vCPU
RAM2 GB2 GB
Storage4 GB20 GB gp3
OSUbuntu 22.04/24.04Ubuntu 22.04/24.04 LTS
NetworkStable connection to both RPCsPrivate RPC endpoints
Run one small VPS per operator (or one per region for high availability). Allow SSH only from your operator IP, allow outbound HTTPS for AWS / RPC providers / image pulls, and do not expose the Prometheus metrics ports publicly. In production, use private RPC endpoints for both chains — never depend on public RPCs.

Pre-Launch Checklist

Before starting the agents, verify every item:

  • :checked-box: Signing keys created — hex keys exported, or AWS KMS key created and verified (agent keys guide)
  • :checked-box: Checkpoint storage configured — local directories created, or S3 bucket set up with public read access (S3 setup)
  • :checked-box: Cosmos signer account funded with TBTSG — the validator cannot announce without gas tokens
  • :checked-box: Validator address funded with ETH on Base Sepolia — the Base Sepolia validator needs gas for on-chain announcements
  • :checked-box: Relayer key funded with ETH on Base Sepolia — the relayer pays gas on the destination chain
  • :checked-box: RPC endpoints reachable — use private RPC URLs, never public endpoints in production
  • :checked-box: Bridge setup completed — all 7 transactions from the setup guide succeeded
Need Base Sepolia ETH? Use the Coinbase Developer Platform faucet to fund both your validator and relayer addresses.

Required Environment Variables

# Agent keys (from the agent keys guide)
export VALIDATOR_KEY="0x<your-validator-private-key>"
export COSMOS_SIGNER_KEY="0x<your-cosmos-signer-key>"
export EVM_RELAYER_KEY="0x<your-relayer-key>"

# Derived addresses
export VALIDATOR_ADDR=$(cast wallet address --private-key $VALIDATOR_KEY)

# Chain configuration
export CHAIN_ID="crescendo-1"
export DOMAIN_ID="71717171"
export REMOTE_DOMAIN="84532"
export NODE="tcp://localhost:26657"
export EVM_RPC="https://sepolia.base.org"
The AWS_REGION environment variable is required for all AWS operations. Without it, the agent fails with Invalid Configuration: Missing Region.

Pull the Agent Docker Image

Terminal
docker pull --platform linux/amd64 ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0

Agent Configuration

The Hyperlane agents need a JSON configuration file that describes both chains in the bridge. Create this file using the IDs from your bridge setup.

Replace the placeholder values (<mailbox-id>, <merkle-hook-id>, <igp-id>) with the actual hex IDs returned during the bridge setup.

If you didn't save the IDs from the setup guide, you can query them from the chain:
Terminal
# Mailbox ID (also used for validatorAnnounce)
bitsongd query hyperlane mailboxes --output json --node $NODE | jq '.mailboxes'

# MerkleTreeHook ID
bitsongd query hyperlane hooks merkle-tree-hooks --output json --node $NODE | jq '.merkle_tree_hooks'

# IGP ID
bitsongd query hyperlane hooks igps --output json --node $NODE | jq '.igps'
The Base Sepolia values are official Hyperlane contracts from the Hyperlane registry — they are already filled in and do not need to be changed.

Create the directory and save the agent configuration file:

Terminal
mkdir -p $HOME/.bitsongd/hyperlane
$HOME/.bitsongd/hyperlane/agent-config.json
{
  "chains": {
    "bitsong": {
      "name": "bitsong",
      "chainId": "crescendo-1",
      "domainId": 71717171,
      "protocol": "cosmosNative",
      "bech32Prefix": "bitsong",
      "slip44": 639,
      "contractAddressBytes": 32,
      "canonicalAsset": "utbtsg",
      "rpcUrls": [{ "http": "http://localhost:26657" }],
      "grpcUrls": [{ "http": "http://localhost:9090" }],
      "nativeToken": {
        "name": "BitSong",
        "symbol": "BTSG",
        "decimals": 6,
        "denom": "utbtsg"
      },
      "gasPrice": { "amount": "0.025", "denom": "utbtsg" },
      "gasMultiplier": "1.5",
      "blocks": {
        "confirmations": 1,
        "estimateBlockTime": 6,
        "reorgPeriod": 1
      },
      "index": { "from": 3895000, "chunk": 500 },
      "mailbox": "0x68797065726c616e650000000000000000000000000000000000000000000000",
      "validatorAnnounce": "0x68797065726c616e650000000000000000000000000000000000000000000000",
      "merkleTreeHook": "0x726f757465725f706f73745f6469737061746368000000030000000000000000",
      "interchainGasPaymaster": "0x726f757465725f706f73745f6469737061746368000000040000000000000001"
    },
    "basesepolia": {
      "name": "basesepolia",
      "chainId": 84532,
      "domainId": 84532,
      "protocol": "ethereum",
      "rpcUrls": [{ "http": "https://sepolia.base.org" }],
      "nativeToken": {
        "name": "Ether",
        "symbol": "ETH",
        "decimals": 18
      },
      "blocks": {
        "confirmations": 1,
        "estimateBlockTime": 2,
        "reorgPeriod": 1
      },
      "index": { "from": 39040037, "chunk": 999 },
      "mailbox": "0x6966b0E55883d49BFB24539356a2f8A673E02039",
      "validatorAnnounce": "0x20c44b1E3BeaDA1e9826CFd48BeEDABeE9871cE9",
      "merkleTreeHook": "0x86fb9F1c124fB20ff130C41a79a432F770f67AFD",
      "interchainGasPaymaster": "0x28B02B97a850872C4D33C3E024fab6499ad96564"
    }
  }
}

Save this file to $HOME/.bitsongd/hyperlane/agent-config.json.

The validatorAnnounce field for BitSong uses the same ID as the mailbox. This is because on Cosmos-native Hyperlane, the mailbox module handles validator announcements internally.
Cosmos chains require bothrpcUrls and grpcUrls. EVM chains only need rpcUrls. If you omit grpcUrls for BitSong, the agent will fail to start.
  • BitSong: point rpcUrls/grpcUrls at your own synced node (http://localhost:26657 / :9090). If you must use a shared node, prefer its direct IP over a TLS proxy — under indexing load a proxy such as https://rpc.testnet.bitsong.io can return 502, and it does not expose a public gRPC endpoint.
  • Base Sepolia: use an RPC with generous limits. Coinbase CDP (https://api.developer.coinbase.com/rpc/v1/base-sepolia/<API_KEY>) is recommended — note the path is /base-sepolia/ (chain 84532); /base/ is mainnet (8453). CDP caps getLogs at 1000 blocks, so keep basesepolia.index.chunk999. The public sepolia.base.org caps at 2000 and rate-limits quickly.

Configuration Fields

bitsong.index.from
integer
Block height to start indexing from. Use a recent height (current height minus a few thousand), not1. The Cosmos validator builds the Merkle tree from chain state (gRPC), so a recent value is still correct — while index.from: 1 on a multi-million-block chain makes every agent rescan all history and exhausts the RPC's max_open_connections.
bitsong.index.chunk
integer
Number of blocks to query per batch. 500 is a safe value for the BitSong CometBFT RPC.
basesepolia.index.from
integer
Block height to start indexing Base Sepolia from. 39040037 matches the current bridge deployment. Setting it lower lets you recover older messages; setting it higher speeds up indexing.
basesepolia.index.chunk
integer
Blocks per getLogs batch on Base Sepolia. Keep it within the RPC provider's range limit: ≤ 999 for Coinbase CDP (getLogs capped at 1000 blocks) and ≤ 2000 for the public sepolia.base.org. Too large a value stalls indexing with query exceeds max block range.
gasMultiplier
string
Gas multiplier for Cosmos transactions. 2.0 provides margin for gas estimation variance.
rpcUrls
array
List of RPC endpoints. For production, use private RPC URLs and configure multiple for redundancy. Never use public endpoints for production validators.

How Cross-Chain Validation Works

Before starting the agents, it helps to understand how messages are validated in each direction:

  • BitSong → Base Sepolia: The BitSong validator signs checkpoints over messages dispatched from BitSong. On the EVM side, a custom MultisigISM (deployed in the Warp Token guide) verifies these checkpoints before the relayer can deliver the message.
  • Base Sepolia → BitSong: The Base Sepolia validator signs checkpoints over messages dispatched from Base Sepolia. On the Cosmos side, the RoutingISM configured during bridge setup verifies these checkpoints.

This is why you must run two validator instances — one monitoring each origin chain. Each validator produces checkpoints that the destination chain's ISM uses for verification.

Start the Validator

The validator agent monitors the BitSong chain for dispatched messages, signs checkpoints, and stores them for relayers to read. You need one validator instance per origin chain.

Every docker run below includes --user $(id -u):$(id -g) and --ulimit nofile=65536:65536. The first prevents Permission denied on the mounted /hyperlane_db volume (Docker would otherwise create it as root); the second prevents Too many open files while the agent indexes. If you hit a persistent permission error, running as --user 0:0 is an equivalent alternative.

BitSong Validator

mkdir -p $HOME/.bitsongd/hyperlane/validator-bitsong-db $HOME/.bitsongd/hyperlane/checkpoints-bitsong

docker run -d \
    --name hyperlane-validator-bitsong \
    --network host \
    --user $(id -u):$(id -g) \
    --ulimit nofile=65536:65536 \
    -e CONFIG_FILES=/config/agent-config.json \
    -v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
    -v $HOME/.bitsongd/hyperlane/validator-bitsong-db:/hyperlane_db \
    -v $HOME/.bitsongd/hyperlane/checkpoints-bitsong:/checkpoints-bitsong \
    ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
    ./validator \
    --db /hyperlane_db \
    --originChainName bitsong \
    --reorgPeriod 1 \
    --interval 5 \
    --validator.type hexKey \
    --validator.key $VALIDATOR_KEY \
    --chains.bitsong.signer.type cosmosKey \
    --chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
    --chains.bitsong.signer.prefix bitsong \
    --checkpointSyncer.type localStorage \
    --checkpointSyncer.path /checkpoints-bitsong \
    --metrics-port 9101
The BitSong validator always uses cosmosKey for the chain signer (to sign on-chain announcement transactions) regardless of whether the validator checkpoint signer uses hex or AWS KMS. The cosmosKey type uses the bitsong prefix to derive the correct Bech32 address.

Base Sepolia Validator

The Base Sepolia validator needs ETH on Base Sepolia to pay for the on-chain announcement transaction. Fund your validator address ($VALIDATOR_ADDR) using the Coinbase faucet before starting.
mkdir -p $HOME/.bitsongd/hyperlane/validator-basesepolia-db $HOME/.bitsongd/hyperlane/checkpoints-basesepolia

docker run -d \
    --name hyperlane-validator-basesepolia \
    --network host \
    --user $(id -u):$(id -g) \
    --ulimit nofile=65536:65536 \
    -e CONFIG_FILES=/config/agent-config.json \
    -v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
    -v $HOME/.bitsongd/hyperlane/validator-basesepolia-db:/hyperlane_db \
    -v $HOME/.bitsongd/hyperlane/checkpoints-basesepolia:/checkpoints-basesepolia \
    ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
    ./validator \
    --db /hyperlane_db \
    --originChainName basesepolia \
    --reorgPeriod 1 \
    --interval 5 \
    --validator.type hexKey \
    --validator.key $VALIDATOR_KEY \
    --chains.basesepolia.signer.type hexKey \
    --chains.basesepolia.signer.key $EVM_RELAYER_KEY \
    --checkpointSyncer.type localStorage \
    --checkpointSyncer.path /checkpoints-basesepolia \
    --metrics-port 9201
The Base Sepolia validator uses hexKey (local) or aws (production) for both the checkpoint signer and the chain transaction signer. Unlike BitSong, EVM chains do not need a separate Cosmos signer.
When using AWS KMS with S3, you can use the same KMS key and the same S3 bucket for both validators. Differentiate them by using unique --checkpointSyncer.folder values (bitsong vs basesepolia).

Validator CLI Reference

--originChainName
string required
The chain this validator monitors. Run one validator instance per chain.
--db
string required
Path to the persistent database directory. Each validator instance must have its own database path.
--reorgPeriod
integer
Number of block confirmations before signing a checkpoint. 1 is appropriate for testnet. For production, match the chain's finality guarantees.
--interval
integer
Polling interval in seconds. 5 balances responsiveness with RPC load.
--validator.type
string required
Signing key type for checkpoints. hexKey for local hex private keys, aws for AWS KMS.
--validator.key
string
The hex private key for checkpoint signing. Required when --validator.type is hexKey.
--validator.region
string
AWS region of the KMS key. Required when --validator.type is aws.
--validator.id
string
AWS KMS key ID or alias (e.g. alias/hyperlane-validator-signer-bitsong). Required when --validator.type is aws.
--checkpointSyncer.type
string required
Storage backend for signed checkpoints. localStorage for local testing, s3 for production with AWS S3.
--checkpointSyncer.path
string
Local directory path for checkpoints. Required when --checkpointSyncer.type is localStorage.
--checkpointSyncer.bucket
string
S3 bucket name. Required when --checkpointSyncer.type is s3.
--checkpointSyncer.region
string
AWS region of the S3 bucket. Required when --checkpointSyncer.type is s3.
--checkpointSyncer.folder
string
Folder path inside the S3 bucket. Use different folders for each chain when sharing a bucket. Required when --checkpointSyncer.type is s3.
--chains.[chainName].signer.type
string
Transaction signer type. hexKey for EVM hex keys, cosmosKey for Cosmos chains, aws for AWS KMS.
--chains.[chainName].customRpcUrls
string
Override RPC URLs from the agent config. Comma-separated list of URLs. Useful for configuring private or redundant endpoints.

Wait for Validator Announcements

After starting, validators automatically announce their checkpoint storage locations on-chain. This is how relayers discover where to read signed checkpoints. The announcement requires the Cosmos signer account to have a TBTSG balance for gas fees.

Wait for announcements (this can take up to 2 minutes):

Terminal
# Check BitSong validator announcement
bitsongd query hyperlane ism announced-storage-locations \
    <mailbox-id> $(echo $VALIDATOR_ADDR | tr '[:upper:]' '[:lower:]') \
    --output json --node $NODE | jq '.storage_locations'

A non-empty storage_locations array confirms the validator has announced successfully. If the array is empty after 2 minutes, check the validator logs and ensure the Cosmos signer account is funded.

Start the Relayer

The relayer reads signed checkpoints from validators and delivers messages to their destination chains. It bridges messages in both directions between BitSong and Base Sepolia.

The relayer needs ETH on Base Sepolia to pay gas when delivering messages to the EVM chain. Fund your relayer address using the Coinbase faucet before starting.
mkdir -p $HOME/.bitsongd/hyperlane/relayer-db

docker run -d \
    --name hyperlane-relayer \
    --network host \
    --user $(id -u):$(id -g) \
    --ulimit nofile=65536:65536 \
    -e CONFIG_FILES=/config/agent-config.json \
    -v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
    -v $HOME/.bitsongd/hyperlane/relayer-db:/hyperlane_db \
    -v $HOME/.bitsongd/hyperlane/checkpoints-bitsong:/checkpoints-bitsong:ro \
    -v $HOME/.bitsongd/hyperlane/checkpoints-basesepolia:/checkpoints-basesepolia:ro \
    ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
    ./relayer \
    --db /hyperlane_db \
    --relayChains bitsong,basesepolia \
    --allowLocalCheckpointSyncers true \
    --gaspaymentenforcement '[{"type": "none"}]' \
    --chains.bitsong.signer.type cosmosKey \
    --chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
    --chains.bitsong.signer.prefix bitsong \
    --chains.basesepolia.signer.type hexKey \
    --chains.basesepolia.signer.key $EVM_RELAYER_KEY \
    --metrics-port 9301
The --allowLocalCheckpointSyncers true flag is required when validators use localStorage for checkpoint storage. It lets the relayer read checkpoints from the local filesystem. When using S3, this flag is not needed — the relayer reads directly from the S3 bucket.
The --gaspaymentenforcement '[{"type": "none"}]' flag disables gas payment enforcement for testnet. In production, configure this to enforce IGP payments so relayers are compensated.
The relayer always uses cosmosKey for the BitSong chain signer (same as the validator), since BitSong is a Cosmos-native chain. For Base Sepolia, use hexKey (local) or aws (production).

Relayer Whitelist

To limit the relayer to only relay your warp token messages (avoiding noise from other projects on shared testnets), add a whitelist:

Terminal
# Warp token ID (BitSong side) and HypERC20 contract (Base Sepolia side)
TOKEN_ID="0x726f757465725f61707000000000000000000000000000010000000000000000"
EVM_HYP_ERC20="0x27d0F8cBF125995f91407e9E5F70EF52c4412878"
EVM_PADDED=$(printf "0x%064s" "${EVM_HYP_ERC20#0x}" | tr ' ' '0')

# Add --whitelist flag to the relayer docker run command:
--whitelist "[{\"senderAddress\":\"$TOKEN_ID\",\"destinationDomain\":\"$REMOTE_DOMAIN\"},{\"senderAddress\":\"$EVM_PADDED\",\"destinationDomain\":\"$DOMAIN_ID\"}]"

The two entries whitelist both directions: messages from the BitSong warp token to Base Sepolia, and messages from the HypERC20 contract back to BitSong. Replace the values with your own warp route's IDs if you deployed a different one.

Running Multiple Validators

If you are running validators for multiple operators or coordinating a multi-validator setup:

  • A single signing key can serve multiple validator instances
  • A shared AWS account is permitted across validators
  • A shared S3 bucket is allowed — use unique --checkpointSyncer.folder values to separate checkpoint data (e.g. validator-1, validator-2)
  • Each validator instance must have its own --db path, unique metrics port, and separate log output

Manage Agents

Check agent status

Terminal
docker ps --filter "name=hyperlane" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

View logs

docker logs -f --tail 100 hyperlane-validator-bitsong

Verify checkpoints are being written

# BitSong checkpoints
ls -la $HOME/.bitsongd/hyperlane/checkpoints-bitsong/

# Base Sepolia checkpoints
ls -la $HOME/.bitsongd/hyperlane/checkpoints-basesepolia/

New checkpoint files are written every time a new outbound message is inserted into the mailbox.

Stop all agents

Terminal
docker stop hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer

Restart agents

Terminal
docker start hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer

Remove agents (preserves data)

Terminal
docker rm hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer

Upgrade the agent image

To move to a newer agent release, pull the new image, recreate the containers, and keep the database directories — they hold indexing state and must survive upgrades:

Terminal
docker pull --platform linux/amd64 ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0
docker stop hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
docker rm hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
# Re-run the docker run commands from the sections above

Back up

Keep a secure copy of the items needed to rebuild the stack:

  • The agent config (agent-config.json) and any environment file
  • The AWS KMS key alias/ID and the S3 bucket name + policy (AWS deployments)
  • The Cosmos signer key (mnemonic or hex)

The local /hyperlane_db directories are rebuildable from chain state, so they do not need backing up — but do not delete them during routine restarts or upgrades.

Monitoring

Hyperlane agents expose Prometheus metrics on the port passed via --metrics-port. This guide assigns:

ContainerMetrics port
hyperlane-validator-bitsong9101
hyperlane-validator-basesepolia9201
hyperlane-relayer9301

Example Prometheus scrape config:

prometheus.yml
scrape_configs:
  - job_name: hyperlane-validator-bitsong
    static_configs:
      - targets: ["127.0.0.1:9101"]

  - job_name: hyperlane-validator-basesepolia
    static_configs:
      - targets: ["127.0.0.1:9201"]

  - job_name: hyperlane-relayer
    static_configs:
      - targets: ["127.0.0.1:9301"]

Key metrics to watch:

MetricWhat it should do
hyperlane_latest_checkpointIncrease when the origin chain dispatches messages
hyperlane_block_heightKeep increasing while the RPC is healthy
hyperlane_contract_sync_livenessKeep moving; a flat value means the indexing loop may be stuck
hyperlane_cursor_current_blockAdvance as the agent indexes
hyperlane_span_events_total{event_level="error"}Stay flat; alert on repeated increases

Recommended alerts:

  • hyperlane_block_height does not increase for 30 minutes
  • hyperlane_contract_sync_liveness is flat for 15 minutes
  • hyperlane_latest_checkpoint does not increase while messages are being dispatched and block height is rising
  • Error-level log events increase repeatedly over a 1-hour window
  • No checkpoint objects appear in S3 after a known Mailbox dispatch

When an alert fires, check the logs first, then RPC health, AWS KMS access, S3 permissions, and account balances.

With --network host the metrics ports bind to all interfaces. Restrict them with a firewall/security group so only your monitoring system can reach them.

Troubleshooting

References

Next Steps

Agent Keys Reference

Review key management options including AWS KMS and S3 checkpoint storage for production environments.

Bridge Setup

Review the on-chain bridge setup including ISM, Mailbox, and warp token configuration.

Rotate the Multisig

Add or replace validators and change the threshold on a live bridge (owner operation).
Copyright © 2026