Run Validators
This guide walks you through running Hyperlane validator and relayer agents that power cross-chain message delivery between BitSong and Base Sepolia. It follows the official Hyperlane validator documentation, adapted for the BitSong testnet.
Two deployment configurations are covered:
- Local (Testnet) — Hex private keys with localStorage checkpoint storage. Simplest to set up, suitable for testing.
- AWS (Production) — AWS KMS signing keys with S3 checkpoint storage. Hardware-backed security, recommended for production.
Prerequisites
Before running agents, ensure you have completed the previous steps:
- Agent keys generated and exported as environment variables
- Hyperlane bridge setup completed (all 7 on-chain transactions)
- Docker installed and running (see install Docker below)
- Foundry installed (provides
cast) - A funded Cosmos signer account on BitSong (see agent keys guide)
Install Docker
The Hyperlane agents run as Docker containers. Install Docker Engine on your server if you don't have it already:
# Remove any old Docker packages
sudo apt-get remove -y docker docker-engine docker.io containerd runc 2>/dev/null
# Install prerequisites
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the Docker repository
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker Engine
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin
After installation, add your user to the docker group so you can run containers without sudo:
sudo usermod -aG docker $USER
newgrp docker
Verify Docker is working:
docker --version
# Docker version 28.x.x, build ...
docker run --rm hello-world
# Hello from Docker!
System Requirements
| Resource | Minimum | Recommended (production) |
|---|---|---|
| CPU | 2 cores | 2 vCPU |
| RAM | 2 GB | 2 GB |
| Storage | 4 GB | 20 GB gp3 |
| OS | Ubuntu 22.04/24.04 | Ubuntu 22.04/24.04 LTS |
| Network | Stable connection to both RPCs | Private RPC endpoints |
Pre-Launch Checklist
Before starting the agents, verify every item:
- :checked-box: Signing keys created — hex keys exported, or AWS KMS key created and verified (agent keys guide)
- :checked-box: Checkpoint storage configured — local directories created, or S3 bucket set up with public read access (S3 setup)
- :checked-box: Cosmos signer account funded with TBTSG — the validator cannot announce without gas tokens
- :checked-box: Validator address funded with ETH on Base Sepolia — the Base Sepolia validator needs gas for on-chain announcements
- :checked-box: Relayer key funded with ETH on Base Sepolia — the relayer pays gas on the destination chain
- :checked-box: RPC endpoints reachable — use private RPC URLs, never public endpoints in production
- :checked-box: Bridge setup completed — all 7 transactions from the setup guide succeeded
Required Environment Variables
# Agent keys (from the agent keys guide)
export VALIDATOR_KEY="0x<your-validator-private-key>"
export COSMOS_SIGNER_KEY="0x<your-cosmos-signer-key>"
export EVM_RELAYER_KEY="0x<your-relayer-key>"
# Derived addresses
export VALIDATOR_ADDR=$(cast wallet address --private-key $VALIDATOR_KEY)
# Chain configuration
export CHAIN_ID="crescendo-1"
export DOMAIN_ID="71717171"
export REMOTE_DOMAIN="84532"
export NODE="tcp://localhost:26657"
export EVM_RPC="https://sepolia.base.org"
# AWS credentials
export AWS_ACCESS_KEY_ID="<your-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
export AWS_REGION="<your-aws-region>"
# Cosmos signer key (hex — AWS KMS is only for validator/relayer EVM signers)
export COSMOS_SIGNER_KEY="0x<your-cosmos-signer-key>"
# Relayer uses its own KMS key: alias/hyperlane-relayer-bitsong
# No EVM_RELAYER_KEY variable needed
# Derived validator address from KMS
export VALIDATOR_ADDR=$(AWS_KMS_KEY_ID=alias/hyperlane-validator-signer-bitsong cast wallet address --aws)
# Chain configuration
export CHAIN_ID="crescendo-1"
export DOMAIN_ID="71717171"
export REMOTE_DOMAIN="84532"
export NODE="tcp://localhost:26657"
export EVM_RPC="https://sepolia.base.org"
AWS_REGION environment variable is required for all AWS operations. Without it, the agent fails with Invalid Configuration: Missing Region.Pull the Agent Docker Image
docker pull --platform linux/amd64 ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0
Agent Configuration
The Hyperlane agents need a JSON configuration file that describes both chains in the bridge. Create this file using the IDs from your bridge setup.
Replace the placeholder values (<mailbox-id>, <merkle-hook-id>, <igp-id>) with the actual hex IDs returned during the bridge setup.
# Mailbox ID (also used for validatorAnnounce)
bitsongd query hyperlane mailboxes --output json --node $NODE | jq '.mailboxes'
# MerkleTreeHook ID
bitsongd query hyperlane hooks merkle-tree-hooks --output json --node $NODE | jq '.merkle_tree_hooks'
# IGP ID
bitsongd query hyperlane hooks igps --output json --node $NODE | jq '.igps'
Create the directory and save the agent configuration file:
mkdir -p $HOME/.bitsongd/hyperlane
{
"chains": {
"bitsong": {
"name": "bitsong",
"chainId": "crescendo-1",
"domainId": 71717171,
"protocol": "cosmosNative",
"bech32Prefix": "bitsong",
"slip44": 639,
"contractAddressBytes": 32,
"canonicalAsset": "utbtsg",
"rpcUrls": [{ "http": "http://localhost:26657" }],
"grpcUrls": [{ "http": "http://localhost:9090" }],
"nativeToken": {
"name": "BitSong",
"symbol": "BTSG",
"decimals": 6,
"denom": "utbtsg"
},
"gasPrice": { "amount": "0.025", "denom": "utbtsg" },
"gasMultiplier": "1.5",
"blocks": {
"confirmations": 1,
"estimateBlockTime": 6,
"reorgPeriod": 1
},
"index": { "from": 3895000, "chunk": 500 },
"mailbox": "0x68797065726c616e650000000000000000000000000000000000000000000000",
"validatorAnnounce": "0x68797065726c616e650000000000000000000000000000000000000000000000",
"merkleTreeHook": "0x726f757465725f706f73745f6469737061746368000000030000000000000000",
"interchainGasPaymaster": "0x726f757465725f706f73745f6469737061746368000000040000000000000001"
},
"basesepolia": {
"name": "basesepolia",
"chainId": 84532,
"domainId": 84532,
"protocol": "ethereum",
"rpcUrls": [{ "http": "https://sepolia.base.org" }],
"nativeToken": {
"name": "Ether",
"symbol": "ETH",
"decimals": 18
},
"blocks": {
"confirmations": 1,
"estimateBlockTime": 2,
"reorgPeriod": 1
},
"index": { "from": 39040037, "chunk": 999 },
"mailbox": "0x6966b0E55883d49BFB24539356a2f8A673E02039",
"validatorAnnounce": "0x20c44b1E3BeaDA1e9826CFd48BeEDABeE9871cE9",
"merkleTreeHook": "0x86fb9F1c124fB20ff130C41a79a432F770f67AFD",
"interchainGasPaymaster": "0x28B02B97a850872C4D33C3E024fab6499ad96564"
}
}
}
Save this file to $HOME/.bitsongd/hyperlane/agent-config.json.
validatorAnnounce field for BitSong uses the same ID as the mailbox. This is because on Cosmos-native Hyperlane, the mailbox module handles validator announcements internally.rpcUrls and grpcUrls. EVM chains only need rpcUrls. If you omit grpcUrls for BitSong, the agent will fail to start.- BitSong: point
rpcUrls/grpcUrlsat your own synced node (http://localhost:26657/:9090). If you must use a shared node, prefer its direct IP over a TLS proxy — under indexing load a proxy such ashttps://rpc.testnet.bitsong.iocan return 502, and it does not expose a public gRPC endpoint. - Base Sepolia: use an RPC with generous limits. Coinbase CDP (
https://api.developer.coinbase.com/rpc/v1/base-sepolia/<API_KEY>) is recommended — note the path is/base-sepolia/(chain84532);/base/is mainnet (8453). CDP capsgetLogsat 1000 blocks, so keepbasesepolia.index.chunk≤999. The publicsepolia.base.orgcaps at 2000 and rate-limits quickly.
Configuration Fields
1. The Cosmos validator builds the Merkle tree from chain state (gRPC), so a recent value is still correct — while index.from: 1 on a multi-million-block chain makes every agent rescan all history and exhausts the RPC's max_open_connections.500 is a safe value for the BitSong CometBFT RPC.39040037 matches the current bridge deployment. Setting it lower lets you recover older messages; setting it higher speeds up indexing.getLogs batch on Base Sepolia. Keep it within the RPC provider's range limit: ≤ 999 for Coinbase CDP (getLogs capped at 1000 blocks) and ≤ 2000 for the public sepolia.base.org. Too large a value stalls indexing with query exceeds max block range.2.0 provides margin for gas estimation variance.How Cross-Chain Validation Works
Before starting the agents, it helps to understand how messages are validated in each direction:
- BitSong → Base Sepolia: The BitSong validator signs checkpoints over messages dispatched from BitSong. On the EVM side, a custom MultisigISM (deployed in the Warp Token guide) verifies these checkpoints before the relayer can deliver the message.
- Base Sepolia → BitSong: The Base Sepolia validator signs checkpoints over messages dispatched from Base Sepolia. On the Cosmos side, the RoutingISM configured during bridge setup verifies these checkpoints.
This is why you must run two validator instances — one monitoring each origin chain. Each validator produces checkpoints that the destination chain's ISM uses for verification.
Start the Validator
The validator agent monitors the BitSong chain for dispatched messages, signs checkpoints, and stores them for relayers to read. You need one validator instance per origin chain.
docker run below includes --user $(id -u):$(id -g) and --ulimit nofile=65536:65536. The first prevents Permission denied on the mounted /hyperlane_db volume (Docker would otherwise create it as root); the second prevents Too many open files while the agent indexes. If you hit a persistent permission error, running as --user 0:0 is an equivalent alternative.BitSong Validator
mkdir -p $HOME/.bitsongd/hyperlane/validator-bitsong-db $HOME/.bitsongd/hyperlane/checkpoints-bitsong
docker run -d \
--name hyperlane-validator-bitsong \
--network host \
--user $(id -u):$(id -g) \
--ulimit nofile=65536:65536 \
-e CONFIG_FILES=/config/agent-config.json \
-v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
-v $HOME/.bitsongd/hyperlane/validator-bitsong-db:/hyperlane_db \
-v $HOME/.bitsongd/hyperlane/checkpoints-bitsong:/checkpoints-bitsong \
ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
./validator \
--db /hyperlane_db \
--originChainName bitsong \
--reorgPeriod 1 \
--interval 5 \
--validator.type hexKey \
--validator.key $VALIDATOR_KEY \
--chains.bitsong.signer.type cosmosKey \
--chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
--chains.bitsong.signer.prefix bitsong \
--checkpointSyncer.type localStorage \
--checkpointSyncer.path /checkpoints-bitsong \
--metrics-port 9101
mkdir -p $HOME/.bitsongd/hyperlane/validator-bitsong-db
docker run -d \
--name hyperlane-validator-bitsong \
--network host \
--user $(id -u):$(id -g) \
--ulimit nofile=65536:65536 \
-e CONFIG_FILES=/config/agent-config.json \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
-v $HOME/.bitsongd/hyperlane/validator-bitsong-db:/hyperlane_db \
ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
./validator \
--db /hyperlane_db \
--originChainName bitsong \
--reorgPeriod 1 \
--interval 5 \
--validator.type aws \
--validator.region $AWS_REGION \
--validator.id alias/hyperlane-validator-signer-bitsong \
--chains.bitsong.signer.type cosmosKey \
--chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
--chains.bitsong.signer.prefix bitsong \
--checkpointSyncer.type s3 \
--checkpointSyncer.bucket hyperlane-validator-signatures-bitsong \
--checkpointSyncer.region $AWS_REGION \
--checkpointSyncer.folder bitsong \
--metrics-port 9101
cosmosKey for the chain signer (to sign on-chain announcement transactions) regardless of whether the validator checkpoint signer uses hex or AWS KMS. The cosmosKey type uses the bitsong prefix to derive the correct Bech32 address.Base Sepolia Validator
$VALIDATOR_ADDR) using the Coinbase faucet before starting.mkdir -p $HOME/.bitsongd/hyperlane/validator-basesepolia-db $HOME/.bitsongd/hyperlane/checkpoints-basesepolia
docker run -d \
--name hyperlane-validator-basesepolia \
--network host \
--user $(id -u):$(id -g) \
--ulimit nofile=65536:65536 \
-e CONFIG_FILES=/config/agent-config.json \
-v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
-v $HOME/.bitsongd/hyperlane/validator-basesepolia-db:/hyperlane_db \
-v $HOME/.bitsongd/hyperlane/checkpoints-basesepolia:/checkpoints-basesepolia \
ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
./validator \
--db /hyperlane_db \
--originChainName basesepolia \
--reorgPeriod 1 \
--interval 5 \
--validator.type hexKey \
--validator.key $VALIDATOR_KEY \
--chains.basesepolia.signer.type hexKey \
--chains.basesepolia.signer.key $EVM_RELAYER_KEY \
--checkpointSyncer.type localStorage \
--checkpointSyncer.path /checkpoints-basesepolia \
--metrics-port 9201
mkdir -p $HOME/.bitsongd/hyperlane/validator-basesepolia-db
docker run -d \
--name hyperlane-validator-basesepolia \
--network host \
--user $(id -u):$(id -g) \
--ulimit nofile=65536:65536 \
-e CONFIG_FILES=/config/agent-config.json \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
-v $HOME/.bitsongd/hyperlane/validator-basesepolia-db:/hyperlane_db \
ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
./validator \
--db /hyperlane_db \
--originChainName basesepolia \
--reorgPeriod 1 \
--interval 5 \
--validator.type aws \
--validator.region $AWS_REGION \
--validator.id alias/hyperlane-validator-signer-bitsong \
--chains.basesepolia.signer.type aws \
--chains.basesepolia.signer.region $AWS_REGION \
--chains.basesepolia.signer.id alias/hyperlane-validator-signer-bitsong \
--checkpointSyncer.type s3 \
--checkpointSyncer.bucket hyperlane-validator-signatures-bitsong \
--checkpointSyncer.region $AWS_REGION \
--checkpointSyncer.folder basesepolia \
--metrics-port 9201
hexKey (local) or aws (production) for both the checkpoint signer and the chain transaction signer. Unlike BitSong, EVM chains do not need a separate Cosmos signer.--checkpointSyncer.folder values (bitsong vs basesepolia).Validator CLI Reference
1 is appropriate for testnet. For production, match the chain's finality guarantees.5 balances responsiveness with RPC load.hexKey for local hex private keys, aws for AWS KMS.--validator.type is hexKey.--validator.type is aws.alias/hyperlane-validator-signer-bitsong). Required when --validator.type is aws.localStorage for local testing, s3 for production with AWS S3.--checkpointSyncer.type is localStorage.--checkpointSyncer.type is s3.--checkpointSyncer.type is s3.--checkpointSyncer.type is s3.hexKey for EVM hex keys, cosmosKey for Cosmos chains, aws for AWS KMS.Wait for Validator Announcements
After starting, validators automatically announce their checkpoint storage locations on-chain. This is how relayers discover where to read signed checkpoints. The announcement requires the Cosmos signer account to have a TBTSG balance for gas fees.
Wait for announcements (this can take up to 2 minutes):
# Check BitSong validator announcement
bitsongd query hyperlane ism announced-storage-locations \
<mailbox-id> $(echo $VALIDATOR_ADDR | tr '[:upper:]' '[:lower:]') \
--output json --node $NODE | jq '.storage_locations'
A non-empty storage_locations array confirms the validator has announced successfully. If the array is empty after 2 minutes, check the validator logs and ensure the Cosmos signer account is funded.
Start the Relayer
The relayer reads signed checkpoints from validators and delivers messages to their destination chains. It bridges messages in both directions between BitSong and Base Sepolia.
mkdir -p $HOME/.bitsongd/hyperlane/relayer-db
docker run -d \
--name hyperlane-relayer \
--network host \
--user $(id -u):$(id -g) \
--ulimit nofile=65536:65536 \
-e CONFIG_FILES=/config/agent-config.json \
-v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
-v $HOME/.bitsongd/hyperlane/relayer-db:/hyperlane_db \
-v $HOME/.bitsongd/hyperlane/checkpoints-bitsong:/checkpoints-bitsong:ro \
-v $HOME/.bitsongd/hyperlane/checkpoints-basesepolia:/checkpoints-basesepolia:ro \
ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
./relayer \
--db /hyperlane_db \
--relayChains bitsong,basesepolia \
--allowLocalCheckpointSyncers true \
--gaspaymentenforcement '[{"type": "none"}]' \
--chains.bitsong.signer.type cosmosKey \
--chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
--chains.bitsong.signer.prefix bitsong \
--chains.basesepolia.signer.type hexKey \
--chains.basesepolia.signer.key $EVM_RELAYER_KEY \
--metrics-port 9301
mkdir -p $HOME/.bitsongd/hyperlane/relayer-db
docker run -d \
--name hyperlane-relayer \
--network host \
--user $(id -u):$(id -g) \
--ulimit nofile=65536:65536 \
-e CONFIG_FILES=/config/agent-config.json \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-v $HOME/.bitsongd/hyperlane/agent-config.json:/config/agent-config.json:ro \
-v $HOME/.bitsongd/hyperlane/relayer-db:/hyperlane_db \
ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0 \
./relayer \
--db /hyperlane_db \
--relayChains bitsong,basesepolia \
--gaspaymentenforcement '[{"type": "none"}]' \
--chains.bitsong.signer.type cosmosKey \
--chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
--chains.bitsong.signer.prefix bitsong \
--chains.basesepolia.signer.type aws \
--chains.basesepolia.signer.region $AWS_REGION \
--chains.basesepolia.signer.id alias/hyperlane-relayer-bitsong \
--metrics-port 9301
--allowLocalCheckpointSyncers true flag is required when validators use localStorage for checkpoint storage. It lets the relayer read checkpoints from the local filesystem. When using S3, this flag is not needed — the relayer reads directly from the S3 bucket.--gaspaymentenforcement '[{"type": "none"}]' flag disables gas payment enforcement for testnet. In production, configure this to enforce IGP payments so relayers are compensated.cosmosKey for the BitSong chain signer (same as the validator), since BitSong is a Cosmos-native chain. For Base Sepolia, use hexKey (local) or aws (production).Relayer Whitelist
To limit the relayer to only relay your warp token messages (avoiding noise from other projects on shared testnets), add a whitelist:
# Warp token ID (BitSong side) and HypERC20 contract (Base Sepolia side)
TOKEN_ID="0x726f757465725f61707000000000000000000000000000010000000000000000"
EVM_HYP_ERC20="0x27d0F8cBF125995f91407e9E5F70EF52c4412878"
EVM_PADDED=$(printf "0x%064s" "${EVM_HYP_ERC20#0x}" | tr ' ' '0')
# Add --whitelist flag to the relayer docker run command:
--whitelist "[{\"senderAddress\":\"$TOKEN_ID\",\"destinationDomain\":\"$REMOTE_DOMAIN\"},{\"senderAddress\":\"$EVM_PADDED\",\"destinationDomain\":\"$DOMAIN_ID\"}]"
The two entries whitelist both directions: messages from the BitSong warp token to Base Sepolia, and messages from the HypERC20 contract back to BitSong. Replace the values with your own warp route's IDs if you deployed a different one.
Running Multiple Validators
If you are running validators for multiple operators or coordinating a multi-validator setup:
- A single signing key can serve multiple validator instances
- A shared AWS account is permitted across validators
- A shared S3 bucket is allowed — use unique
--checkpointSyncer.foldervalues to separate checkpoint data (e.g.validator-1,validator-2) - Each validator instance must have its own
--dbpath, unique metrics port, and separate log output
Manage Agents
Check agent status
docker ps --filter "name=hyperlane" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
View logs
docker logs -f --tail 100 hyperlane-validator-bitsong
docker logs -f --tail 100 hyperlane-validator-basesepolia
docker logs -f --tail 100 hyperlane-relayer
Verify checkpoints are being written
# BitSong checkpoints
ls -la $HOME/.bitsongd/hyperlane/checkpoints-bitsong/
# Base Sepolia checkpoints
ls -la $HOME/.bitsongd/hyperlane/checkpoints-basesepolia/
# List checkpoint files in S3
aws s3 ls s3://hyperlane-validator-signatures-bitsong/bitsong/ --recursive
aws s3 ls s3://hyperlane-validator-signatures-bitsong/basesepolia/ --recursive
New checkpoint files are written every time a new outbound message is inserted into the mailbox.
Stop all agents
docker stop hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
Restart agents
docker start hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
Remove agents (preserves data)
docker rm hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
Upgrade the agent image
To move to a newer agent release, pull the new image, recreate the containers, and keep the database directories — they hold indexing state and must survive upgrades:
docker pull --platform linux/amd64 ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0
docker stop hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
docker rm hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer
# Re-run the docker run commands from the sections above
Back up
Keep a secure copy of the items needed to rebuild the stack:
- The agent config (
agent-config.json) and any environment file - The AWS KMS key alias/ID and the S3 bucket name + policy (AWS deployments)
- The Cosmos signer key (mnemonic or hex)
The local /hyperlane_db directories are rebuildable from chain state, so they do not need backing up — but do not delete them during routine restarts or upgrades.
Monitoring
Hyperlane agents expose Prometheus metrics on the port passed via --metrics-port. This guide assigns:
| Container | Metrics port |
|---|---|
hyperlane-validator-bitsong | 9101 |
hyperlane-validator-basesepolia | 9201 |
hyperlane-relayer | 9301 |
Example Prometheus scrape config:
scrape_configs:
- job_name: hyperlane-validator-bitsong
static_configs:
- targets: ["127.0.0.1:9101"]
- job_name: hyperlane-validator-basesepolia
static_configs:
- targets: ["127.0.0.1:9201"]
- job_name: hyperlane-relayer
static_configs:
- targets: ["127.0.0.1:9301"]
Key metrics to watch:
| Metric | What it should do |
|---|---|
hyperlane_latest_checkpoint | Increase when the origin chain dispatches messages |
hyperlane_block_height | Keep increasing while the RPC is healthy |
hyperlane_contract_sync_liveness | Keep moving; a flat value means the indexing loop may be stuck |
hyperlane_cursor_current_block | Advance as the agent indexes |
hyperlane_span_events_total{event_level="error"} | Stay flat; alert on repeated increases |
Recommended alerts:
hyperlane_block_heightdoes not increase for 30 minuteshyperlane_contract_sync_livenessis flat for 15 minuteshyperlane_latest_checkpointdoes not increase while messages are being dispatched and block height is rising- Error-level log events increase repeatedly over a 1-hour window
- No checkpoint objects appear in S3 after a known Mailbox dispatch
When an alert fires, check the logs first, then RPC health, AWS KMS access, S3 permissions, and account balances.
--network host the metrics ports bind to all interfaces. Restrict them with a firewall/security group so only your monitoring system can reach them.Troubleshooting
The Cosmos signer account needs TBTSG tokens to pay for the announcement transaction. Fund it:
bitsongd tx bank send <your-key-name> <cosmos-signer-address> 10000000utbtsg \
--keyring-backend test --chain-id crescendo-1 --fees 10000utbtsg -y
Check the validator logs for messages like Failed to announce or insufficient funds.
The AWS_REGION environment variable must be set when using AWS KMS or S3. Pass it to the Docker container:
docker run -d \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-e AWS_REGION=$AWS_REGION \
...
Without this, the agent fails with Invalid Configuration: Missing Region.
Ensure the checkpoint directories are mounted correctly in the relayer container and that --allowLocalCheckpointSyncers true is set. Verify checkpoints exist:
ls -la $HOME/.bitsongd/hyperlane/checkpoints-bitsong/
ls -la $HOME/.bitsongd/hyperlane/checkpoints-basesepolia/
The directories must contain JSON checkpoint files. If empty, the validators may not be running or the chain has no dispatched messages yet.
Verify the S3 bucket has public read access and contains checkpoint files:
aws s3 ls s3://hyperlane-validator-signatures-bitsong/bitsong/ --recursive
Check that the bucket policy grants s3:GetObject and s3:ListBucket to Principal: "*". See the S3 bucket setup in the agent keys guide.
Check the container logs for the error:
docker logs hyperlane-validator-bitsong
Common causes:
- Invalid private key format (must be
0x-prefixed hex) - Agent config JSON syntax errors
- RPC endpoint unreachable (check
--network hostis set) - Missing
grpcUrlsin the BitSong agent config (Cosmos chains require both RPC and gRPC) - AWS credentials not passed to the container (when using KMS/S3)
If you see IO error: While open a file for appending: /hyperlane_db/LOG: Permission denied, the mounted host directories are owned by root instead of your user. This typically happens when Docker auto-creates mount targets on the first run.
Fix the ownership of the existing directories:
sudo chown -R $(id -u):$(id -g) $HOME/.bitsongd/hyperlane/
All docker run commands in this guide include --user $(id -u):$(id -g) to prevent this issue. If you're using custom commands, make sure to include this flag.
If you used a placeholder address when creating the MultisigISM during the bridge setup, you need to update it before the validator will work. Create a new MultisigISM with your real validator address and update the RoutingISM:
# 1. Create a new MultisigISM with your real address
bitsongd tx hyperlane ism create-message-id-multisig \
$VALIDATOR_ADDR 1 \
--from $KEY_NAME --keyring-backend test \
--chain-id $CHAIN_ID --node $NODE \
--gas auto --gas-adjustment 1.5 --fees 10000utbtsg -y
# 2. Remove the old domain mapping from the RoutingISM
bitsongd tx hyperlane ism remove-routing-ism-domain \
<routing-ism-id> $REMOTE_DOMAIN \
--from $KEY_NAME --keyring-backend test \
--chain-id $CHAIN_ID --node $NODE \
--gas auto --gas-adjustment 1.5 --fees 10000utbtsg -y
# 3. Set the new mapping
bitsongd tx hyperlane ism set-routing-ism-domain \
<routing-ism-id> $REMOTE_DOMAIN <new-multisig-ism-id> \
--from $KEY_NAME --keyring-backend test \
--chain-id $CHAIN_ID --node $NODE \
--gas auto --gas-adjustment 1.5 --fees 10000utbtsg -y
Replace <routing-ism-id> with the RoutingISM ID from the bridge setup and <new-multisig-ism-id> with the ID returned in step 1.
The remove then set order matters: set-routing-ism-domain does not overwrite a domain that is already mapped. This same create → remove → set flow is how you rotate the validator set on a live bridge — see Rotate the Multisig for the full 2/3 procedure including the Base Sepolia-side ISM update.
- Verify both validators have announced: check logs for
Announced storage location - Check that the relayer can reach both RPC endpoints
- Ensure the relayer key has ETH on Base Sepolia for gas fees
- Verify the ISM on both sides uses the correct validator address
- Check the relayer logs for
Unable to processorgas estimation failederrors
bitsong.index.from should be a recent height, not 1. With index.from: 1 on a multi-million-block chain, every agent rescans all history and exhausts the RPC's max_open_connections — HTTP stops responding (502 / connection reset) even though the chain itself is healthy. Reduce the load: recent index.from, chunk ~500, --ulimit nofile=65536:65536, and point at your own node or a node's direct IP rather than a TLS proxy.getLogs range. Set basesepolia.index.chunk within the limit: ≤ 999 for Coinbase CDP (caps at 1000 blocks — a 1000-block inclusive range is 1001 and is rejected) and ≤ 2000 for the public sepolia.base.org (which also rate-limits quickly). Prefer CDP on the /base-sepolia/ path (/base/ is mainnet 8453).References
- Hyperlane: Run Validators
- Hyperlane: AWS Signatures Bucket Setup
- Hyperlane: Validator Monitoring & Alerting
- Hyperlane: Agent Configuration Reference