Hyperlane

Validator V2

Production AWS guide for running BitSong Hyperlane validators with KMS signing, S3 checkpoint storage, private RPCs, and monitoring.

This guide explains how to run the BitSong Hyperlane validator stack in production on AWS.

It is based on the official Hyperlane validator guides for running validators, AWS signature storage, and monitoring, adapted for the BitSong crescendo-1 testnet and Base Sepolia.

This is the production-style guide. It does not use plaintext EVM validator private keys or local checkpoint storage. Use AWS KMS for validator signing and S3 for checkpoint storage.

What You Are Running

Hyperlane validators are not consensus validators. They are off-chain agents that watch an origin chain's Mailbox, sign message checkpoints, and publish those signatures where relayers can read them.

For a two-way bridge between BitSong and Base Sepolia, run two validator containers:

Container	Origin chain	What it signs	Why it matters
`hyperlane-validator-bitsong`	BitSong `crescendo-1`	BitSong Mailbox checkpoints	Needed for messages from BitSong to Base Sepolia
`hyperlane-validator-basesepolia`	Base Sepolia	Base Sepolia Mailbox checkpoints	Needed for messages from Base Sepolia to BitSong

Each validator needs:

A secure checkpoint signing key in AWS KMS
A publicly readable S3 bucket for signed checkpoint files
Private, reliable RPC endpoints for the chain it validates
A persistent local database directory
Prometheus metrics enabled

The same AWS KMS validator key can be used for both validator containers if the same EVM validator address is registered in both ISM configurations. Use separate S3 folders and separate database paths for each container.

Production Checklist

Before you start containers, confirm every item:

:checked-box: Your BitSong full node is synced, or you have private BitSong RPC and gRPC endpoints
:checked-box: You have a private Base Sepolia RPC endpoint; do not depend on public RPCs in production
:checked-box: The BitSong Hyperlane Mailbox, MerkleTreeHook, IGP, and ISM IDs are known
:checked-box: The Base Sepolia Hyperlane contract addresses are known
:checked-box: The validator EVM address from AWS KMS is registered in the relevant MultisigISM
:checked-box: The AWS IAM user has permission to use the KMS key
:checked-box: The S3 bucket is publicly readable and writable by the validator IAM user
:checked-box: The Cosmos announcement signer account has TBTSG for BitSong gas
:checked-box: The validator KMS address has Base Sepolia ETH for the Base Sepolia announcement transaction
:checked-box: Metrics are scraped by Prometheus or your monitoring system

AWS Server

Use one small EC2 instance per operator, or separate instances per region if you want high availability.

Resource	Recommended minimum
Instance	`t3.small` or better
CPU	2 vCPU
RAM	2 GB
Disk	20 GB gp3
OS	Ubuntu 22.04 LTS or 24.04 LTS

Security group rules:

Allow SSH only from your operator IP
Do not expose Prometheus metrics publicly
Allow outbound HTTPS for AWS, RPC providers, and container image pulls
If the BitSong node is on the same server, bind RPC/gRPC locally where possible

Install required packages:

Terminal

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg jq awscli

Install Foundry for the cast command used to derive AWS KMS EVM addresses:

Terminal

curl -L https://foundry.paradigm.xyz | bash
source ~/.bashrc
foundryup
cast --version

Install Docker:

Terminal

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER

Log out and back in, then verify Docker:

Terminal

docker --version
docker run --rm hello-world

AWS Resources

Create the IAM User

Create an IAM user dedicated to the validator stack:

IAM user name

hyperlane-validator-bitsong

Create an access key for Application running outside AWS and save:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

On the server, export the AWS environment:

Terminal

export AWS_ACCESS_KEY_ID="<your-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
export AWS_REGION="us-east-1"

Use the same AWS region for KMS and S3 unless you have a specific operational reason to split them.

Create the KMS Validator Key

Create one customer-managed KMS key:

Setting	Value
Key type	Asymmetric
Key usage	Sign and verify
Key spec	`ECC_SECG_P256K1`
Alias	`hyperlane-validator-signer-bitsong`

Grant key usage permissions to the IAM user hyperlane-validator-bitsong.

Verify the validator EVM address:

Terminal

AWS_KMS_KEY_ID=alias/hyperlane-validator-signer-bitsong \
cast wallet address --aws

Save the result:

Terminal

export VALIDATOR_ADDR="0x<kms-derived-validator-address>"

Register this exact EVM address in the MultisigISM. If the ISM contains a placeholder or a different address, the bridge will not verify messages signed by your validator.

Create the S3 Checkpoint Bucket

Create a bucket:

S3 bucket

hyperlane-validator-signatures-bitsong

Use the same region as the KMS key. Keep ACLs disabled.

In Block Public Access settings, use:

:unchecked-box: Block all public access
:checked-box: Block public access through new ACLs
:checked-box: Block public access through any ACLs
:unchecked-box: Block public access through new public bucket policies
:unchecked-box: Block public and cross-account access through any public bucket policies

Add this bucket policy, replacing ${BUCKET_ARN} and ${USER_ARN}:

Bucket Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "${BUCKET_ARN}",
        "${BUCKET_ARN}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "${USER_ARN}"
      },
      "Action": [
        "s3:DeleteObject",
        "s3:PutObject"
      ],
      "Resource": "${BUCKET_ARN}/*"
    }
  ]
}

Validate write access from the server:

Terminal

aws s3 cp /etc/hostname s3://hyperlane-validator-signatures-bitsong/healthcheck.txt
aws s3 ls s3://hyperlane-validator-signatures-bitsong/
aws s3 rm s3://hyperlane-validator-signatures-bitsong/healthcheck.txt

Cosmos Announcement Signer

BitSong is a Cosmos chain. The validator checkpoint signature uses AWS KMS, but the BitSong announcement transaction still needs a Cosmos signer key.

Create a dedicated key:

Terminal

bitsongd keys add hyperlane-signer --keyring-backend test

Export the private key as hex:

Terminal

COSMOS_SIGNER_KEY=0x$(bitsongd keys export hyperlane-signer --unarmored-hex --unsafe --keyring-backend test 2>&1 | tail -1)
export COSMOS_SIGNER_KEY

Fund the signer with TBTSG:

Terminal

bitsongd tx bank send <your-funded-key> $(bitsongd keys show hyperlane-signer -a --keyring-backend test) 10000000utbtsg \
  --from <your-funded-key> \
  --keyring-backend test \
  --chain-id crescendo-1 \
  --node tcp://localhost:26657 \
  --fees 10000utbtsg \
  -y

Environment File

Create the production working directory:

Terminal

mkdir -p $HOME/hyperlane-bitsong/{config,db}

Create an environment file:

$HOME/hyperlane-bitsong/validator.env

AWS_ACCESS_KEY_ID=<your-access-key-id>
AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
AWS_REGION=us-east-1
COSMOS_SIGNER_KEY=0x<your-cosmos-signer-private-key>
S3_BUCKET=hyperlane-validator-signatures-bitsong
HYPERLANE_IMAGE=ghcr.io/hyperlane-xyz/hyperlane-agent:agents-v2.2.0

Restrict access:

Terminal

chmod 600 $HOME/hyperlane-bitsong/validator.env

Pull the Hyperlane agent image:

Terminal

source $HOME/hyperlane-bitsong/validator.env
docker pull --platform linux/amd64 $HYPERLANE_IMAGE

Agent Configuration

Create the agent config:

Terminal

nano $HOME/hyperlane-bitsong/config/agent-config.json

Use this template and replace the BitSong IDs with the values from the bridge deployment:

$HOME/hyperlane-bitsong/config/agent-config.json

{
  "chains": {
    "bitsong": {
      "name": "bitsong",
      "chainId": "crescendo-1",
      "domainId": 7171,
      "protocol": "cosmosNative",
      "bech32Prefix": "bitsong",
      "slip44": 639,
      "contractAddressBytes": 32,
      "canonicalAsset": "utbtsg",
      "rpcUrls": [
        { "http": "http://127.0.0.1:26657" }
      ],
      "grpcUrls": [
        { "http": "http://127.0.0.1:9090" }
      ],
      "nativeToken": {
        "name": "BitSong",
        "symbol": "BTSG",
        "decimals": 6,
        "denom": "utbtsg"
      },
      "gasPrice": { "amount": "0.025", "denom": "utbtsg" },
      "gasMultiplier": "1.5",
      "blocks": {
        "confirmations": 1,
        "estimateBlockTime": 6,
        "reorgPeriod": 1
      },
      "index": { "from": 1, "chunk": 50 },
      "mailbox": "<bitsong-mailbox-id>",
      "validatorAnnounce": "<bitsong-mailbox-id>",
      "merkleTreeHook": "<bitsong-merkle-tree-hook-id>",
      "interchainGasPaymaster": "<bitsong-igp-id>"
    },
    "basesepolia": {
      "name": "basesepolia",
      "chainId": 84532,
      "domainId": 84532,
      "protocol": "ethereum",
      "rpcUrls": [
        { "http": "https://<your-private-base-sepolia-rpc>" }
      ],
      "nativeToken": {
        "name": "Ether",
        "symbol": "ETH",
        "decimals": 18
      },
      "blocks": {
        "confirmations": 1,
        "estimateBlockTime": 2,
        "reorgPeriod": 1
      },
      "index": { "from": 13850000, "chunk": 1999 },
      "mailbox": "0x6966b0E55883d49BFB24539356a2f8A673E02039",
      "validatorAnnounce": "0x20c44b1E3BeaDA1e9826CFd48BeEDABeE9871cE9",
      "merkleTreeHook": "0x86fb9F1c124fB20ff130C41a79a432F770f67AFD",
      "interchainGasPaymaster": "0x28B02B97a850872C4D33C3E024fab6499ad96564"
    }
  }
}

For Cosmos chains, configure both RPC and gRPC endpoints. For production, point these to your own BitSong node or a private provider. Public RPCs are not production infrastructure.

If you need to recover the BitSong IDs:

Terminal

bitsongd query hyperlane mailboxes --output json --node tcp://localhost:26657 | jq '.mailboxes'
bitsongd query hyperlane hooks merkle-tree-hooks --output json --node tcp://localhost:26657 | jq '.merkle_tree_hooks'
bitsongd query hyperlane hooks igps --output json --node tcp://localhost:26657 | jq '.igps'

Validate the JSON:

Terminal

jq . $HOME/hyperlane-bitsong/config/agent-config.json > /dev/null

Start Validators

Create persistent database directories:

Terminal

mkdir -p $HOME/hyperlane-bitsong/db/validator-bitsong
mkdir -p $HOME/hyperlane-bitsong/db/validator-basesepolia

BitSong Validator

This validator watches BitSong and writes BitSong checkpoint signatures to S3.

Terminal

source $HOME/hyperlane-bitsong/validator.env

docker run -d \
  --name hyperlane-validator-bitsong \
  --restart unless-stopped \
  --network host \
  --user $(id -u):$(id -g) \
  --env-file $HOME/hyperlane-bitsong/validator.env \
  -e CONFIG_FILES=/config/agent-config.json \
  -v $HOME/hyperlane-bitsong/config/agent-config.json:/config/agent-config.json:ro \
  -v $HOME/hyperlane-bitsong/db/validator-bitsong:/hyperlane_db \
  --log-driver json-file \
  --log-opt max-size=50m \
  --log-opt max-file=5 \
  $HYPERLANE_IMAGE \
  ./validator \
  --db /hyperlane_db \
  --originChainName bitsong \
  --reorgPeriod 1 \
  --interval 10 \
  --validator.type aws \
  --validator.region $AWS_REGION \
  --validator.id alias/hyperlane-validator-signer-bitsong \
  --chains.bitsong.signer.type cosmosKey \
  --chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
  --chains.bitsong.signer.prefix bitsong \
  --checkpointSyncer.type s3 \
  --checkpointSyncer.bucket $S3_BUCKET \
  --checkpointSyncer.region $AWS_REGION \
  --checkpointSyncer.folder bitsong \
  --metrics-port 9090 \
  --log.format json \
  --log.level info

Base Sepolia Validator

This validator watches Base Sepolia and writes Base Sepolia checkpoint signatures to S3.

Terminal

source $HOME/hyperlane-bitsong/validator.env

docker run -d \
  --name hyperlane-validator-basesepolia \
  --restart unless-stopped \
  --network host \
  --user $(id -u):$(id -g) \
  --env-file $HOME/hyperlane-bitsong/validator.env \
  -e CONFIG_FILES=/config/agent-config.json \
  -v $HOME/hyperlane-bitsong/config/agent-config.json:/config/agent-config.json:ro \
  -v $HOME/hyperlane-bitsong/db/validator-basesepolia:/hyperlane_db \
  --log-driver json-file \
  --log-opt max-size=50m \
  --log-opt max-file=5 \
  $HYPERLANE_IMAGE \
  ./validator \
  --db /hyperlane_db \
  --originChainName basesepolia \
  --reorgPeriod 1 \
  --interval 10 \
  --validator.type aws \
  --validator.region $AWS_REGION \
  --validator.id alias/hyperlane-validator-signer-bitsong \
  --chains.basesepolia.signer.type aws \
  --chains.basesepolia.signer.region $AWS_REGION \
  --chains.basesepolia.signer.id alias/hyperlane-validator-signer-bitsong \
  --checkpointSyncer.type s3 \
  --checkpointSyncer.bucket $S3_BUCKET \
  --checkpointSyncer.region $AWS_REGION \
  --checkpointSyncer.folder basesepolia \
  --metrics-port 9091 \
  --log.format json \
  --log.level info

The Base Sepolia validator sends an on-chain announcement transaction. Fund the KMS-derived VALIDATOR_ADDR with Base Sepolia ETH before starting this container.

Verify Operation

Check that both containers are running:

Terminal

docker ps --filter "name=hyperlane-validator" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Read logs:

Terminal

docker logs -f --tail 100 hyperlane-validator-bitsong
docker logs -f --tail 100 hyperlane-validator-basesepolia

Check for checkpoint files:

Terminal

source $HOME/hyperlane-bitsong/validator.env
aws s3 ls s3://$S3_BUCKET/bitsong/ --recursive
aws s3 ls s3://$S3_BUCKET/basesepolia/ --recursive

Check BitSong storage announcement:

Terminal

bitsongd query hyperlane ism announced-storage-locations \
  <bitsong-mailbox-id> $(echo $VALIDATOR_ADDR | tr '[:upper:]' '[:lower:]') \
  --output json \
  --node tcp://localhost:26657 | jq '.storage_locations'

A non-empty storage_locations array means the validator announced where relayers can find its checkpoints.

Checkpoint files appear when messages are dispatched from the origin Mailbox. An empty S3 folder can be normal before any bridge activity, but the containers should still stay healthy and logs should not show signing, RPC, KMS, S3, or announcement errors.

Monitoring

Hyperlane validators expose Prometheus metrics. This guide uses:

Container	Metrics port
`hyperlane-validator-bitsong`	`9090`
`hyperlane-validator-basesepolia`	`9091`

Example Prometheus scrape config:

prometheus.yml

scrape_configs:
  - job_name: hyperlane-validator-bitsong
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: hyperlane-validator-basesepolia
    static_configs:
      - targets: ["127.0.0.1:9091"]

Important metrics:

Metric	What to watch
`hyperlane_latest_checkpoint`	Should increase when the origin chain dispatches messages
`hyperlane_block_height`	Should keep increasing while the RPC is healthy
`hyperlane_contract_sync_liveness`	Should continue moving; a flat value means the indexing loop may be stuck
`hyperlane_contract_sync_block_height`	Should follow the chain's block production
`hyperlane_cursor_current_block`	Should advance as the validator indexes
`hyperlane_span_events_total{agent="validator", event_level="error"}`	Alert on repeated increases
`hyperlane_span_events_total{agent="validator", event_level="warn"}`	Investigate sustained increases

Recommended alerts:

hyperlane_block_height does not increase for 30 minutes
hyperlane_contract_sync_liveness is flat for 15 minutes
hyperlane_latest_checkpoint does not increase while messages are being dispatched and block height is increasing
Error logs increase repeatedly over a 1 hour window
No checkpoint objects appear in S3 after known Mailbox dispatches

When an alert fires, check logs first, then RPC health, AWS KMS access, S3 permissions, and account balances.

Optional Relayer

Validators provide security signatures. A relayer provides transport. If your team is also operating the bridge relayer, run it with a separate KMS key so relayer gas spending is isolated from validator signing.

Create another KMS key:

KMS alias

hyperlane-relayer-bitsong

Fund the relayer KMS address with Base Sepolia ETH:

Terminal

AWS_KMS_KEY_ID=alias/hyperlane-relayer-bitsong cast wallet address --aws

Start the relayer:

Terminal

mkdir -p $HOME/hyperlane-bitsong/db/relayer
source $HOME/hyperlane-bitsong/validator.env

docker run -d \
  --name hyperlane-relayer \
  --restart unless-stopped \
  --network host \
  --user $(id -u):$(id -g) \
  --env-file $HOME/hyperlane-bitsong/validator.env \
  -e CONFIG_FILES=/config/agent-config.json \
  -v $HOME/hyperlane-bitsong/config/agent-config.json:/config/agent-config.json:ro \
  -v $HOME/hyperlane-bitsong/db/relayer:/hyperlane_db \
  --log-driver json-file \
  --log-opt max-size=50m \
  --log-opt max-file=5 \
  $HYPERLANE_IMAGE \
  ./relayer \
  --db /hyperlane_db \
  --relayChains bitsong,basesepolia \
  --gaspaymentenforcement '[{"type":"none"}]' \
  --chains.bitsong.signer.type cosmosKey \
  --chains.bitsong.signer.key $COSMOS_SIGNER_KEY \
  --chains.bitsong.signer.prefix bitsong \
  --chains.basesepolia.signer.type aws \
  --chains.basesepolia.signer.region $AWS_REGION \
  --chains.basesepolia.signer.id alias/hyperlane-relayer-bitsong \
  --metrics-port 9092 \
  --log.format json \
  --log.level info

--gaspaymentenforcement '[{"type":"none"}]' is acceptable for this testnet guide. For a production mainnet bridge, configure gas payment enforcement so relayers are compensated.

Operations

Restart one validator:

Terminal

docker restart hyperlane-validator-bitsong

Stop the stack:

Terminal

docker stop hyperlane-validator-bitsong hyperlane-validator-basesepolia hyperlane-relayer

Upgrade the Hyperlane image:

Terminal

source $HOME/hyperlane-bitsong/validator.env
docker pull --platform linux/amd64 $HYPERLANE_IMAGE
docker stop hyperlane-validator-bitsong hyperlane-validator-basesepolia
docker rm hyperlane-validator-bitsong hyperlane-validator-basesepolia

Then rerun the start commands. Keep the database directories; do not delete them during normal upgrades.

Back up:

$HOME/hyperlane-bitsong/config/agent-config.json
$HOME/hyperlane-bitsong/validator.env
The AWS KMS key alias and key ID
S3 bucket name and policy
The Cosmos signer key mnemonic or private key

Troubleshooting

Check logs:

Terminal

docker logs hyperlane-validator-bitsong

Common causes are invalid JSON config, unreachable RPC/gRPC endpoints, missing AWS credentials, wrong KMS alias, or missing COSMOS_SIGNER_KEY.

The KMS-derived validator address needs Base Sepolia ETH. Query it:

Terminal

AWS_KMS_KEY_ID=alias/hyperlane-validator-signer-bitsong cast wallet address --aws

Fund that address and restart hyperlane-validator-basesepolia.

Final Validation

You are production-ready for this testnet when:

Both validator containers restart automatically after reboot
Both validators expose metrics and are scraped
S3 contains checkpoint objects after bridge messages are dispatched
Validator announcements are visible on-chain
No validator errors are increasing over time
The relayer, if operated by you, can read checkpoints and deliver messages in both directions

References

Edit this pageorReport an issue

Testing Warp Route

Test cross-chain token transfers between BitSong and Base Sepolia using your deployed Hyperlane warp route.

Introduction

Developer overview of the BitSong crescendo-1 testnet with chain specs, endpoints, and tools.