Validator Deployment Guide - Mainnet

This guide will cover the end-to-end process of deploying a Helium Validator on Mainnet. It assumes you're starting from scratch. At the end, you'll have a Helium Validator deployed on the Network.

info

If at any point during this process you need assistance or have general questions, please join the #validator channel on Helium Discord.

Prerequisites

To successfully deploy a Helium Validator, you'll need to do the following:

Install the Helium CLI Wallet. You are welcome to build from source, but easiest to download the latest binary release.
Deploy a virtual server on a cloud service like AWS, GCP, Azure, or Digital Ocean. Or deploy a physical server on which to run the Validator. The recommended hardware and networking requirements can be found here.

Create Wallet

Once you have your Helium CLI Wallet installed locally, it's time to create your Helium Wallet. Run the following command to create it. (This command format assumes you're using the executable. If you've built the wallet from source, it will look slightly different.)

At any time, use -h or --help to get more help for a command.

warning

It is a best practice to create your Helium CLI wallet on a separate machine from the one where you will run your Validator.

helium-wallet create basic

You'll be prompted to supply a new passphrase to complete it. This is used to encrypt/decrypt the wallet.key file and is needed to sign transactions. Don't lose it.

This command will produce a wallet.key file on your machine, along with output similar to the following:

+-----------------------------------------------------+---------+--------+------------+
| Address                                             | Sharded | Verify | PwHash     |
+-----------------------------------------------------+---------+--------+------------+
| 1aP7nm6mGLMFtgiHQQxbPgKcBwnuQ6ehgTgTN8zjFuxByzJ8eA5 | false   | true   | Argon2id13 |
+-----------------------------------------------------+---------+--------+------------+

Next, run the info command to get all the relevant details.

helium-wallet info

The output will look similar to this:

+--------------------+-----------------------------------------------------+
| Key                | Value                                               |
+--------------------+-----------------------------------------------------+
| Address            | 1aP7nm6mGLMFtgiHQQxbPgKcBwnuQ6ehgTgTN8zjFuxByzJ8eA5 |
+--------------------+-----------------------------------------------------+
| Network            | mainnet                                             |
+--------------------+-----------------------------------------------------+
| Type               | ed25519                                             |
+--------------------+-----------------------------------------------------+
| Sharded            | false                                               |
+--------------------+-----------------------------------------------------+
| PwHash             | Argon2id13                                          |
+--------------------+-----------------------------------------------------+
| Balance            | 0.00000000                                          |
+--------------------+-----------------------------------------------------+
| DC Balance         | 0                                                   |
+--------------------+-----------------------------------------------------+
| Securities Balance | 0.00000000                                          |
+--------------------+-----------------------------------------------------+

Each Validator Requires 10000 HNT

Running a Validator on Mainnet requires you to stake 10000 HNT per Validator, so you need to already own 10,000 HNT before you can stake a Validator. After you've created your wallet, transfer 10001 HNT into it for each Validator you intend to run (the extra 1 HNT covers the staking transaction fee).

info

As is the case with every transaction on the Helium blockchain, staking requires a small fee paid in Data Credits. Read more about transaction fees here.

Run a Validator

With your wallet funded, it's time to deploy and run the actual Validator.

warning

If you haven't already, please review the Validator Technical Requirements. Specifically, on non-ARM systems AVX support is required. Verify that it exists on your host system by running the following:

grep avx /proc/cpuinfo

If nothing is returned from this command, your host system does not have AVX support and your Validator may not be stable.

You have three options for deploying the Validator. The deb package, a community member maintained package by PaulVMo, is the least complex to deploy and has very high performance. The Source build has the highest level of complexity to deploy, requires a higher level of Linux technical expertise, and also has very high performance. The docker version is less complex to deploy than Source, if you already have experience with Docker, but does not perform as well as the other two options. The Debian package is also the simplest to deploy and maintain when running multiple validator instances on a single host.

Performance

Performance comparison of three validators on a single 16 vCPU bare metal server, showing validation times from absorb_and_commit. Bottom line in graph is source build, middle is Debian package, top is quay docker image.

absorb_and_commit performance metrics

Choose one of the following options for your deployment:

Option 1

Deploy the Validator from deb package

NOTE: After deploying the deb package, come back to this page to perform the remaining steps starting at Enable Firewall on the server

Option 2

Deploy the Validator Using Docker

Start by updating your packager manager registry:

sudo apt-get update

And install Docker itself. (If needed, full directions on installing Docker on Ubuntu can be found here.)

sudo apt-get install docker.io

To avoid needing to use Docker with sudo privileges, add your user to the docker group, replacing $USER with your username here:

sudo usermod -aG docker $USER

Now that docker is installed and ready, let's run the Validator.

Run the Docker Container

Before running the container for the first time, it is mandatory to pick a "system mount point". This allows you to easily maintain your Validator's blockchain identity - specifically swarm keys - and blockchain state through image updates. Without doing this, you will lose these items when upgrading docker images.

The validator directory should be created on a persistent EBS volume (if you're using AWS) or something similar that lives across server lifetimes.

mkdir $HOME/validator_data

You can then use the run command to start your container for the first time:

docker run -d --init \
--restart always \
--publish 2154:2154/tcp \
--publish 8080:8080/tcp \
--name validator \
--network host \
--mount type=bind,source=$HOME/validator_data,target=/var/data \
quay.io/team-helium/validator:latest-validator-amd64

-d option runs in detached mode, which makes the command return or not; you may want to omit if you have a daemon manager running the docker for you.
--init to indicate that an init process should be used as the PID 1 in the container. This ensures the usual responsibilities of an init system, such as reaping zombie processes, are performed inside the created container. When docker exec commands are run, it's possible for these to create zombie processes on the host. This eliminates that issue. You can read more about this here and here.
--restart always asks Docker to keep the image running, starting the image on boot and restarting the image if it crashes. Depending on how you installed Docker in your system, it'll start on boot.
--publish 2154:2154/tcp maps the port 2154 on the docker container to 2154 on the host machine. This allows inbound connections on port 2154 to the host machine to be routed to the validator container. In addition to this mapping, it is up to you to ensure that this port is available from the public internet to the host machine. This will require modifying firewall and/or security group settings in your cloud provider of choice.
--publish 8080:8080/tcp maps the port 8080 on the docker container to 8080 on the host machine. This allows inbound connections on port 8080 to the host machine to be routed to the validator container. In addition to this mapping, it is up to you to ensure that this port is available from the public internet to the host machine. This will require modifying firewall and/or security group settings in your cloud provider of choice.
--name validator names the container, which makes interacting with the docker easier, but feel free to name the container whatever you want.
--mount the parameters above will mount the container's /var/data/ directory to the systems directory $HOME/validator_data. You will want this folder to be persistent across runs of the docker container as it will contain both the blockchain data and the miner key of your Validator.
quay.io/team-helium/validator:latest-validator-amd64 Lastly, this points to the docker image for the Validator that is tagged as the "latest" for the amd64 architecture. For arm architectures, replace with latest-validator-arm64.
If your host machine is behind a firewall that is performing Network Address Translation (NAT), add all four of these NAT_variables to assist the Validator in quickly determining the external IP address of the host machine. NOTE: You must add ALL four variables, for any one of them to take effect:
-e "NAT_INTERNAL_IP=192.168.0.139" - Replace 192.168.0.139 with your local IP shown when running ip addr`.
-e "NAT_EXTERNAL_IP=$( curl -s ipv4.icanhazip.com )" - Determines the external IP of the NAT. The curl command queries an external provider and returns the public IP address of the interface. You can replace $( curl -s ipv4.icanhazip.com ) with your external IP, if you have a static IP that you know will not change.
-e "NAT_INTERNAL_PORT=2154" - Replace the port number with the port you've assigned to the validator, if not using the default 2154.
-e "NAT_EXTERNAL_PORT=2154" - Replace the port number with the port you've opened on the firewall, if not using the default 2154.

Once you run the command above, docker will retrieve the latest Validator image, and start the docker process. Barring any errors, your Validator is running.

info

Please watch and respond to upgrade requests on the #validators-announcements channel on the Helium Discord server in a timely manner.

Interact with the Validator within the Container

You may want to interrogate the Validator or interact with it. Docker's exec command enables this. For example:

docker exec validator miner info height

As shown above, you can prepend docker exec validator to any of the commands documented or create an alias such as

alias miner="docker exec validator miner"

And start the container again as described above, but with the new release tag.

And thanks to the --mount option, the blockchain data and the validator keys are preserved through updates.

Upgrading your Docker container

To do manual upgrades, the latest miner tag can be found here.

To upgrade your docker container to the latest version, do the following. These commands will stop the running container then delete it; pull an updated Validator docker image from the repository; and finally delete any out of date Validator docker images.

docker stop validator && docker rm validator

docker pull quay.io/team-helium/validator:latest-validator-amd64

docker image prune -f

Finally, re-launch docker with the same command as above under the Run the Docker Container section.

info

Now that you have your Validator running, you need to actually Stake tokens to make it official.

Option 3

Deploy Validator from Source

Ok, brave soul. All of the information here can and will change but this document should help you get across the finish line. Here's what you'll need to build and deploy the miner from source:

miner
Erlang package (the latest 22 or 23 versions are supported, the 24 release candidate will NOT work)
libwxgtk package
git

Before we get started, make sure you're fully up to date:

sudo apt-get update -y

Next, let's install all the things.

Get you some Erlang (here we're specifying 22.3.1):

wget https://packages.erlang-solutions.com/erlang/debian/pool/esl-erlang_22.3.1-1~ubuntu~bionic_amd64.deb

Next, get the libwxgtk package:

wget http://mirrors.kernel.org/ubuntu/pool/universe/w/wxwidgets3.0/libwxgtk3.0-0v5_3.0.4+dfsg-3_amd64.deb

And finish things off by acquiring some wonderful dependencies:

sudo apt install -y libdbus-1-dev autoconf automake libtool flex libgmp-dev cmake libsodium-dev libssl-dev bison libsnappy-dev libclang-dev doxygen make cargo g++ libsctp1 libncurses5 libwxbase3.0-0v5 build-essential cmake libdbus-1-dev mosh vim parallel

Then we need to unpack and install all of this stuff. Start with libwxgtk:

sudo dpkg -i libwxgtk3.0-0v5_3.0.4+dfsg-3_amd64.deb
sudo apt update -y

Then on to Erlang. First unpack then install:

sudo dpkg -i esl-erlang_22.3.1-1~ubuntu~bionic_amd64.deb

sudo apt-get install -f

We also need to use a newer version of Rust. The one that comes by default with Ubuntu will not work.

#Remove rustc so we can install a newer version
sudo apt -y remove rustc

#Install newer version of rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-host x86_64-unknown-linux-gnu --default-toolchain stable --profile complete -y

#This allows you to proceed without logging out and logging in on the server
source $HOME/.cargo/env

Then navigate to your miner directory:

cd miner

Build the miner

./rebar3 as validator do release -n miner -v 1.x.x

Go get a cup of coffee and wait for the build to finish.

Start the miner:

_build/validator/rel/miner/bin/miner start

Build the miner as a tar package

./rebar3 as validator do tar -n miner -v 1.x.x

Load the Genesis Block

One last step. When building from source, you'll need to manually load the Helium blockchain genesis block once your miner is deployed and running. To do this, run the following:

wget https://snapshots.helium.wtf/genesis.mainnet
miner genesis load </absolute/path/to/genesis/block>

Note that in the above miner command, the miner binary here is nested within _build/validator/rel/miner/bin/miner.

Next Steps

After deploying one of the above three options, perform the remaining steps starting here:

Enable Firewall on the server

It's good practice to restrict access to the server and you should allow access only to ports that are needed. If you also use node_exporter or a different type of exporter for monitoring, you should add those ports as well.

#Enable Firewall
sudo ufw enable

#Allow connections to SSH Port. It's good practice to use a different port other than 22 for ssh connections
sudo ufw limit 22/tcp

#Allow connections to the Validator default port
sudo ufw allow 2154/tcp

#Allow connections for Light Hotspots
sudo ufw allow 8080/tcp

A Note on Errors

Source builders have sometimes reported error messages (after the miner has been running well for a while) referencing Too many open files. This can occur when a ulimit limitation is exceeded for the number of open files.

You can check your soft limit (often 1024 by default) with:

ulimit -Sn

Specifics instructions to increase the ulimit vary by OS (readily found with a Google search) and are beyond the scope of this guide. The generally accepted number among the Validator community is to set this to 128000 and ensure it is persistent across reboots.

Stake HNT to Your Validator

Now that your Validator node is running, the final step in the process is to formally stake HNT to your Validator. As part of the staking process the Validator address needs to both be in the staking transaction and sign the transaction. After a wallet stakes a validator node, the wallet becomes that node’s owner, has control over that validator node, and receives rewards.

info

First, double check your wallet balance to make sure you have the 10000 HNT required to stake, along with a few extra to cover the transaction fees.

helium-wallet balance

+-----------------------------------------------------+----------------+--------------+-----------------+
| Address                                             | Balance        | Data Credits | Security Tokens |
+-----------------------------------------------------+----------------+--------------+-----------------+
| 1aP7nm6mGLMFtgiHQQxbPgKcBwnuQ6ehgTgTN8zjFuxByzJ8eA5 | 10005.00000000 | 0            | 0.00000000      |
+-----------------------------------------------------+----------------+--------------+-----------------+

To stake HNT, we need to get the validator node address. Obtain it using the following:

miner peer addr

The resulting output will look like this (except with your specific validator address). The string after /p2p/ is your Validator address. For example:

/p2p/1YwLbGTCEhVbwKEehRVQRC8N3q35ydXTH1B6BQys5FB1paHssdR

We can now use this address with the Helium Wallet CLI validators stake command to formally stake the 10000 HNT required along with extra to cover the fee. Here's the full command using the Validator address from above as an example. (Make sure you replace it with yours.)

helium-wallet validators stake one 1YwLbGTCEhVbwKEehRVQRC8N3q35ydXTH1B6BQys5FB1paHssdR 10000 --commit

After running this, you'll need to input your wallet passphrase to sign the transaction.

And with that, you're done. Congratulations! You're running a Helium Validator on Mainnet!

Overriding Default Configuration

Depending on your deployment, you may wish to override default configuration of the validator. There are multiple ways to edit the configuration including setting environment variables and overlaying the config files.

Environment Variables

At runtime, the miner startup process will read the following environment variables and, if present, override the default values. This is the most straightforward way to configure your validator. Environment variables may be set in the shell session used to launch the miner, within a systemd service file, part of a docker run command, or included in a docker-compose file.

LISTEN_ADDRESS (default: /ip4/0.0.0.0/tcp/2154/) - the address that the miner will listen to on the P2P network; currently only a single IP4 address is supported and it must be in multiaddr format. This address must be publicly accessible.
BASE_DIR (default: data) - Directory location of data files. May be absolute or relative path. Relative paths are relative to miner install location.
LOG_ROOT (default: log) - Directory location of log files. May be absolute or relative path. Relative paths are relative to miner install location.
UPDATE_DIR (default: /opt/miner/update) - Location that the miner looks in for a genesis block on startup. If a new genesis block found, the existing data dir will be removed and the new genesis block loaded.
HOTFIX_DIR (default: /opt/miner/hotfix) - Location that the miner looks in for code hotfixes.
GRPC_PORT (default: 8080) - Port used by light gateways to interact with the validator. This port must be publicly accessible.
JSONRPC_IP (default: 127,0,0,1) - IP address that the JSON RPC endpoint listens on. JSRONRPC is used for monitoring and interacting with the validator.
JSONRPC_PORT (default: 4467) - Port number that the JSON RPC endpoint listens on.
VALIDATION_WIDTH - Maxium number of concurrent transaction validations that the miner will do when absorbing blocks. Default is N/2 + 1 where N is the number of Erlang schedulers which is by default the number of CPUs/vCPUs.
SIDECAR_PARALLELISM_LIMIT (default: 3) - Maximum number of concurrent transaction validations the miner will attempt while in consensus group.
If your host machine is behind a firewall that is performing Network Address Translation (NAT), add all four of these NAT_variables to assist the Validator in quickly determining the external IP address of the host machine.
NOTE: You must add ALL FOUR variables, for any one of them to take effect.
NOTE: Expand Run the Docker Container section for details on adding these variables to a validator running in Docker.
NAT_INTERNAL_IP=192.168.0.139 - Replace 192.168.0.139 with your local IP shown when running ip addr.
NAT_EXTERNAL_IP=$( curl -s ipv4.icanhazip.com ) - Determines the external IP of the NAT. The curl command queries an external provider and returns the public IP address of the interface. You can replace $( curl -s ipv4.icanhazip.com ) with your external IP, if you have a static IP that you know will not change.
NAT_INTERNAL_PORT=2154 - Replace the port number with the port you've assigned to the validator, if not using the default 2154.
NAT_EXTERNAL_PORT=2154 - Replace the port number with the port you've opened on the firewall, if not using the default 2154.

Erlang sys.config overlay

For more advanced configuration of the miner process, the Erlang application configuration files can also be edited. For this approach, it is recommend to add or update configuration parameters to releases/<version number>/sys.config.src. This file overlays (overrides) configuration on the base config file which is config/sys.config which includes all parameters used by the miner.

Verifying Validator Victory

Now that you're up and running with a Validator, there are a few things you can examine to make sure things are hunky dory.

Time Expectations

Before your Validator will show Online status in Explorer, it must first sync up to the tip of the chain AND send a Heartbeat. The first Heartbeat will not be sent until after it syncs to the tip of the chain. You can check to see when your first heartbeat is sent within Explorer, on the Activity tab for your Validator. The Heartbeat is a transaction that each validator sends to the consensus group, approximately every 100 blocks, to assert that it is still alive and synced to the tip of the chain.

The blockchain takes time. The distributed nature of verifying transactions into the blockchain takes time. The validator information gossiped on the blockchain takes time. APIs that run to update the explorer are scheduled and have intervals. Basically, you won't be able to run the validator and instantly see that everything is "good".

After the validator has been up and running for 15 minutes, proceed with some of the steps below.

Check the API

The Validator API provides several useful calls to help monitor your Validator and the state of the Mainnet.

https://api.helium.io/v1/validators/

https://api.helium.io/v1/validators/<validator_address>

will be very useful.

You should see JSON output that looks similar to this. You are looking for "online": "online" and "stake_status": "staked"

{
  "data": {
    "version_heartbeat": 2,
    "status": {
    "online": "online",
    "listen_addrs": [
        "/ip4/1.2.3.4/tcp/2154"
    ],
    "height": xxxx
  },
  "stake_status": "staked",
  "stake": 1000000000000,
  "owner": "1aHpEUzcsBvjw1xv8PnoYAYM5yrodqbXKwBitHS8hamWT4TQVDp",
  "last_heartbeat": xxxx,
  "block": xxx,
  "address": "1ZobTUK43hjTwTvwihEoCvh3SuuvfGp9AAR85c8mQdpULjntYWH"
  }
}

Status Commands to Run on the Validator Server

Note that you may need to adjust these if you're running in Docker.

miner info p2p_status

+---------+------+
|  name   |result|
+---------+------+
|connected| yes  |
|dialable | yes  |
|nat_type | none |
| height  | 2447 |
+---------+------+

connected means you have at least one connection to a peer (outgoing connections OK)
dialable means peers can reach you (incoming connections OK)
nat_type of none is best for validators. Anything else means the validator code thinks it is behind some kind of NAT
height is the currently synced block

miner peer book -s

+------------------------------------------------+-------------+----------+---------+---+----------+
|                    address                     |    name     |listen_add|connectio|nat|last_updat|
+------------------------------------------------+-------------+----------+---------+---+----------+
|/p2p/1YwLbGTCEhVbwKEehRVQRC8N3q35ydXTH1B6BQys5FB|short-umber-b|    1     |    4   |non| 111.88s  |
+------------------------------------------------+-------------+----------+---------+---+----------+

+----------------------------+
| listen_addrs (prioritized) |
+----------------------------+
|/ip4/173.230.156.39/tcp/2154|
+----------------------------+

+-----------------+--------------------+---------------------------------------+-------------------+
|      local      |       remote       |                  p2p                  |       name        |
+-----------------+--------------------+---------------------------------------+-------------------+
|/ip4/172.17.0.2/t|/ip4/50.16.94.64/tcp|/p2p/1Y9MoitLaEbXPdrZZogURSNQ54eXcp2Ljv|agreeable-tweed-pir|
|/ip4/172.17.0.2/t|/ip4/143.110.235.209|/p2p/1YAFBayNk8bXkkEpqXdxu73kFTjtMqXKTy|prehistoric-hemp-ra|
|/ip4/172.17.0.2/t|/ip4/73.2.34.208/tcp|/p2p/1YEC3g9Hep4hzbDwLFbJ4SqrthTcjGDWdP|wonderful-arctic-ch|
|/ip4/172.17.0.2/t|/ip4/172.90.214.198/|/p2p/1YLJC8DSN6SF17S9nXfYEwTXKtoXpgqBSX|howling-ultraviolet|
|/ip4/172.17.0.2/t|/ip4/13.59.168.136/t|/p2p/1YZhCNPj181YL21aHECXLS1zvDzVY43px9|macho-chocolate-but|
+-----------------+--------------------+---------------------------------------+-------------------+

You're looking for listen_addrs. If you don't have at least one, your Validator hasn't settled on how to tell other peers to reach it. Often this can take 15-30 mins, sometimes longer.

Backup your `swarm_key`

Your validator's swarm_key is located in the validator_data/miner directory

In order to copy the swarm_key locally to your personal computer, you'll need to use scp (outside of/instead of an ssh session). Replace the user name and IP address (root@192.0.2.1) in the example below with the actual user name and IP address of your docker container instance.

scp root@192.0.2.1:validator_data/miner/swarm_key .

The . at the end of the example above is a placeholder for your current directory on your local machine. This is where your swarm_key will be copied to on your local machine. Use the command pwd to determine the location where the file was copied to.

A Note on the Purpose of a `swarm_key`

The swarm_key equates to your validator's unique identity on the Helium blockchain. Backing up the swarm_key enables you to maintain your validator's identity in the event that your node becomes compromised in some way, or needs to be rebuilt on another server for any reason.

Check the Explorer

Head over to the Validator Mainnet Explorer and search for your Validator. This may take 10 to 15 minutes from the time that your Validator is both online and staked before it appears on explorer. You should see your 3-word validator name (short-umber-bull in the example above) listed. It may be easier to sort by #, descending, as your new Validator will be near the top of the list. Remember you can view the name of your validator through miner info summary or just miner info name.

Troubleshooting Guide

If your validator does not show online in Explorer after 24 hours, view the Validator Troubleshooting Guide here.

Visit the Helium Validator Status Page (and sign up for alerts, if you haven't already). This status page will provide the latest updates on important issues that impact validators like chain halts or other major events.

Thank you for being a part of the Helium Validators initiative.

info

Prerequisites

Create Wallet

warning

Each Validator Requires 10000 HNT

info

Run a Validator

warning

Performance

Choose one of the following options for your deployment:

Option 1

Option 2

Run the Docker Container

info

Interact with the Validator within the Container

Upgrading your Docker container

info

Option 3

Next, let's install all the things.

Load the Genesis Block

Next Steps

Enable Firewall on the server

A Note on Errors

Stake HNT to Your Validator

info

Overriding Default Configuration

Environment Variables

Erlang sys.config overlay

Verifying Validator Victory

Time Expectations

Check the API

Status Commands to Run on the Validator Server

Backup your swarm_key

A Note on the Purpose of a swarm_key

Check the Explorer

Troubleshooting Guide

Backup your `swarm_key`

A Note on the Purpose of a `swarm_key`