HIL NixOS Setup
Eventually, we will support fully automating the setup process. But for now, one first needs to do some manual bootstrapping.
Installing NixOS to a liveusb
On the ASUS NUCs, they don't support MBR partitioned live usbs. But for some inexplicable reason the official NixOS installer only exists as a MBR partitioned disk. This means we need to build our own GPT/UEFI based NixOS live usb ;(
To work around this limitation of the official installer, we provide a liveusb image that has NixOS on it, via disko. The easiest way to get this liveusb image is from the CI artifacts, it is built by the Nix CI job.
Once you download it, unzip it and zstd --decompress
it, you will have a nixos.raw
file. Plug your flashdrive in, identity the disk (not partition) of the flashdrive
using either sudo fdisk -l
on linux or Disk Utility
on macos. For example,
/dev/sda
on linux (not /dev/sda1
) or /dev/diskX
on macos (not /dev/diskXsY
).
Run the following:
sudo cp nixos.raw /dev/<your-usb-disk>
This loads the liveusb onto the flashdrive.
Use the liveusb to install NixOS
Booting from the liveusb
This is the same as any other linux liveusb. Get into your boot menu using the function keys at boot, and select the USB from the boot options. Note: on the NUC, it can only boot GPT/UEFI based liveusbs, MBR ones won't show up in the boot options. This is why we had to build our own liveusb in the previous section. You will likely need to disable UEFI secure boot as well.
Configuring WIFI
You can use nmcli
.
nmcli device wifi connect "Your SSID Here" password "your password here"
Performing installation
Assuming your intended hostname is worldcoin-hil-sf-0
, run:
git clone https://github.com/worldcoin/orb-software.git --branch <your branch> ~/orb-software
sudo disko-install --flake ~/orb-software#worldcoin-hil-sf-0 --disk main /dev/nvme0n1
Setting up Teleport
- Request teleport token for a HIL in slack. You will receive a bash one-liner.
DO NOT RUN THE BASH, THIS IS AN EXAMPLE:
sudo bash -c "$(curl -fsSL https://teleport-cluster.orb.internal-tools.worldcoin.dev/scripts/ffffffffffffffffffffffffffffffff/install-node.sh)"
The command you received on slack should look like something of the above.
Instead of running the command, delete everything except the curl
command and then
redirect that to a file called teleport-install.sh
, for example:
curl -fsSL https://teleport-cluster.orb.internal-tools.worldcoin.dev/scripts/ffffffffffffffffffffffffffffffff/install-node.sh > teleport-install.sh
Be sure that teleport-install.sh
is put on the HIL, you can put it in the home directory
for now. Again, DO NOT RUN THIS SCRIPT.
- Place the following content on the HIL at
/etc/teleport.yaml
:
version: v3
teleport:
nodename: SED_HOSTNAME
data_dir: /var/lib/teleport
join_params:
token_name: SED_TOKEN
method: token
proxy_server: teleport-cluster.orb.internal-tools.worldcoin.dev:443
log:
output: stderr
severity: INFO
format:
output: text
ca_pin: sha256:e0974d24cee9f3494a7ca9d8496f5c67f3fc60ee4bff2f823d2bbdb2c0ea4a2c
diag_addr: ""
auth_service:
enabled: "no"
ssh_service:
enabled: "yes"
labels:
hostname: SED_HOSTNAME
commands:
proxy_service:
enabled: "no"
https_keypairs: []
https_keypairs_reload_interval: 0s
acme: {}
- run the following from the same directory that
teleport-install.sh
is at on the HIL:
TELEPORT_TOKEN="$(cat teleport-install.sh | grep -m1 -oP "^JOIN_TOKEN='\K[^']+")"
TELEPORT_HOSTNAME="$(hostname)"
sudo sed -i "s/SED_TOKEN/${TELEPORT_TOKEN}/" /etc/teleport.yaml
sudo sed -i "s/SED_HOSTNAME/${TELEPORT_HOSTNAME}/" /etc/teleport.yaml
This will edit the contents of /etc/teleport.yaml
to replace the SED_*
strings with
your hostname and the token.
You can sudo cat /etc/teleport.yaml
and inspect the file to see the new contents.
- Run
sudo rm -rf /var/lib/teleport
sudo systemctl restart teleport.service && sudo journalctl -fu teleport.service
You will see log messages from teleport. Make sure it looks roughly like everything is normal. Teleport should now be set up.
You will also need to make sure your machine's hostname matches the regex in our terraform config here.