Detailed HowTo: PXE boot multiple RPi (Birthday Edition 🍰)

introduction

I’ve always been intrigued by the idea of creating a home data center with Raspberry PIs as a computable instances. However, managing individuals node with SD cards is not fun 😼. So decided to try PXE (Preboot Execution Environment) booting. Instead of dealing with each SD card individually, PXE protocol lets me manage all my RPis from one central server, making updates and further patches easier. In this post, I’ll share how you can set up PXE booting for your RPi fleet and enjoy these benefits to :)

my appliances

I;ll provide your with examples that are specific to my devices but once you would grasp the idea you could to the same on your set up.

1 x UDM Dream Machine SE
1 x Synology NAS
4 x RaspberryPI 5
4 x PoE Hat for RaspberryPI 5
1 x microSD card
(optional) 1 x monitor + HDMI to mini HDMI cable

plan

After a series of trials and errors, I found out the recipe that worked for me. I am happy to share it with you. :)

[UDM] create a separate subnet (e.g VLAN ) for RPi’s and NAS
[UDM] configure DHCP server
[SD card] Write OS
[RPi] update EEPROM firmware for all RPi machines
[NAS] enable NFS service
[NAS] create SharedFolder
[NAS] enable TFTP service
[NAS] enable rsync
[NAS] create folders node1/rootfs in TFTP root folder
[RPi node1] mount NFS shared folder /volume1/RPI5-PXE/node1/rootfs to node1
[RPI node1] upload / root filesystem to [NAS] /volume1/RPI5-PXE/node1/rootfs
[NAS] copy the content of /volume1/RPI5-PXE/node1/rootfs/boot/firmware into the /volume1/RPI5-PXE/node1/
[NAS] modify cmdline.txt in /volume1/RPI5-PXE/node1/
[NAS] modify fstab in /volume1/RPI5-PXE/node1/rootfs/etc/
repeat steps from 10 - 14 for other nodes{2,3,4,..n}
final step

1 [UDM] create a separate subnet (e.g VLAN ) for RPi’s and NAS

Network segmentation gives me better devices organization / isolation, security benefits (firewall rules, content filtering, etc). For now, in my home network, I have 2 subnets: default and cluster. default includes all devices that connected to home mesh, including phones, PlayStations, IoT devices, ect. And cluster is a dedicated segment with network devices dedicated to my home data center project. Our main focus will be concentrated in cluster subnet today 🎯. Assign cluster subent to specific ports. Bottom row is RPi’s ports. In the top row, the port on the far right belongs to the NAS.

2 [UDM] configure DHCP server

DHCP server plays is big role in PXE envirounment. And it requires extra configuraions because it not only provide IP address, IP mask, etc to all booting DHCP clients but also provides TFTP server IP address and name of NBP only to identified booting clients. Practicly speaking, you need to configure DHCP Options which iusually is and advanced section in your DHCP settings.

Adjust your DHCP Options to these values:

code	value
60	PXEClient
66	192.168.101.253
67	bootcode.bin
43	1.2.3.4

43 - Vendor Specific Information

60 - Vendor Class Identifier

66 - TFTP Server Name

67 - Bootfile Name

I would like to point your attention to option 43. In the official RPi tech spec it was told:

Vendor-Option Option 43 contains the important part of the reply. This must contain the string “Raspberry Pi Boot”. Due to a bug in the boot ROM, you may need to add three spaces to the end of the string.

It might work in your case, but in my no. This field on my router has a specific validation rule and it accepts only IP address format. So I set 1.2.3.4

3 [SD card] Write OS

You can install any OS using Raspberry Pi Imager. My choice is Raspberry PI OS (Lite) 64-bit. A port of Debian Bookworm with no desktop environment.

4 [RPi] update EEPROM firmware for all RPi machines

Install SD card into RPi node1. Connect to node1 via SSH. Run to see your current EEPROM settings.

vcgencmd bootloader_config

Run command below to extract settings into the file bootconf.txt

cp /lib/firmware/raspberrypi/bootloader-2712/stable/pieeprom-2024-06-05.bin pieeprom.bin
rpi-eeprom-config pieeprom.bin > bootconf.txt

Modify bootconf.txt :

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0x21
TFTP_PREFIX=1
TFTP_PREFIX_STR=node1/
PXE_OPTION43=1.2.3.4

Apply new EEPROM config.

rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt pieeprom.bin
rpi-eeprom-update -d -f ./pieeprom-new.bin
reboot

Do the same steps for all machines. But change TFTP_PREFIX_STR=node{2,3,4}/

5 [NAS] enable NFS service

6 [NAS] create SharedFolder

7 [NAS] enable TFTP service

8 [NAS] enable rsync

9 [NAS] create folders `node1/rootfs` in TFTP root folder

SSH to your NAS and create subfolders for each node

drwxrwxrwx+ 1 root   root    52 Jul 25 18:31 .
drwxr-xr-x  1 root   root   680 Jul 25 17:41 ..
drwxrwxrwx+ 1 yuklia users 1280 Jul 27 23:12 node1
drwxrwxrwx+ 1 yuklia users 1232 Jul 25 18:11 node2
drwxrwxrwx+ 1 yuklia users 1232 Jul 25 18:19 node3
drwxrwxrwx+ 1 yuklia users 1232 Jul 25 18:39 node4
yuklia@nas:/volume1/RPi5-PXE$

10 [RPi node1] mount NFS shared folder `/volume1/RPI5-PXE/node1/rootfs` to `node1`

sudo mkdir -p /mnt/img
sudo mount -t nfs 192.168.101.253:/volume1/RPI5-PXE/node1/rootfs /mnt/img

11 [RPI node1] upload `/` root filesystem to [NAS] `/volume1/RPI5-PXE/node1/rootfs`

sudo rsync -a --exclude={"/proc/*","/sys/*","/dev/*","/run/*","/tmp/*","/mnt/*"} --info=progress2 / /mnt/img

12 [NAS] copy the content of `/volume1/RPI5-PXE/node1/rootfs/boot/firmware` into the `/volume1/RPI5-PXE/node1/`

cp -r /volume1/RPI5-PXE/node1/rootfs/boot/firmware/* /volume1/RPI5-PXE/node1/

13 [NAS] modify `cmdline.txt` in `/volume1/RPI5-PXE/node1/`

dwcotg.lpm_enable=0 console=serial0,115200 console=tty1 elevator=deadline rootwait rw root=/dev/nfs nfsroot=192.168.101.253:/volume1/RPi5-PXE/node1/rootfs,v3,tcp ip=dhcp

14 [NAS] modify `fstab` in `/volume1/RPI5-PXE/node1/rootfs/etc/`

proc            /proc           proc    defaults          0       0
192.168.101.253:/volume1/RPi5-PXE/node1 /boot nfs defaults,vers=3,proto=tcp 0 0

15 repeat steps from 10 - 14 for other nodes{2,3,4,..n}

16 final step

Remove micro SD card, power on PRi’s 🔌. DHCP server should navigate them to the bootfile location.

Run findmnt for validation

yuklia@node-1:~ $ findmnt
TARGET                         SOURCE      FSTYPE    OPTIONS
/                              192.168.101.253:/volume1/RPi5-PXE/node1/rootfs
                                           nfs       rw,relatime,vers=3,rsize=131072,wsize=131072,na
|-/sys                         sysfs       sysfs     rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/security       securityfs  securityf rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup             cgroup2     cgroup2   rw,nosuid,nodev,noexec,relatime,nsdelegate,memo
| |-/sys/fs/pstore             pstore      pstore    rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf                bpf         bpf       rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/kernel/debug          debugfs     debugfs   rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/tracing        tracefs     tracefs   rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/config         configfs    configfs  rw,nosuid,nodev,noexec,relatime
| `-/sys/fs/fuse/connections   fusectl     fusectl   rw,nosuid,nodev,noexec,relatime
|-/proc                        proc        proc      rw,relatime
| `-/proc/sys/fs/binfmt_misc   systemd-1   autofs    rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,m
|   `-/proc/sys/fs/binfmt_misc binfmt_misc binfmt_mi rw,nosuid,nodev,noexec,relatime
|-/dev                         udev        devtmpfs  rw,nosuid,relatime,size=3950816k,nr_inodes=2469
| |-/dev/pts                   devpts      devpts    rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmo
| |-/dev/shm                   tmpfs       tmpfs     rw,nosuid,nodev
| `-/dev/mqueue                mqueue      mqueue    rw,nosuid,nodev,noexec,relatime
|-/run                         tmpfs       tmpfs     rw,nosuid,nodev,noexec,relatime,size=824576k,mo
| |-/run/lock                  tmpfs       tmpfs     rw,nosuid,nodev,noexec,relatime,size=5120k
| |-/run/credentials/systemd-sysusers.service
| |                            ramfs       ramfs     ro,nosuid,nodev,noexec,relatime,mode=700
| |-/run/credentials/systemd-sysctl.service
| |                            ramfs       ramfs     ro,nosuid,nodev,noexec,relatime,mode=700
| |-/run/credentials/systemd-tmpfiles-setup-dev.service
| |                            ramfs       ramfs     ro,nosuid,nodev,noexec,relatime,mode=700
| |-/run/rpc_pipefs            sunrpc      rpc_pipef rw,relatime
| |-/run/credentials/systemd-tmpfiles-setup.service
| |                            ramfs       ramfs     ro,nosuid,nodev,noexec,relatime,mode=700
| `-/run/user/1000             tmpfs       tmpfs     rw,nosuid,nodev,relatime,size=824560k,nr_inodes
`-/boot                        192.168.101.253:/volume1/RPi5-PXE/node1
                                           nfs       rw,relatime,vers=3,rsize=131072,wsize=131072,na

This is how my home server rack looks like 🥰

Inner side of the server rack

summary

PXE boot up & running and it means that it is a time to host something in my cluster 🤔 As a next step I want to set up Kubernetes “the hard-way”. Stay tuned 😄

That is the end of the journey. :) I hope it might be handy for your own setup! If you have any questions, don’t hesitate to write in the comments or DM me on Linkedin. I am always passionate about discussing geeky stuff. 😉

introduction#

my appliances#

plan#

1 [UDM] create a separate subnet (e.g VLAN ) for RPi’s and NAS#

2 [UDM] configure DHCP server#

3 [SD card] Write OS#

4 [RPi] update EEPROM firmware for all RPi machines#

5 [NAS] enable NFS service#

6 [NAS] create SharedFolder#

7 [NAS] enable TFTP service#

8 [NAS] enable rsync#

9 [NAS] create folders node1/rootfs in TFTP root folder#

10 [RPi node1] mount NFS shared folder /volume1/RPI5-PXE/node1/rootfs to node1#

11 [RPI node1] upload / root filesystem to [NAS] /volume1/RPI5-PXE/node1/rootfs#

12 [NAS] copy the content of /volume1/RPI5-PXE/node1/rootfs/boot/firmware into the /volume1/RPI5-PXE/node1/#

13 [NAS] modify cmdline.txt in /volume1/RPI5-PXE/node1/#

14 [NAS] modify fstab in /volume1/RPI5-PXE/node1/rootfs/etc/#

15 repeat steps from 10 - 14 for other nodes{2,3,4,..n}#

16 final step#

summary#