Overview
- Enable USB boot mode on the Pi (enables booting via network)
- Setup dnsmasq on the server (provides /boot)
- Setup NFS on the server (privides /)
Enable USB boot mode
To start, we need to enable USB boot mode in your Pi 3’s firmware. I’m assuming you’re working in a terminal on your Pi running some version of Raspbian. You should be working as root or using sudo
to perform these tasks.
Issue the following command to append program_usb_boot_mode=1
to your /boot/config.txt file:
echo program_usb_boot_mode=1 | sudo tee -a /boot/config.txt
Can check /boot/config.txt to make sure it actually got appended with your favorite text editor, cat
, less
, whatever. Once you’ve verified that line is present in /boot/config.txt, reboot the Pi.
shutdown -r now
Once it comes back up, login, and issue:
vcgencmd otp_dump | grep 17:
Assuming what we did worked, you should see:
17:3020000a
If you get this result, you have successfully enabled USB boot mode on your Pi 3. If you don’t, double check that what we wanted to append to /boot/config.txt actually got appended, and reboot again.
This doesn’t break SD card booting functionality. It just means that you can also boot from USB and ethernet now.
More information can be found here.
You wont be needing that SD card in the Pi anymore, but we’ll be using the contents of /boot and / on the SD card in the setup of dnsmasq and NFS. Attach the SD card you were using in the Pi to your server. On the server, let’s make a couple of directories to mount SD card partitions on:
mkdir /mnt/raspbian-boot
mkdir /mnt/raspbian-root
Depending on how you attached the SD card to your server, the exact path to the SD card partitions will vary. Mount the first one (the smaller one) on /mnt/raspbian-boot and the second one (the larger one) on /mnt/raspbian-root. NOTE: You’ll need to change these paths to suit how your machine sees the SD card when you attach it.
mount /path/to/sdcard/boot-partition /mnt/raspbian-boot
mount /path/to/sdcard/root-partition /mnt/raspbian-root
We’ll be copying the contents of /mnt/raspbian-boot and /mnt/raspbian-root later on.
Setup dnsmasq
dnsmasq
can do a number for things. The man
pages describe dnsmasq
as “A lightweight DHCP and caching DNS server.” As it turns out, we aren’t going to be using dnsmasq
for either of these. Instead, dnsmasq
is going to serve your Pi the /boot part of the filesystem. When your Pi starts up, having since enabled USB boot mode, it will broadcast bootp requests over the network its attached to. Once our dnsmasq
service on our server is up and listening, dnsmasq
will get these requests, and respond by sending files needed by the Pi to bootstrap.
Start by installing the required packages not on the Pi, but on the server/computer/toaster the Pi will be booting off of:
apt install dnsmasq tcpdump
We should test to make sure the server is hearing the bootp requests the Pi is sending out before going any further. You can check this by issuing:
tcpdump -i <insert network interface here> port bootpc
Powercycle the Pi, wait a few seconds, and you should start seeing things pop up after that command. If you do see something, great! We have connectivity! If not, double check that both the Pi and your server are really on the same network. Exit with Ctrl-c and we can continue.
Make a directory for your Pi’s /boot files to live in. Let’s call it /nfs/client1/boot:
mkdir -p /nfs/client1/boot
Copy the contents of /mnt/raspbian-boot to /nfs/client1/boot:
rsync -xa --progress /mnt/raspbian-boot/* /nfs/client1/boot
Just for testing, open up the permissions of /nfs/client1/boot to make sure dnsmasq can DEFINITELY access those files and send them out:
chmod -R 777 /nfs/client1/boot
Move your current /etc/dnsmasq.conf out of the way:
mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
This is my entire /etc/dnsmasq.conf on the machine (KDE Neon, but its Ubuntu, whatever) where dnsmasq
is running:
#port=0
dhcp-range=192.168.1.255,proxy
log-dhcp
enable-tftp
tftp-root=/nfs/client1/boot
pxe-service=0,"Raspberry Pi Boot "
Make a new /etc/dnsmasq.conf in the editor of your choice (as root) and copy/paste my configuration into it.
You may need to change the 192.168.1.255
part to suit your particular LAN. The IP of my server is 192.168.1.106 and has a subnet of /24 (I usually check mine with ip a
). If your subnet is /24, which it usually is in residential LAN environments, the IP listed there will be first.three.octets.255. The 192.168.1 part may change depending on how things are in your home/apartment/office. I’m assuming that if you’ve made it this far you can figure out what your IP and subnet are. Just in case.
Regarding the #port=0
part: This part (basically) determines whether or not dnsmasq is doing DNS things. I should have been allowed to disable this (port=0), but in my case, if I want to have DNS resolution working on the server, I did need to leave it commented out. I leave this part here for you to test. I got everything working, Pi 3 booting, great. Then I went to do something on the server that involved internet and couldn’t because DNS wasn’t working. Commented that part out, restarted dnsmasq, and voila, DNS works again.
Once your /etc/dnsmasq.conf is ready to go, restart dnsmasq
. On Ubuntu I use:
systemctl enable dnsmasq
systemctl restart dnsmasq
Now we can reboot the Pi 3 and see if stuff actually gets sent back to it. It won’t boot, because it still needs the rest of the filesystem to do that, but dnsmasq
should still pass at least SOMETHING to it. Monitor with
tail -f /var/log/syslog | grep dnsmasq
You should see a whole bunch of stuff fly by indicating files being sent and not found and a whole bunch of other stuff.
Ctrl-c to exit and we can continue.
On my particular network, even if my server gets bootp requests from the Pi, the Pi doesnt always get stuff from The errors I was getting are attributed to out of date code burned into the Pi 3. Usage of the newer bootcode.bin that comes with current versions of Raspbian resolve this issue. More information about this fix can be found at the end of this guide.dnsmasq
. I’m not sure why, but sometimes, the Pi will turn on, send out its requests, the server will see them, but stuff won’t get sent back. Unplug the power, plug back in, and then it’ll work. I suspect this is an issue with network equipment more than with dnsmasq
configuration. See here.
dnsmasq
takes care of /boot, now we need to setup NFS, which takes care of everything else.
Setup NFS
On the server, install packages:
apt-get install nfs-kernel-server
Add an entry to /etc/exports:
echo "/nfs/client1 *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
Copy the contents of the second SD card partition into our NFS export directory /nfs/client1:
rsync -xa --progress /mnt/raspbian-root/* /nfs/client1/
Enable and (re)start services so our changes take effect:
systemctl enable rpcbind
systemctl enable nfs-kernel-server
systemctl start rpcbind
systemctl start nfs-kernel-server
Make sure NFS is doing its thing using showmount
:
showmount -e localhost
You should get something back like:
$ showmount -e localhost
Export list for localhost:
/nfs/client1 *
We need to adjust /nfs/client1/boot/cmdline.txt to use this NFS export. This is what my /nfs/client1/boot/cmdline.txt looks like:
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.1.106:/nfs/client1,vers=3 rw ip=dhcp rootwait elevator=deadline fsck.repair=yes rootwait
You can see that everything is the same except for the root= part. Adjust 192.168.1.106 to the IP of your server.
Adjust /nfs/client1/etc/fstab to only include the proc line (delete or comment out the others):
proc /proc proc defaults 0 0
You’re ready to test!
Consideration: /boot on SD card might still be a good idea bootcode.bin in the FAT32 partition is all you need, put /boot in the NFS share
I have two reasons for continuing to use an SD card: inconsistent boots over the network, and updates touching files in /boot.
If I knew that 100% of the time I could boot via (my) network, I might still be cardless. This is a minor annoyance as I don’t reboot very often, but the degree of inconsistency I experienced would be a thorn in my side. Updates to how the Pi establishes a tftp session have been made in bootcode.bin. What I’ve done in my particular setup, is leave the SD card in the Pi, with only that file in the FAT32 partition (the /boot partition). The Pi searches that partition for that file on boot regardless of any other configuration file. At that point of the boot process, the Pi isn’t even considering mountpoints. It looks for a partition of that type (FAT32) and tries to find that file. If it does, it uses that file to do things instead of whats burned into the chip when it left the factory. 100% boot success over tftp using only that file in that partition.
Updates touching files in /boot is more of a concern. I don’t want to have to think about whether a package install or an update is going to mess with something in that partition before I run Thanks to the recommendations of some fine people on Reddit and #raspberrypi on Freenode, the separate /tftpboot and /nfs/client1 directories have been combined. apt
. Figuring out a way to give the Pi access to the contents of /tftpboot is definitely on my list of to-do’s for this guide. So, while I still have dnsmasq
running, I’m not really taking advantage of it. Pi boots (reliably) off of /boot on the SD card, and then proceeds to pull everything over NFS.dnsmasq
gets its things from /nfs/client1/boot, NFS remains unchanged. This allows updates and installations to update the contents of /boot on the server just as it would if it were still on the SD card.
This could also be streamlined a bit with regards to the initial mounts. Mount the / SD card patition on /mnt, and mount the /boot SD card partition on /mnt/boot. Then, rsync everything into /nfs/client1.
Credits
HOW TO BOOT FROM A USB MASS STORAGE DEVICE ON A RASPBERRY PI 3
Network boot a bunch of Raspberry Pi 3
NETWORK BOOT YOUR RASPBERRY PI
2021 update, just use this: https://www.raspberrypi.com/documentation/computers/remote-access.html#network-boot-your-raspberry-pi
Leave a Reply