FreeBSD Root on ZFS mirror using GPT
ZFS has been ported to FreeBSD by Pawel Jakub Dawidek for quite some time and it has been rather stable although some bleeding edges in complex setup. The current missing features are kernel CIFS server iSCSI. Nonetheless, it is powerful I decided to give my freebsd box a spin on the power file system.
1. THE SYSTEM
The box is running FreeBSD 8.0-RC1 amd64 with 2 SATA HDDs, ad4 and ad6. System is installed on ad4. The second hdd is intended for gmirror. Unfortunately, I did not do so.
This is initial GPT disk output.
# gpart show
=> 63 167772087 ad4 MBR (80G)
63 167766732 1 freebsd [active] (80G)
167766795 5355 - free - (2.6M)
=> 0 167766732 ad4s1 BSD (80G)
0 4194304 1 freebsd-ufs (2.0G)
4194304 4194304 2 freebsd-swap (2.0G)
8388608 83886080 4 freebsd-ufs (40G)
92274688 4194304 5 freebsd-ufs (2.0G)
96468992 71297740 6 freebsd-ufs (34G)
Clear out ad6 (WARNING! This is destructive. All data in ad6 will be wiped out. Please make sure that you back up before proceed!)
# dd if=/dev/zero of=/dev/ad6 count=79
79+0 records in
79+0 records out
49448 bytes transferred in 0.122015 secs (331500 bytes/sec)
2. CREATE GPT DISK
First, gpt disk was created on ad6. In this setup, only 2 partitions, freebsd-boot and freebsd-zfs will be created as swap will be on ZFS volume. Please note that crash dumps can’t be created on the ZFS swap volume. If you need crash dumps, please consider creating additional freebsd-swap partition on gpt disk.
# gpart create -s gpt ad6
ad6 created
# gpart add -b 34 -s 128 -t freebsd-boot ad6
ad6p1 added
# gpart add -b 162 -s 167771965 -t freebsd-zfs ad6
ad6p2 added
Installing the Protected MBR (pmbr) and gptzfsboot loader
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad6
ad6 has bootcode
This is GPT disk output after gpt disk was created on ad6.
# gpart show
=> 63 167772087 ad4 MBR (80G)
63 167766732 1 freebsd [active] (80G)
167766795 5355 - free - (2.6M)
=> 0 167766732 ad4s1 BSD (80G)
0 4194304 1 freebsd-ufs (2.0G)
4194304 4194304 2 freebsd-swap (2.0G)
8388608 83886080 4 freebsd-ufs (40G)
92274688 4194304 5 freebsd-ufs (2.0G)
96468992 71297740 6 freebsd-ufs (34G)
=> 34 167772093 ad6 GPT (80G)
34 128 1 freebsd-boot (64K)
162 167771965 2 freebsd-zfs (80G)
3. CREATE ZFS POOL, ZSTORE (you can use any name for your ZFS pool)
Compression was used on this example. Please note that compression will cause some latency when accessing files on the ZFS filesystems. Use compression on ZFS filesystems which will not be accessed that often. Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compresion ratio. gzip is equivalent to gzip-6).
# zpool create zstore ad6p2
# zfs create zstore/usr
# zfs create -o compression=on zstore/usr/src
# zfs create -o compression=gzip zstore/usr/ports
# zfs create zstore/home
# zfs create zstore/var
# zfs create -V 2gb zstore/swap
# zfs set org.freebsd:swap=on zstore/swap
# zfs set checksum=off zstore/swap
2GB of ZFS swap volume was created without checksum.
Enabling ZFS and directing the system to mount root from zpool.
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# echo 'vfs.root.mountfrom="zfs:zstore"' >> /boot/loader.conf
# echo 'LOADER_ZFS_SUPPORT=YES' >> /etc/src.conf
Adding LOADER_ZFS_SUPPORT=YES to /etc/src.conf is meant for later use when building zfs aware loader.
Let’s take a look at current disk layout after all the commands above. ZFS will automatically mount volumes after they are created.
# df -h
Filesystem Size Used Avail Capacity Mounted on /dev/ad4s1a 1.9G 253M 1.5G 14% / devfs 1.0K 1.0K 0B 100% /dev /dev/ad4s1f 33G 4.0K 30G 0% /home /dev/ad4s1d 39G 661M 35G 2% /usr /dev/ad4s1e 1.9G 250K 1.8G 0% /var zstore 76G 0B 76G 0% /zstore zstore/usr 76G 0B 76G 0% /zstore/usr zstore/usr/src 76G 0B 76G 0% /zstore/usr/src zstore/usr/ports 76G 0B 76G 0% /zstore/usr/ports zstore/home 76G 0B 76G 0% /zstore/home zstore/var 76G 0B 76G 0% /zstore/var
4. BUILD AND INSTALL ZFS AWARE LOADER
It is important that you did not miss out adding LOADER_ZFS_SUPPORT=YES to /etc/src.conf. If you missed that out, this part is useless and you will not be able to boot from ZFS.
# cd /usr/src/sys/boot/
# make obj && make depend && make
# cd i386/loader
# make install
5. REPLICATE CURRENT SYSTEM TO ZFS SYSTEM
# dump -L -0 -f- / | (cd /zstore ; restore -r -f-)
# dump -L -0 -f- /usr | (cd /zstore/usr; restore -r -f-)
# dump -L -0 -f- /home | (cd /zstore/home; restore -r -f-)
# dump -L -0 -f- /var | (cd /zstore/var; restore -r -f-)
I found this step not necessary. However, I did experience boot prompt failed to mount ZFS volume properly. Create it anyway.
# cat << EOF > /zstore/etc/fstab
# Device Mountpoint FStype Options Dump Pass# zstore / zfs rw 0 0 zstore/home /home zfs rw 0 0 zstore/usr /usr zfs rw 0 0 zstore/usr/src /usr/src zfs rw 0 0 zstore/usr/ports /usr/ports zfs rw 0 0 zstore/var /var zfs rw 0 0 /dev/acd0 /cdrom cd9660 ro,noauto 0 0 EOF
6. CHANGE MOUNTING POINTS FOR ZFS POOL
# zfs set mountpoint=legacy zstore
# zfs set mountpoint=/usr zstore/usr
# zfs set mountpoint=/usr/src zstore/usr/src
# zfs set mountpoint=/usr/ports zstore/usr/ports
# zfs set mountpoint=/var zstore/var
# zfs set mountpoint=/home zstore/home
# zpool set bootfs=zstore zstore
# zfs umount -a
At this stage, it is almost done. Finger crossed and reboot the box. ;P
# reboot
Note : If the case of failure to boot ZFS root, you could just power down the box and reboot it. Press “6″ on the Boot menu to escape to loader prompt and set vfs.root.mountfrom=”ufs:ad4s1a” to boot back to your original setup without ZFS.
After reboot, the system is mounted on ZFS root. The output of df will show these.
# df -h
Filesystem Size Used Avail Capacity Mounted on zstore 76G 270M 76G 0% / devfs 1.0K 1.0K 0B 100% /dev zstore/home 76G 6.9M 76G 0% /home zstore/usr 76G 166M 76G 0% /usr zstore/usr/src 76G 297M 76G 0% /usr/src zstore/usr/ports 76G 0B 76G 0% /usr/ports zstore/var 76G 640K 76G 0% /var
7. PREPARE AD4 FOR GPT DISK
Clear out disk layout for ad4 with dd.
# dd if=/dev/zero of=/dev/ad4 count=79
79+0 records in
79+0 records out
49448 bytes transferred in 0.122015 secs (331500 bytes/sec)
Steps below are similar to previous steps in GPT disk on ad6.
# gpart create -s gpt ad4
ad4 created
# gpart add -b 34 -s 128 -t freebsd-boot ad4
ad4p1 added
# gpart add -b 162 -s 167771965 -t freebsd-zfs ad4
ad4p2 added
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad4
ad4 has bootcode
# gpart show
=> 34 167772093 ad6 GPT (80G)
34 128 1 freebsd-boot (64K)
162 167771965 2 freebsd-zfs (80G)
=> 34 167772093 ad4 GPT (80G)
34 128 1 freebsd-boot (64K)
162 167771965 2 freebsd-zfs (80G)
8. ADDING AD4P2 TO ZSTORE AS MIRROR
Attach ad4p2 to zstore and wait for it to be resilvered. zpool status is handy for checking the status.
# zpool attach zstore ad6p2 ad4p2
# zpool status
pool: zstore state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h0m, 58.66% done, 0h0m to go config: NAME STATE READ WRITE CKSUM zstore ONLINE 0 0 0 mirror ONLINE 0 0 0 ad6p2 ONLINE 0 0 0 2.24M resilvered ad4p2 ONLINE 0 0 0 434M resilvered errors: No known data errors
# df -h
Filesystem Size Used Avail Capacity Mounted on zstore 76G 270M 76G 0% / devfs 1.0K 1.0K 0B 100% /dev zstore/home 76G 6.9M 76G 0% /home zstore/usr 76G 166M 76G 0% /usr zstore/usr/src 76G 297M 76G 0% /usr/src zstore/usr/ports 76G 0B 76G 0% /usr/ports zstore/var 76G 640K 76G 0% /var
There you have it. Converted your existing FreeBSD box to ZFS root with mirroring.
9. REFERENCES
1. http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
