Archive for September 25th, 2009

FreeBSD Root on ZFS mirror using GPT

ZFS has been ported to FreeBSD by Pawel Jakub Dawidek for quite some time and it has been rather stable although some bleeding edges in complex setup. The current missing features are kernel CIFS server iSCSI. Nonetheless, it is powerful I decided to give my freebsd box a spin on the power file system.

1. THE SYSTEM
The box is running FreeBSD 8.0-RC1 amd64 with 2 SATA HDDs, ad4 and ad6. System is installed on ad4. The second hdd is intended for gmirror. Unfortunately, I did not do so.

This is initial GPT disk output.
# gpart show

=>       63  167772087  ad4  MBR  (80G)
         63  167766732    1  freebsd  [active]  (80G)
  167766795       5355       - free -  (2.6M)

=>        0  167766732  ad4s1  BSD  (80G)
          0    4194304      1  freebsd-ufs  (2.0G)
    4194304    4194304      2  freebsd-swap  (2.0G)
    8388608   83886080      4  freebsd-ufs  (40G)
   92274688    4194304      5  freebsd-ufs  (2.0G)
   96468992   71297740      6  freebsd-ufs  (34G)

Clear out ad6 (WARNING! This is destructive. All data in ad6 will be wiped out. Please make sure that you back up before proceed!)
# dd if=/dev/zero of=/dev/ad6 count=79
79+0 records in
79+0 records out
49448 bytes transferred in 0.122015 secs (331500 bytes/sec)

2. CREATE GPT DISK
First, gpt disk was created on ad6. In this setup, only 2 partitions, freebsd-boot and freebsd-zfs will be created as swap will be on ZFS volume. Please note that crash dumps can’t be created on the ZFS swap volume. If you need crash dumps, please consider creating additional freebsd-swap partition on gpt disk.

# gpart create -s gpt ad6
ad6 created

# gpart add -b 34 -s 128 -t freebsd-boot ad6
ad6p1 added

# gpart add -b 162 -s 167771965 -t freebsd-zfs ad6
ad6p2 added

Installing the Protected MBR (pmbr) and gptzfsboot loader
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad6
ad6 has bootcode

This is GPT disk output after gpt disk was created on ad6.
# gpart show

=>       63  167772087  ad4  MBR  (80G)
         63  167766732    1  freebsd  [active]  (80G)
  167766795       5355       - free -  (2.6M)

=>        0  167766732  ad4s1  BSD  (80G)
          0    4194304      1  freebsd-ufs  (2.0G)
    4194304    4194304      2  freebsd-swap  (2.0G)
    8388608   83886080      4  freebsd-ufs  (40G)
   92274688    4194304      5  freebsd-ufs  (2.0G)
   96468992   71297740      6  freebsd-ufs  (34G)

=>       34  167772093  ad6  GPT  (80G)
         34        128    1  freebsd-boot  (64K)
        162  167771965    2  freebsd-zfs  (80G)

3. CREATE ZFS POOL, ZSTORE (you can use any name for your ZFS pool)
Compression was used on this example. Please note that compression will cause some latency when accessing files on the ZFS filesystems. Use compression on ZFS filesystems which will not be accessed that often. Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compresion ratio. gzip is equivalent to gzip-6).

# zpool create zstore ad6p2
# zfs create zstore/usr
# zfs create -o compression=on zstore/usr/src
# zfs create -o compression=gzip zstore/usr/ports
# zfs create zstore/home
# zfs create zstore/var
# zfs create -V 2gb zstore/swap
# zfs set org.freebsd:swap=on zstore/swap
# zfs set checksum=off zstore/swap

2GB of ZFS swap volume was created without checksum.

Enabling ZFS and directing the system to mount root from zpool.

# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# echo 'vfs.root.mountfrom="zfs:zstore"' >> /boot/loader.conf
# echo 'LOADER_ZFS_SUPPORT=YES' >> /etc/src.conf

Adding LOADER_ZFS_SUPPORT=YES to /etc/src.conf is meant for later use when building zfs aware loader.

Let’s take a look at current disk layout after all the commands above. ZFS will automatically mount volumes after they are created.
# df -h

Filesystem          Size    Used   Avail Capacity  Mounted on
/dev/ad4s1a         1.9G    253M    1.5G    14%    /
devfs               1.0K    1.0K      0B   100%    /dev
/dev/ad4s1f          33G    4.0K     30G     0%    /home
/dev/ad4s1d          39G    661M     35G     2%    /usr
/dev/ad4s1e         1.9G    250K    1.8G     0%    /var
zstore               76G      0B     76G     0%    /zstore
zstore/usr           76G      0B     76G     0%    /zstore/usr
zstore/usr/src       76G      0B     76G     0%    /zstore/usr/src
zstore/usr/ports     76G      0B     76G     0%    /zstore/usr/ports
zstore/home          76G      0B     76G     0%    /zstore/home
zstore/var           76G      0B     76G     0%    /zstore/var

4. BUILD AND INSTALL ZFS AWARE LOADER
It is important that you did not miss out adding LOADER_ZFS_SUPPORT=YES to /etc/src.conf. If you missed that out, this part is useless and you will not be able to boot from ZFS.
# cd /usr/src/sys/boot/
# make obj && make depend && make
# cd i386/loader
# make install

5. REPLICATE CURRENT SYSTEM TO ZFS SYSTEM
# dump -L -0 -f- / | (cd /zstore ; restore -r -f-)
# dump -L -0 -f- /usr | (cd /zstore/usr; restore -r -f-)
# dump -L -0 -f- /home | (cd /zstore/home; restore -r -f-)
# dump -L -0 -f- /var | (cd /zstore/var; restore -r -f-)

I found this step not necessary. However, I did experience boot prompt failed to mount ZFS volume properly. Create it anyway.

# cat << EOF > /zstore/etc/fstab

# Device                Mountpoint      FStype  Options         Dump    Pass#
zstore                  /               zfs     rw              0       0
zstore/home             /home           zfs     rw              0       0
zstore/usr              /usr            zfs     rw              0       0
zstore/usr/src          /usr/src        zfs     rw              0       0
zstore/usr/ports        /usr/ports      zfs     rw              0       0
zstore/var              /var            zfs     rw              0       0
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0
EOF

6. CHANGE MOUNTING POINTS FOR ZFS POOL
# zfs set mountpoint=legacy zstore
# zfs set mountpoint=/usr zstore/usr
# zfs set mountpoint=/usr/src zstore/usr/src
# zfs set mountpoint=/usr/ports zstore/usr/ports
# zfs set mountpoint=/var zstore/var
# zfs set mountpoint=/home zstore/home
# zpool set bootfs=zstore zstore
# zfs umount -a

At this stage, it is almost done. Finger crossed and reboot the box. ;P
# reboot
Note : If the case of failure to boot ZFS root, you could just power down the box and reboot it. Press “6″ on the Boot menu to escape to loader prompt and set vfs.root.mountfrom=”ufs:ad4s1a” to boot back to your original setup without ZFS.

After reboot, the system is mounted on ZFS root. The output of df will show these.


# df -h

Filesystem          Size    Used   Avail Capacity  Mounted on
zstore               76G    270M     76G     0%    /
devfs               1.0K    1.0K      0B   100%    /dev
zstore/home          76G    6.9M     76G     0%    /home
zstore/usr           76G    166M     76G     0%    /usr
zstore/usr/src       76G    297M     76G     0%    /usr/src
zstore/usr/ports     76G      0B     76G     0%    /usr/ports
zstore/var           76G    640K     76G     0%    /var


7. PREPARE AD4 FOR GPT DISK
Clear out disk layout for ad4 with dd.
# dd if=/dev/zero of=/dev/ad4 count=79
79+0 records in
79+0 records out
49448 bytes transferred in 0.122015 secs (331500 bytes/sec)

Steps below are similar to previous steps in GPT disk on ad6.
# gpart create -s gpt ad4
ad4 created

# gpart add -b 34 -s 128 -t freebsd-boot ad4
ad4p1 added

# gpart add -b 162 -s 167771965 -t freebsd-zfs ad4
ad4p2 added

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad4
ad4 has bootcode

# gpart show

=>       34  167772093  ad6  GPT  (80G)
         34        128    1  freebsd-boot  (64K)
        162  167771965    2  freebsd-zfs  (80G)

=>       34  167772093  ad4  GPT  (80G)
         34        128    1  freebsd-boot  (64K)
        162  167771965    2  freebsd-zfs  (80G)

8. ADDING AD4P2 TO ZSTORE AS MIRROR
Attach ad4p2 to zstore and wait for it to be resilvered. zpool status is handy for checking the status.
# zpool attach zstore ad6p2 ad4p2
# zpool status

  pool: zstore
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 58.66% done, 0h0m to go
config:

	NAME        STATE     READ WRITE CKSUM
	zstore      ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    ad6p2   ONLINE       0     0     0  2.24M resilvered
	    ad4p2   ONLINE       0     0     0  434M resilvered

errors: No known data errors

# df -h

Filesystem          Size    Used   Avail Capacity  Mounted on
zstore               76G    270M     76G     0%    /
devfs               1.0K    1.0K      0B   100%    /dev
zstore/home          76G    6.9M     76G     0%    /home
zstore/usr           76G    166M     76G     0%    /usr
zstore/usr/src       76G    297M     76G     0%    /usr/src
zstore/usr/ports     76G      0B     76G     0%    /usr/ports
zstore/var           76G    640K     76G     0%    /var

There you have it. Converted your existing FreeBSD box to ZFS root with mirroring.

9. REFERENCES
1. http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

2. http://yds.coolrat.org/zfsboot.shtml

3. GPT(8) manpage

Friday, September 25th, 2009

FreeBSD, Howto No Comments