FreeBSD : Glusterfs with SSL (via stunnel)
I have been working on parallel round-robin web clusters (is this the right term?) using 2 x FreeBSD 7.1 AMD64 boxes, nginx (patched with fair upstream), apache + php (backend), glusterfs, tinydns (sitting on another box, a name server, for round robin A record) and mysql multi-master replication. The setup is mainly making use of round-robin replication concept. Although I have yet fully hammered the configuration, it was pretty impressive and secure.
Glusterfs and mysql replicate with SSL. Nginx with SSL. These, however, are slightly at the expense of CPU and performance. I can live it that though.
The write-up of the setup is in progress as I am quite tied up with my day job, HeX project and glusterfs 2.0 ports for FreeBSD. Hopefully, I can manage the time well to complete all these. Nevertheless, here is partial (optional) write-up for glusterfs replication with SSL.
Note: server1 and server2 denote the FreeBSD clusters.
1) Installing required software
Most of the software except glusterfs (not in the freebsd ports as of this posting) is available via the FreeBSD ports. I’m aware of that TimurBakeyev is working on glusterfs ports.
# cd /usr/ports/security/stunnel && make install clean
2) Creating SSL certs (on either of the box)
Generally, it is easier to manage all certs/keys generation on a single box and duplicate required certs to the rest of the boxes. But YMMV. Commonly, cacert.pem and cert/key generated are copied.
2.1) For the impatient
Just create the certificate in 1 liner. Remember to modify the content of “-subj”.
# openssl req -new -outform PEM -out /etc/ssl/stunnel-cert.pem -newkey rsa:1024 \ -nodes -keyout /etc/ssl/private/stunnel-key.pem -keyform PEM -days 3650 -x509 -subj \ '/C=ur country code/ST=ur state/L=ur location/CN=ur server common name/O=ur org/OU=ur org unit'
2.2) For the patient
Creating necessary directories for ssl with the following commands.
# mkdir /etc/ssl/newcerts # mkdir /etc/ssl/private # echo '01' >/etc/ssl/serial # touch /etc/ssl/index.txt
Next, let’s generate a CA. You will be prompted with questions of your country, state, location etc and password for the CA key.
# openssl req -new -x509 -extensions v3_ca -keyout /etc/ssl/private/cakey.pem \ -out /etc/ssl/cacert.pem -days 3650 -config /etc/ssl/openssl.cnf
Generating a cert request for stunnel
# openssl req -outform PEM -out /etc/ssl/server-req.pem -newkey rsa:1024 -nodes \ -keyout /etc/ssl/private/stunnel-key.pem -keyform PEM -days 3650 -subj \ '/C=ur country code/ST=ur state/L=ur location/CN=ur server common name/O=ur org/OU=ur org unit'
Lastly using the CA key to sign the cert.
# openssl ca -in /etc/ssl/stunnel-req.pem -notext -out /etc/ssl/stunnel-cert.pem
3) Modifying stunnel rc.d for stunnel running client mode
The rc.d startup for stunnel is meant for running either server or client mode only. I need both modes here. Thus, a quick replication of stunnel rc.d to run another client mode instance of stunnel. I named it /usr/local/etc/rc.d/stunnelc.
#!/bin/sh
#
# $FreeBSD: ports/security/stunnel/files/stunnel.in,v 1.9 2008/01/26 14:18:12 roam Exp $
#
# PROVIDE: stunnelc
# REQUIRE: NETWORKING SERVERS
# BEFORE: DAEMON glusterfs
# KEYWORD: shutdown
#
# Add some of the following variables to /etc/rc.conf to configure stunnel:
# stunnelc_enable (bool): Set to "NO" by default.
# Set it to "YES" to enable stunnel.
# stunnelc_config (str): Default "/usr/local/etc/stunnel/stunnel-client.conf"
# Set it to the full path to the config file
# that stunnel will use during the automated
# start-up.
# stunnelc_pidfile (str): Default "/var/tmp/stunnel/stunnel-client.pid"
# Set it to the value of 'pid' in
# the stunnel.conf file.
#
. /etc/rc.subr
name="stunnelc"
rcvar=`set_rcvar`
load_rc_config $name
: ${stunnelc_enable="NO"}
: ${stunnelc_config="/usr/local/etc/stunnel/stunnel-client.conf"}
: ${stunnelc_pidfile="/var/tmp/stunnel/stunnel-client.pid"}
procname="/usr/local/bin/stunnel"
command="/usr/local/bin/stunnel"
command_args=${stunnelc_config}
pidfile=${stunnelc_pidfile}
required_files="${stunnelc_config}"
run_rc_command "$1"
4) glusterfs vol configuration
In this setup, glusterfsd is listening on lo0 127.0.0.1 port 6996 and stunnel server listening on em0 (net facing nic) port 8996. Stunnel client, on the other hand, is listening on 127.0.0.1 port 7996, forwarding to remote host on port 8996. Glusterfs client mount volume which is on 127.0.0.1 port 6996 and 7996 (which is tunneled to port 8996 of remote host). Refer to the configurations below:-
i) stunnel-server.conf.
[glusterfsd] accept = 8996 connect = 127.0.0.1:6996
ii) stunnel-client.conf.
[glusterfs] accept = 127.0.0.1:7996 connect = server2:8996
Auth login was used due to privileged port ceiling of 1024 imposed by auth addr method. Auth login method care less about privileged port ceiling.
Please refer to
http://www.gluster.org/docs/index.php/GlusterFS_Encrypted_network
http://www.gluster.org/docs/index.php/Translators_v2.0#auth.login
As I’m still working on glusterfs 2.0 ports, you can use the rc.d scripts that I have completed glusterfs and glusterfsd.
APPENDIX
Configuration files on server1
I) /etc/rc.conf
fusefs_enable="YES" glusterfsd_enable="YES" glusterfs_enable="YES" glusterfs_mount="/usr/home/www" stunnel_enable="YES" stunnel_config="/usr/local/etc/stunnel/stunnel-server.conf" stunnel_pidfile="/var/tmp/stunnel/stunnel-server.pid" stunnelc_enable="YES" stunnelc_config="/usr/local/etc/stunnel/stunnel-client.conf" stunnelc_pidfile="/var/tmp/stunnel/stunnel-client.pid"
II) Stunnel configuration for glusterfsd (/usr/local/etc/stunnel/stunnel-server.conf)
cert = /etc/ssl/stunnel-cert.pem key = /etc/ssl/private/stunnel-key.pem sslVersion = SSLv3 chroot = /var/tmp/stunnel setuid = stunnel setgid = stunnel ; PID is created inside chroot jail pid = /stunnel-server.pid socket = l:TCP_NODELAY=1 socket = r:TCP_NODELAY=1 CAfile = /etc/ssl/cacert.pem output = /var/log/stunnel.log [glusterfsd] accept = 8996 connect = 127.0.0.1:6996
III) Stunnel configuration for glusterfs (/usr/local/etc/stunnel/stunnel-client.conf)
cert = /etc/ssl/stunnel-cert.pem key = /etc/ssl/private/stunnel-key.pem sslVersion = SSLv3 chroot = /var/tmp/stunnel setuid = stunnel setgid = stunnel ; PID is created inside chroot jail pid = /stunnel-client.pid socket = l:TCP_NODELAY=1 socket = r:TCP_NODELAY=1 CAfile = /etc/ssl/cacert.pem output = /var/log/stunnelc.log client = yes [glusterfs] accept = 127.0.0.1:7996 connect = server2:8996
IV) Glusterfs client configuration (/usr/local/etc/glusterfs/glusterfs.vol)
volume remote1 type protocol/client option transport-type tcp option remote-host 127.0.0.1 option remote-port 6996 option remote-subvolume brick end-volume volume remote2 type protocol/client option transport-type tcp option remote-host 127.0.0.1 option remote-port 7996 option username yourusername option password yourpassword option remote-subvolume brick end-volume volume replicate type cluster/replicate subvolumes remote1 remote2 end-volume volume writebehind type performance/write-behind option block-size 128KB option cache-size 1MB subvolumes replicate end-volume volume cache type performance/io-cache option cache-size 512MB subvolumes writebehind end-volume
V) Glusterfs server configuration (/usr/local/etc/glusterfs/glusterfsd.vol)
volume posix type storage/posix option directory /usr/home/www-shared end-volume volume locks type features/locks subvolumes posix end-volume volume brick type performance/io-threads option thread-count 8 subvolumes locks end-volume volume server type protocol/server option transport-type tcp option transport.socket.bind-address 127.0.0.1 option auth.addr.brick.allow 127.0.0.1 option auth.login.brick.allow yourusername option auth.login.yourusername.password yourpassword subvolumes brick end-volume
