I have been working on parallel round-robin web clusters (is this the right term?) using 2 x FreeBSD 7.1 AMD64 boxes, nginx (patched with fair upstream), apache + php (backend), glusterfs, tinydns (sitting on another box, a name server, for round robin A record) and mysql multi-master replication. The setup is mainly making use of round-robin replication concept. Although I have yet fully hammered the configuration, it was pretty impressive and secure.
Glusterfs and mysql replicate with SSL. Nginx with SSL. These, however, are slightly at the expense of CPU and performance. I can live it that though.
The write-up of the setup is in progress as I am quite tied up with my day job, HeX project and glusterfs 2.0 ports for FreeBSD. Hopefully, I can manage the time well to complete all these. Nevertheless, here is partial (optional) write-up for glusterfs replication with SSL.
Note: server1 and server2 denote the FreeBSD clusters.
1) Installing required software
Most of the software except glusterfs (not in the freebsd ports as of this posting) is available via the FreeBSD ports. I’m aware of that TimurBakeyev is working on glusterfs ports.
# cd /usr/ports/security/stunnel && make install clean
2) Creating SSL certs (on either of the box)
Generally, it is easier to manage all certs/keys generation on a single box and duplicate required certs to the rest of the boxes. But YMMV. Commonly, cacert.pem and cert/key generated are copied.
2.1) For the impatient
Just create the certificate in 1 liner. Remember to modify the content of “-subj”.
# openssl req -new -outform PEM -out /etc/ssl/stunnel-cert.pem -newkey rsa:1024 \
-nodes -keyout /etc/ssl/private/stunnel-key.pem -keyform PEM -days 3650 -x509 -subj \
'/C=ur country code/ST=ur state/L=ur location/CN=ur server common name/O=ur org/OU=ur org unit'
2.2) For the patient
Creating necessary directories for ssl with the following commands.
# mkdir /etc/ssl/newcerts
# mkdir /etc/ssl/private
# echo '01' >/etc/ssl/serial
# touch /etc/ssl/index.txt
Next, let’s generate a CA. You will be prompted with questions of your country, state, location etc and password for the CA key.
# openssl req -new -x509 -extensions v3_ca -keyout /etc/ssl/private/cakey.pem \
-out /etc/ssl/cacert.pem -days 3650 -config /etc/ssl/openssl.cnf
Generating a cert request for stunnel
# openssl req -outform PEM -out /etc/ssl/server-req.pem -newkey rsa:1024 -nodes \
-keyout /etc/ssl/private/stunnel-key.pem -keyform PEM -days 3650 -subj \
'/C=ur country code/ST=ur state/L=ur location/CN=ur server common name/O=ur org/OU=ur org unit'
Lastly using the CA key to sign the cert.
# openssl ca -in /etc/ssl/stunnel-req.pem -notext -out /etc/ssl/stunnel-cert.pem
3) Modifying stunnel rc.d for stunnel running client mode
The rc.d startup for stunnel is meant for running either server or client mode only. I need both modes here. Thus, a quick replication of stunnel rc.d to run another client mode instance of stunnel. I named it /usr/local/etc/rc.d/stunnelc.
#!/bin/sh
#
# $FreeBSD: ports/security/stunnel/files/stunnel.in,v 1.9 2008/01/26 14:18:12 roam Exp $
#
# PROVIDE: stunnelc
# REQUIRE: NETWORKING SERVERS
# BEFORE: DAEMON glusterfs
# KEYWORD: shutdown
#
# Add some of the following variables to /etc/rc.conf to configure stunnel:
# stunnelc_enable (bool): Set to "NO" by default.
# Set it to "YES" to enable stunnel.
# stunnelc_config (str): Default "/usr/local/etc/stunnel/stunnel-client.conf"
# Set it to the full path to the config file
# that stunnel will use during the automated
# start-up.
# stunnelc_pidfile (str): Default "/var/tmp/stunnel/stunnel-client.pid"
# Set it to the value of 'pid' in
# the stunnel.conf file.
#
. /etc/rc.subr
name="stunnelc"
rcvar=`set_rcvar`
load_rc_config $name
: ${stunnelc_enable="NO"}
: ${stunnelc_config="/usr/local/etc/stunnel/stunnel-client.conf"}
: ${stunnelc_pidfile="/var/tmp/stunnel/stunnel-client.pid"}
procname="/usr/local/bin/stunnel"
command="/usr/local/bin/stunnel"
command_args=${stunnelc_config}
pidfile=${stunnelc_pidfile}
required_files="${stunnelc_config}"
run_rc_command "$1"
4) glusterfs vol configuration
In this setup, glusterfsd is listening on lo0 127.0.0.1 port 6996 and stunnel server listening on em0 (net facing nic) port 8996. Stunnel client, on the other hand, is listening on 127.0.0.1 port 7996, forwarding to remote host on port 8996. Glusterfs client mount volume which is on 127.0.0.1 port 6996 and 7996 (which is tunneled to port 8996 of remote host). Refer to the configurations below:-
i) stunnel-server.conf.
[glusterfsd]
accept = 8996
connect = 127.0.0.1:6996
ii) stunnel-client.conf.
[glusterfs]
accept = 127.0.0.1:7996
connect = server2:8996
Auth login was used due to privileged port ceiling of 1024 imposed by auth addr method. Auth login method care less about privileged port ceiling.
Please refer to
http://www.gluster.org/docs/index.php/GlusterFS_Encrypted_network
http://www.gluster.org/docs/index.php/Translators_v2.0#auth.login
As I’m still working on glusterfs 2.0 ports, you can use the rc.d scripts that I have completed glusterfs and glusterfsd.
APPENDIX
Configuration files on server1
I) /etc/rc.conf
fusefs_enable="YES"
glusterfsd_enable="YES"
glusterfs_enable="YES"
glusterfs_mount="/usr/home/www"
stunnel_enable="YES"
stunnel_config="/usr/local/etc/stunnel/stunnel-server.conf"
stunnel_pidfile="/var/tmp/stunnel/stunnel-server.pid"
stunnelc_enable="YES"
stunnelc_config="/usr/local/etc/stunnel/stunnel-client.conf"
stunnelc_pidfile="/var/tmp/stunnel/stunnel-client.pid"
II) Stunnel configuration for glusterfsd (/usr/local/etc/stunnel/stunnel-server.conf)
cert = /etc/ssl/stunnel-cert.pem
key = /etc/ssl/private/stunnel-key.pem
sslVersion = SSLv3
chroot = /var/tmp/stunnel
setuid = stunnel
setgid = stunnel
; PID is created inside chroot jail
pid = /stunnel-server.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
CAfile = /etc/ssl/cacert.pem
output = /var/log/stunnel.log
[glusterfsd]
accept = 8996
connect = 127.0.0.1:6996
III) Stunnel configuration for glusterfs (/usr/local/etc/stunnel/stunnel-client.conf)
cert = /etc/ssl/stunnel-cert.pem
key = /etc/ssl/private/stunnel-key.pem
sslVersion = SSLv3
chroot = /var/tmp/stunnel
setuid = stunnel
setgid = stunnel
; PID is created inside chroot jail
pid = /stunnel-client.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
CAfile = /etc/ssl/cacert.pem
output = /var/log/stunnelc.log
client = yes
[glusterfs]
accept = 127.0.0.1:7996
connect = server2:8996
IV) Glusterfs client configuration (/usr/local/etc/glusterfs/glusterfs.vol)
volume remote1
type protocol/client
option transport-type tcp
option remote-host 127.0.0.1
option remote-port 6996
option remote-subvolume brick
end-volume
volume remote2
type protocol/client
option transport-type tcp
option remote-host 127.0.0.1
option remote-port 7996
option username yourusername
option password yourpassword
option remote-subvolume brick
end-volume
volume replicate
type cluster/replicate
subvolumes remote1 remote2
end-volume
volume writebehind
type performance/write-behind
option block-size 128KB
option cache-size 1MB
subvolumes replicate
end-volume
volume cache
type performance/io-cache
option cache-size 512MB
subvolumes writebehind
end-volume
V) Glusterfs server configuration (/usr/local/etc/glusterfs/glusterfsd.vol)
volume posix
type storage/posix
option directory /usr/home/www-shared
end-volume
volume locks
type features/locks
subvolumes posix
end-volume
volume brick
type performance/io-threads
option thread-count 8
subvolumes locks
end-volume
volume server
type protocol/server
option transport-type tcp
option transport.socket.bind-address 127.0.0.1
option auth.addr.brick.allow 127.0.0.1
option auth.login.brick.allow yourusername
option auth.login.yourusername.password yourpassword
subvolumes brick
end-volume
