Comments on: What HDD? http://dan.langille.org/2010/02/11/what-hdd/ He has another more popular diary. This one is more general. Fri, 18 May 2012 19:51:05 +0000 http://wordpress.org/?v=2.5 By: mike http://dan.langille.org/2010/02/11/what-hdd/#comment-406 mike Fri, 12 Feb 2010 11:35:04 +0000 http://dan.langille.org/?p=265#comment-406 I did a very similar setup to what you are planning - for years our backup server was a combination of gmirrored drives and hard links for snapshots and rsync instead of bacula. The problem with gmirror is the rebuilds after a crash/power issue plus the time to fsck a ufs partition - i found separating off each mirror as its own fs to be the simplest solution rather than have a set of gmirrors with gstripe on top. I then moved the base system over to just a gmirror on ufs with zfs for the main storage - 8 x 500gb drives - this worked reasonably well under 7.x series but rsync did occasionally bring the machine down but with no fsck waiting time it wasnt as much of a pain. Plus the storage pool of drives with zfs is a lot nicer to deal with than the geom layering - we also hit the odd problem of zfs snapshots not being available after the machine had been up for some time and needed a reboot in order to access them (usually the machine would then panic) - fortunately this was only once or twice a month at worst. There is also the added benifit of compressing the zfs fs which has allowed us to cram a lot more onto our aging disks than with ufs. Since moving to freebsd 8 we have not had any issues with rsync bring the machine down nor the snapshot issues. Recently we have deployed a similar setup (imap/file/intranet server not just a giant box for backup) but with a cf card to hold /boot and the rest on a zfs mirrored pool with a hot spare - a 4gb slice of each drive has also been setup as gmirrored swap so we should be able to remove any dead drives without any reboots. The flash memory card (/boot) is mounted read only so there is no need for an fsck should there be a powerloss. I am yet to look at replacing rsync with zfs send/receive as this should cut down on bandwidth for file moves and directory renames - plus the deduplication stuff on the horizon i really like the look of. Im must admit it does sound like im pushing zfs - my initial thought on it were not great - an fs needing a few gb of ram just to run happily didnt sit well however after a few years of using it now i plan to update all clients setups to the one described above - keeping a close eye on dfly and hammer though as the live mirroring and single system image stuff looks very cool - would be nice to have a layer for dealing with the raid stuff like geom and zpools and not rely on hardware raid - also it is currently missing compression which is a big drawback even with prices of drives so low. I did a very similar setup to what you are planning - for years our backup server was a combination of gmirrored drives and hard links for snapshots and rsync instead of bacula. The problem with gmirror is the rebuilds after a crash/power issue plus the time to fsck a ufs partition - i found separating off each mirror as its own fs to be the simplest solution rather than have a set of gmirrors with gstripe on top.

I then moved the base system over to just a gmirror on ufs with zfs for the main storage - 8 x 500gb drives - this worked reasonably well under 7.x series but rsync did occasionally bring the machine down but with no fsck waiting time it wasnt as much of a pain. Plus the storage pool of drives with zfs is a lot nicer to deal with than the geom layering - we also hit the odd problem of zfs snapshots not being available after the machine had been up for some time and needed a reboot in order to access them (usually the machine would then panic) - fortunately this was only once or twice a month at worst. There is also the added benifit of compressing the zfs fs which has allowed us to cram a lot more onto our aging disks than with ufs.

Since moving to freebsd 8 we have not had any issues with rsync bring the machine down nor the snapshot issues. Recently we have deployed a similar setup (imap/file/intranet server not just a giant box for backup) but with a cf card to hold /boot and the rest on a zfs mirrored pool with a hot spare - a 4gb slice of each drive has also been setup as gmirrored swap so we should be able to remove any dead drives without any reboots. The flash memory card (/boot) is mounted read only so there is no need for an fsck should there be a powerloss. I am yet to look at replacing rsync with zfs send/receive as this should cut down on bandwidth for file moves and directory renames - plus the deduplication stuff on the horizon i really like the look of.

Im must admit it does sound like im pushing zfs - my initial thought on it were not great - an fs needing a few gb of ram just to run happily didnt sit well however after a few years of using it now i plan to update all clients setups to the one described above - keeping a close eye on dfly and hammer though as the live mirroring and single system image stuff looks very cool - would be nice to have a layer for dealing with the raid stuff like geom and zpools and not rely on hardware raid - also it is currently missing compression which is a big drawback even with prices of drives so low.

]]>