The house heavily relies on the network file sharing protocol NFS. A folder is shared from one computer, then “mounted” to an empty folder on another computer, and that other computer sees the folder as if it is local to itself. Folders are mounted all over on computers and servers all over the house. To figure out what is currently configured as a share on any computer, look at the contents of /etc/fstab, that will show you what remote folders are mounted locally on reboot.

There are 2 main things to think about with NFS:

1. Is the mount point empty? If we want to make the “Pictures” folder on the server act like it's a user's “Pictures” folder, and the user has pictures in their local folder, they won't be able to see the local contents while the remote folder is mounted; the remote mounted one sits right on top of the local one. They are still there, and would be visible after the remote folder is unmounted.
2. Files and folders are owned by users and groups, identified by numerical id's. If the id for a user or group is not the same on 2 computers, file access gets weird. Much time has been taken to sync user ids and group ids across all computers and servers.

The primary bulk storage of the house is the Network Attached Storage (NAS) in the server closet named sr66-nas-2. It is a TrueNAS Scale system with 12 drives totaling roughly 43TB of storage. The storage is a filesystem called ZFS which is important for several reasons:

1. It is fault tolerant; it can survive a failed hard drive without data loss, and can detect and repair “bit rot” (1s and 0s randomly flipping due to just mechanical age).
2. It is copy-on-write with snapshots. We are currently using several tiers of snapshots. In one, every 30 minutes, a “snapshot” of selected storage is taken, and changes written over the next 30 minutes are stored separately and these snapshots persist for 14 days. This means, if we accidentally deleted a file, or overwrote something with a bad save, we could go back to a previous point in time over the last 14 days and retrieve the previous version. Other snapshot schedules for different groups of storage are things like “once a day for 6 months.”

The NAS storage sr66-nas-2 is presented to other systems via NFS and iSCSI. At this time, all Proxmox VM's have the OS drive and most other storage hosted directly on the NAS via iSCSI (with the exception of the primary storage of the file server, located locally on the Proxmox Hosts). Most secondary bulk file storage is intended to be mounted on the main file server sr66-file-01 via iSCSI and then mounted to other clients via NFS from there, however some specialized storage is mounted to other devices as needed (for example, there is a Minecraft Backup directory where Minecraft clients replicate their config every 30 minutes via NFS).

The file server sr66-file-01 becomes the focal point for all data. The primary repository of “important files” is a LUN hosted on the Proxmox hosts, on ZFS storage, replicated between all hosts. Surfrock66's home directory is the focal point for storage on the server (services like Nextcloud and Jellyfin look to locations within there for their main storage). All additional storage is mounted to the fileserver (into surfrock66's home folder) as additional iSCSI LUNs from the NAS (Videos, Books, Projects, etc), but secondary copies of important files (Documents, Pictures) exist as NFS shares on the NAS. There is a daily replication via cronjob of these important files from the file server to the NAS so that files exist ZFS protected on each Proxmox host, then also ZFS protected on the NAS with 6 months of snapshots to be able to retrieve files.

For backups, the bulk of data backup goes to Crashplan. We pay a license for a backup of unlimited size for 1 computer. Because of how NFS works, Crashplan is completely happy to consider NFS mounted directories as part of a single computer. For that reason, sr66-vdi-01 is the computer we have chosen to back up, primarily because it has a desktop environment and Crashplan GUI tools are far superior to any CLI-only option. Most shared data directories are actually housed on sr66-file-01 which is ZFS storage on a proxmox host, but are mounted on various computers at various mount points via NFS.

Additionally, a routine exists to copy selected directories to a small external hard drive, which resides in the locked compartment of the fireproof safe. This is done roughly quarterly.

As an example of file integrity defense, let's look at the “Documents” directory:

• /home/surfrock66/Documents is NFS storage on sr66-vdi-01 mounted from sr66-file-01 which is a RaidZ2 LUN on a proxmox host (but replicated across all hosts every 2 hours). This should protect against disk failure, full server hardware failure, and bit rot.
• Daily, the contents of /home/surfrock66/Documents are replicated 1-way to /mnt/sr66-nas/Documents which is hosted on sr66-nas-2. This should protect against the demise of data should all Proxmox Hosts, sr66-vdi-01, or sr66-file-01 fail.
• The contents of that replicated directory are snapshotted daily, and these snapshots persist for 6 months. This protects against bit-rot and unintentional destructive changes to files.
• The entire contents of the directory are backed-up live to Crashplan, protecting against the destruction of the entire server room (though with a very difficult and slow recovery procedure).
• The entire contents of the directory are occasionally backed up to the external drive in the fireproof safe, protecting against the destruction of the server room and possibly the house with a rapid recovery procedure, but potentially less fresh data.

Specific info about each system's configuration is backed up by a custom backup script. This script comes in 2 parts; a generic version which covers the common files and folders that exist on each system, then an optional second script (filename format is backup.hostname.sh) which is invoked if the first script detects that it exists. This is so individual systems or services can be backed up and restored as needed. These backups produce a tar.gz archive for each system every 3 days with the current configuration and can be used for a complete system recovery. Backups are retained in 3 locations with 3 different rule sets:

• 20 backups are preserved in /home/surfrock66/.sr66_config_backup on each system; this means we can recover a system from a configuration up to 60 days previously
• 5 backups for each device are pushed to sr66-file-01 in /home/surfrock66/.sr66_config_backup; this means we can recover a system's configuration should the device fail
• 2 backups are pushed to Dropbox for each device via sr66-vdi-01; this means we can recover if all systems are destroyed
• The current backup set on sr66-file-01 is backed up to Crashplan

Back to Tech Documentation