| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| wiki:tech:storage [2024/04/25 13:10] – surfrock66 | wiki:tech:storage [2024/04/30 08:03] (current) – surfrock66 |
|---|
| - It is copy-on-write with snapshots. We are currently using several tiers of snapshots. In one, every 30 minutes, a "snapshot" of selected storage is taken, and changes written over the next 30 minutes are stored separately and these snapshots persist for 14 days. This means, if we accidentally deleted a file, or overwrote something with a bad save, we could go back to a previous point in time over the last 14 days and retrieve the previous version. Other snapshot schedules for different groups of storage are things like "once a day for 6 months." | - It is copy-on-write with snapshots. We are currently using several tiers of snapshots. In one, every 30 minutes, a "snapshot" of selected storage is taken, and changes written over the next 30 minutes are stored separately and these snapshots persist for 14 days. This means, if we accidentally deleted a file, or overwrote something with a bad save, we could go back to a previous point in time over the last 14 days and retrieve the previous version. Other snapshot schedules for different groups of storage are things like "once a day for 6 months." |
| |
| The NAS storage [[wiki:inventories:tech:systems:sr66-nas-2|sr66-nas-2]] is presented to other systems via NFS and iSCSI. At this time, all Proxmox VM's have the OS drive and most other storage hosted directly on the NAS via iSCSI (with the exception of the primary storage of the file server, located locally on the Proxmox Hosts). Most secondary bulk file storage is intended to be mounted on the main file server via iSCSI [[wiki:inventories:tech:systems:sr66-file-01|sr66-file-01]] and then mounted to other clients via NFS from there, however some specialized storage is mounted to other devices as needed (for example, there is a Minecraft Backup directory where Minecraft clients replicate their config every 30 minutes via NFS). | The NAS storage [[wiki:inventories:tech:systems:sr66-nas-2|sr66-nas-2]] is presented to other systems via NFS and iSCSI. At this time, all Proxmox VM's have the OS drive and most other storage hosted directly on the NAS via iSCSI (with the exception of the primary storage of the file server, located locally on the Proxmox Hosts). Most secondary bulk file storage is intended to be mounted on the main file server [[wiki:inventories:tech:systems:sr66-file-01|sr66-file-01]] via iSCSI and then mounted to other clients via NFS from there, however some specialized storage is mounted to other devices as needed (for example, there is a Minecraft Backup directory where Minecraft clients replicate their config every 30 minutes via NFS). |
| |
| The file server [[wiki:inventories:tech:systems:sr66-file-01|sr66-file-01]] becomes the focal point for all data. The primary repository of "important files" is a LUN hosted on the Proxmox hosts, on ZFS storage, replicated between all hosts. Surfrock66's home directory is the focal point for storage on the server (services like Nextcloud and Jellyfin look to locations within there for their main storage). All additional storage is mounted to the fileserver (into surfrock66's home folder) as additional iSCSI LUNs from the NAS (Videos, Books, Projects, etc), but secondary copies of important files (Documents, Pictures) exist as NFS shares on the NAS. There is a daily replication via cronjob of these important files from the file server to the NAS so that files exist ZFS protected on each Proxmox host, then also ZFS protected on the NAS with 6 months of snapshots to be able to retrieve files. | The file server [[wiki:inventories:tech:systems:sr66-file-01|sr66-file-01]] becomes the focal point for all data. The primary repository of "important files" is a LUN hosted on the Proxmox hosts, on ZFS storage, replicated between all hosts. Surfrock66's home directory is the focal point for storage on the server (services like Nextcloud and Jellyfin look to locations within there for their main storage). All additional storage is mounted to the fileserver (into surfrock66's home folder) as additional iSCSI LUNs from the NAS (Videos, Books, Projects, etc), but secondary copies of important files (Documents, Pictures) exist as NFS shares on the NAS. There is a daily replication via cronjob of these important files from the file server to the NAS so that files exist ZFS protected on each Proxmox host, then also ZFS protected on the NAS with 6 months of snapshots to be able to retrieve files. |
| ==== Data Backups ==== | ==== Data Backups ==== |
| |
| For backups, the bulk of data backup goes to Crashplan. We pay a license for a backup of unlimited size for 1 computer. Because of how NFS works, Crashplan is completely happy to consider NFS mounted directories as part of a single computer. For that reason, [[wiki:inventories:tech:systems:sr66-vdi-01|sr66-vdi-01]] is the computer we have chosen to back up. Most shared data directories are actually housed on [[wiki:inventories:tech:systems:sr66-file-01|sr66-file-01]] which is ZFS storage on a proxmox host, but are mounted on various computers at various mount points via NFS. | For backups, the bulk of data backup goes to Crashplan. We pay a license for a backup of unlimited size for 1 computer. Because of how NFS works, Crashplan is completely happy to consider NFS mounted directories as part of a single computer. For that reason, [[wiki:inventories:tech:systems:sr66-vdi-01|sr66-vdi-01]] is the computer we have chosen to back up, primarily because it has a desktop environment and Crashplan GUI tools are far superior to any CLI-only option. Most shared data directories are actually housed on [[wiki:inventories:tech:systems:sr66-file-01|sr66-file-01]] which is ZFS storage on a proxmox host, but are mounted on various computers at various mount points via NFS. |
| |
| Additionally, a routine exists to copy selected directories to a small external hard drive, which resides in the locked compartment of the fireproof safe. This is done roughly quarterly. | Additionally, a routine exists to copy selected directories to a small external hard drive, which resides in the locked compartment of the fireproof safe. This is done roughly quarterly. |
