How things stand now

For the past few years my home storage setup has been basically a clone of what I have in colo, albeit with larger drives - SmartOS on a DL160g6 with 72gb of RAM, four 8TB WD Reds (WD80EFZX) on the built-in AHCI controller, configured as RAIDZ2 with no SLOG or L2ARC. On top of this runs an http file server for the Rokus, Netatalk backup targets for Time Machine, and archival copies of mail, photos, laptop images, etc.

Wishes

• More than four disk slots, and the ability at least to have more than four disks in play
• a better disk controller (HBA, not integrated RAID, need to talk to either SAS or SATA, have better performance and deeper queues than AHCI) - will be used exclusively with ZFS
• a faster "zones" pool (particularly in colo, where it could be an all-SSD pool given current price trajectories)
• 10ge capability
• Ability to add an external storage chassis should I need even more storage space or need to do a migration - you don't realize until you're in the middle of it just how long it takes to move some tens of terabytes of data.
• Avoid earning Gaige's budgetary ire for datacenter systems by keeping a lid on power draw. By the way, power isn't free at home either.

Candidate System, power utilization notes

A very appealing target system is the Dell R720xd. This or the plain R720 could end up being blessed as ClueTrust's next standard server platform. These generation-12 systems are the last generation that takes DDR3 memory which means that RAM is reasonably cheap. They're coming off lease these days. We have two datapoints on power utilization:

• Amanda's tricked out R720xd with 2 x HGST SAS3 SSD, 4 x 7200 RPM 8TB NL-SAS, H310 flashed to IT mode, 768gb of RAM (24 sticks of 32gb each) and two performance CPUs with 10 cores each rings the bell at 280 watts idle. Ouch.

• Andrew W-M's got an R720 with 4 x 7.2k SATA, 4 x HGST SAS3 SSD (same model as Amanda's), H710, 128gb of RAM (8 x 16gb DIMMs), and dual e5-2630L v2 CPUs. He reports that it pulls 126w in near idle with 9 vms online (running ESXi) and spikes from time to time to 137.

One of the interesting datapoints I've learned as part of my research is that DDR3 ECC RDIMMs pull about 3-4 watts per stick and it doesn't matter much how big they are memory-wise. It doesn't sound like a lot, but if you do the math as much as 64 watts of the discrepancy between Andrew's and Amanda's configuration may be due to her having 24 DIMMs vs. Andrew's 8. For an organization like ClueTrust, which is exceedingly power-conscious, it likely makes sense to pay the extra money for higher density in order to keep things within the power envelope which we like (the R720 has 2 x 4-channel CPUs, so we are looking at adding ram sticks in multiples of 8).

Useful Subreddits

There are some useful resources on Reddit for this sort of thing. There are r/homelab for people who want to play with stuff at home (occasionally the home of pointers to really good deals) and r/selfhosted where we can occasionally learn about new-to-us FOSS software solutions for doing various stuff.

But by far the most useful subreddit has been r/DataHoarder - people whose aspirations are more along the lines of "production-ish on the cheap", and have been useful for pointers to enterprisey-grade clues on various storage solutions.

It was here that I learned about diskprices.com. While we're at it, let's talk about disks that we might use and how much they draw, since the differences add up to a difference of as much as 20 watts for an array of four drives; that might inform our choices.

Disks and Power Draw

5400 RPM

Model | Cap | Power (idle) | Power (peak) | Notes | :------- | -------: | --------: | --------: | -------- | WD30EFRX | 3T | 2.7W | 4.1W | used in maersk, pil, sealand Datasheet Local Mirror| WD40EFRX | 4T | 3.3W | 4.5W | used in tarros (same datasheet)| WD80EFZX | 8T | 5.2W | 6.4W | used in evergreen (RS home), zim (same datasheet)| WD100EFAX | 10T | 2.8W | 5.7W | Don't own any yet. Helium, 256m cache. Same datasheet.|

7200 RPM

Model | Cap | Power (idle) | Power (peak) | Notes | :------- | -------: | --------: | --------: | -------- | HUH728080AL5200 | 8T | 5.7W | 9.1W | used in alfar (SAS) DatasheetLocal Mirror| HUS724040ALE640 | 4T | 6.9W | 11.4W | used in Andrew W-M's R720 Datasheet Local Mirror| ST500NM0001 | 500m | 5.5W | 7.0W | pile-o-disks for lab playground in md1200 Datasheet Local Mirror| WD101KFBX | 10T | 2.8W | 5.7W | Don't own any yet. Helium, 256m cache. Datasheet Local Mirror| ST6000NM0034 | 6T | 7.97W | 11.86W | You can save on price but you'll spend a lot on power. These drives are a perennial favorite for cheap end on diskprices.com at $130-ish for 6T new, and available as refurbs for as little as $75. Datasheet Local Mirror|

15000 RPM

Model | Cap | Power (idle) | Power (peak) | Notes | :------- | -------: | --------: | --------: | -------- | HUC156030CSS204 | 300G | 5W | 7.5W | came stock in our toys-r-us r330s Datasheet Local Mirror|

SSD

Model | Cap | Power (idle) | Power (peak) | Notes | :------- | -------: | --------: | --------: | -------- | HUSML4040ASS601 / HRLP0400S5xnEMLC | 400G | 1.7W | 5.5 | used in alfar, greatwhite, etc. High endurance enterprise SAS SSDs. Datasheet Local Mirror EVO 850 | 500G | 70mW | 3.6W | Desktop/Laptop SSDs used in dole Datasheet Local Mirror

Random links - Reddit and elsewhere

Apparently some NetApp shelves (DS4243 and DS4246) make awfully nice, though loud, external shelves for ZFS. I don't think I need 24 drives though.

Some opinions on the best JBOD card including a shout-out to the LSI 9201-16e which I bought one of for something like $30 delivered.

The Dell MD1200 will support SATA drives notwithstanding what the docs may say. They draw about 50W empty plus whatever drives you add in but can be a little noisy for people's taste for home. Confirmed by RS - it worked with a couple of WD Reds in it.

The MD1200 had a bad time with SmartOS (crummy performance in multipathing, which righted itself once the cable to the second controller was unplugged). Someone on the smartos-discuss list fingered this as being fairly common and traceable to drive firmware dealing poorly with round-robin. He said that there are ways to set the scsi-vhci driver to use logical-block for load-balancing (or "none" since it's actually not a great loss to not have load-balancing over the multipath on disks).

Unfortunately this is hard to fix on SmartOS unless you're building your own images since the settings are baked into the thumb drive. Then again, do we really need multipathing? Probably get by without it.

Manual for the MD1200 Local Mirror

I'm sure there wil be more of this in the future, but I'm already 12 days overdue for publishing these notes, so out the door they go!

Comments? Additions? Corrections? You know where to find me.

(Save the observations about "the table formatter is kind of lame. I know it is. This is GFM not professional web design, right?)