[Updated 09/2022] My New Energy-Efficient NAS to Replace the Dell R730xd

 This is really a fine machine. Scored it on eBay, came with 128gb of DDR4 ram, dual Xeon 2630 v4, and 12 3TB Seagate SAS hard drives. It has 4 gigabit ethernet ports built-in, on board IPMI, very server-y stuff.

 I loved this machine. It is reliable as hell, everything just works. I put a 4x4 NVMe card into the PCIe x16 slot, a Mellanox Connect-X3 Pro into the PCIe x8 slot, flashed the onboard RAID controller with IT firmware. All of a sudden, I've got myself hell of a server that can do 40gbps, 4  super fast NVMe drives, and 12 un-restricted HDD slots.

 However, being an enterprise-grade hardware, it is inherently loud and generates lots of heat. This bad boy, living in a southern California garage, often heats the space to upwards of 100F. Furthermore, most of my workloads that run on this server requires higher single-core performance other than the sheer core count. Because of that, I built another DIY server with Intel 10850K, a 10 core CPU with what was at the time the best single core performance, and moved most of the workload off from the Dell R730xd.

 Now all that's left on the Dell was PfSense - a router/firewall combo, Windows Server for my day-to-day Windows needs, and most importantly, Unraid.

 While not having a ton of workload, it is still pulling in excess of 170W from the wall. So, I started searching for my next NAS with a goal of delivering the same or better Unraid performance, while reducing energy usage.

 During last year's Chia mining craze, I started building NASes. When searching for an ideal case to put all those hard drives, I stumbled upon this seemingly perfect NAS case: Fractal Design Node 804.

 It looks great, uses normal PC parts, can support up to 10 3.5' drives, supports 240mm radiators for water-cooling, all for less than $130. So I pulled the trigger on it, and started searching for the components which will go into this case.

 And this is where I ran into a big problem: the limited PCIe lanes on consumer platforms.

 What I need, to put it simply, is an mATX motherboard with the PCIe slots to house:

1. LSI 2008 HBA (PCIe 2.0 x8)
2. Mellanox ConnectX3 NIC (PCIe 3.0 x8 for the 2x40Gbe QSFP+, or x4 for the 2x10Gbe SFP+)
3. Intel Quad-Port NIC (PCIE 2.0 x4)

[Update Sep 2022]: Since I found an M.2 HBA, the LSI 2008 is no longer used in this build. Also, I have reverted back to my trusted Ubiquiti Edgerouter X, so the Intel Quad Port NIC is also decommissioned in favor of IOCREST's dual port 2.5Gbe NIC.

 The prerequisite, is the form factor. Node 804 supports ITX and mATX. ITX is automatically ruled out since motherboard manufacturers unilaterally decided it can only have ONE PCIe slot. So I started to look for mATX boards that suit my needs.

  Since I have a 3900X and a 1800X laying around, we'll first look at AMD's offerings. For my expansion needs, I was looking at B450, X570, and later B550 as well. One of the strength for AMD is that the PCIe x16 slot can be bifurcated to multiple configurations, and X570 even has PCIe 4.0 datalink between the CPU and the chipset. However, since none of my AMD CPU comes with a GPU, I will need a GPU in the PCIe x16 slot just for the system to boot, even though I DON'T need it to run any of my workloads. Some motherboard just won't boot when used headlessly. Rumor has it that AsRock motherboards can boot headless, so I went to AsRock's website to see if there's anything that fits.

 The AsRock X570M Pro4 is the only X570 mATX offering from AsRock, and it has one PCIe x16, one PCIe x4 in the form of x16 slot, and one PCIe x1. Apparently, if I decide to put the HBA in the PCIe x16, the Mellanox NIC in the PCIe x4, I won't be able to fit anymore expansion cards. Furthermore, if the board doesn't boot headlessly, I will need to put a GPU into that x16 slot, further degrading the expandability.

 Next, is B550 chipset. AsRock does offer a very interesting board at this segment, the B550M-C. It has 4 PCIe slots that runs x16, x1, x1, and x4. Compared to the X570M Pro4, I do get one more PCIe x1, but that seemed useless to my use case. I suspect this board is for GPU crypto miners, since the x16 slot can be further bifurcated to four x4 slots using an expansion card.

 So, AMD is a no-go. How about Intel, then? At first look, Intel already has a good start - almost every CPU comes with an integrated GPU, so I can preserve a PCIe x16 slot. However, prior to 12th gen chipsets, Intel connects its CPU and chipset using DMI 3.0 x4, this is effectively a PCIe 3.0 x4 connection. So every peripheral, including the PCIe 3.0 x4 that's provided by the chipset, will have to communicate with the CPU using this PCIe 3.0 x4 bus. If I choose to use the chipset PCIe 3.0 x4 for the 10/40Gbe NIC, the bandwidth limitation will have a good chance of rearing its ugly head, especially when chipset NVMe is also in use.

 In Jan 4, 2022, Intel announces 12th Gen Core Alder Lake CPUs. With the launch of 12th Gen line, we've also got H610, B660 and Z690 chipsets to the company.

 What's really exciting this time around, is that Intel connects B660 with PCIe 4.0 x4, and Z690 with PCIe 4.0 x8, effectively doubling and quadrupling the bus bandwidth. With this upgrade, I can put that NIC in the PCIe slot provided by the chipset, and not having to worry about hitting the bandwidth limit.

 The CPU of choice, of course, is the new Core i3 12100. It provides 45% more single core performance than my Xeon E5 2630 V4, while consuming less power. Of course, I won't be topping any Cinebench benchmarks and it doesn't support ECC, but those are compromises that I am willing to make in order to save a big deal on electricity and reduce noise, and let other equipments in the garage have a better time.

 Starting with the i3-12100, I begin looking for motherboards. Luckily, a simple "B660m" search on google yielded many insights. After some comparative analysis, I settled down on ASUS B660M-A D4.

 As you can see, it is not that different than the X570 and B550 boards I have examined above. However, this x1 slot being in the form of x16, it can really open up some interesting applications. I have a Mellanox ConnectX3 CX311A on hand, this is a single 10Gbe SFP+ card, and utilizes a PCIe 3.0 x4 connection to the motherboard. Single 10Gbe connection speed is 10000 Mbit/s, which is 1250 Mbytes/s, or 1.25 GB/s. Single PCIe 3.0 x1 link is about 0.985 GB/s, which is roughly 79% of the full 10Gbe speed. Based on my real world observations, a 10Gbe link transfer speed is around the neighborhood of 940MB/s. Take this number and multiply it by 0.79, I got 742 MB/s. I know, it is not full 10Gbe, but it is still miles better than 1Gbe, or 2.5Gbe, or even 5Gbe that requires expensive switches to work.

 Finally, the bill of material, as of September 5th, 2022:

1. CPU - Intel Core i3 12100 ($119)
2. Motherboard - ASUS PRIME B660M-A D4 ($119.99)
3. CPU Cooler - Noctua U9S ($59)
4. RAM - 32G G.Skill 4x8GB (about $150, on hand)
5. PSU - Seasonic 660W 80Plus Platinum (about $80, on hand)
6. Case - Fractal Design Node 804 ($130, on hand)
7. Fan - 7x Arctic P12 PWM PST (swapped out the OEM Fractal ones)
8. System Drive - Samsung 980 Pro 2TB, NVMe PCIe 4.0 x4
9. SSD Group A - 2x 500GB Crucial MX500 in RaidZ ($100, on hand)
10. HDD Group A - WD Red Pro 12TB * 8 ($xxx, on hand, four disk RAID-Z1, two vdevs)
11. HDD Group B - WD Red Pro 12TB * 2 (for Unraid)
12. HBA - IOCREST M.2 PCIe 3.0x2 to 8 SATA (passed to TrueNAS Scale)
13. Cache Drive - Virtual Disk from System Drive, 256GB (only for Unraid)
14. NIC - IOCREST Dual Port 2.5Gbe PCIe 2.0 x 1 (RTL8125B, Paravirtualized for VMs)
15. Total - 10 SATA HDDs, 2 SATA SSDs, 1 NVMe SSD, with both PCIe x16 and PCIe x4 slots open.
16. Idle Power Usage: Measured by Kill-A-Watt, 100W.