Hello, author here! It's a nice surprise to notice my own post here, but the timing is unfortunate as I'm shuffling things around on my home server and will accidentally/intentionally take it offline for a bit.
The official TrueNAS docs recommend against using USB drives [1]. My understanding is that between the USB controller, flaky connectors and cables, and usb-to-sata bridges of varying quality, there are just too many unknowns to guarantee a reliable experience. For example, I’ve heard that some usb-to-sata controllers will drop commands and not report SMART data. That said, there are of course many people on the internet who have thrown caution to the wind and report that it’s working fine for them.
Personally I’m in the process of building a NAS with an old 9th gen Intel i5. Many mobos support 6 SATA ports and three mirrored 20 TB pairs is enough storage for me. I’m guessing it’ll be a bit more power hungry than a ugreen/synology/etc appliance but there will also be plenty of headroom for running other services.
Been using like 7 external usb drives with 40-50tb total for a few years with no issues. Not raid, just backing up drive to drive. No controller or drive issues. Mix of seagate and wd 8/12/16gb.
I hate blanket recommendations like this by docs. To me, it just sounds like some guy had a problem a few times and now it's canon. It's like saying "avoid Seagate because their 3tb drives sucked." Well they did, but now they seem to be fine.
What may work anecdotally can't necessarily be used for official recommendations for a large range of users across an unknown range of hardware configurations. If it works for you, that's fine. That isn't sufficient to make a general statement that everybody will be fine using external USB drives, particularly for RAID, especially when people will then make you responsible if something goes wrong for not making sufficiently safe recommendations. You understand that, right?
Mainly I wouldn't do it because of there's space and SATA ports it seems stupid. Hotter. Worse HW.
Can't really see much good reason to do it tbh except it's in a small hot case which is relatively easy to move around. Maybe if you do occasionally backups and you don't care about scrubbing and redundancy? Otherwise why not shuck them and throw them in a case?
Yes I should have specified that this advice is specific to RAID configurations in NAS applications.
If you're occasionally copying data to an external USB drive, that's totally fine. That's what they were designed for.
The issue is that they were not designed for continuous use, or much more demanding applications like rebuilding/resilvering a drive. It's during these applications that issues occur, which is a double whammy, because it can cause permanent data loss if your USB drive fails during a recovery operation. I did a little more research after posting my last comment and came across this helpful post on the TrueNAS forums going into more depth: https://forums.truenas.com/t/why-you-should-avoid-usb-attach...
YMMV. I have a 4-drive 20TB mdraid10 across two different $50 USB3.0 2-drive enclosures, I've read petabytes off this array with years of uptime and absolutely zero problems. And it runs on one of those $300 off brand NUCs. The 2.5G NIC is the bottleneck on reads.
Yes. It's pricey but it's never been a problem. It can connect like 12 HDDs with 256GB ram and has 10GBe and runs at a tiny TDP. Has IPMI. Fits in a tiny case.
The only issue I had with this motherboard was that it was difficult to find someone who sold it. Love it
Also I don't see the built-in UPS. The external drives still use external power
I don't use a laptop, but I use something fairly adjacent: the Beelink SER6 (https://www.amazon.com/Beelink-4-75GHz-PCIe4-0-Supports-HDMI...), which is basically a gaming laptop converted into a small desktop. For the most part, it has actually been pretty great. It's quiet, has a CPU that is much better than I expected, and a decent enough GPU to do hardware transcoding for Jellyfin without much issue.
I use USB chassis of hard drives to work as the "NAS" part, and it works fairly well, and this box is also my router (using a 10 GbE thunderbolt adapter) though my biggest issue comes with large updates in NixOS.
For reasons that are still not completely clear to me, when I do a very large system update (rebuilding Triton-llvm for Immich seems to really do it), the internal network will consistently cut out until I reboot the machine. I can log in through the external interface with Tailscale and my phone, so the machine itself is fine, but for whatever reason the internal network will die.
And that's kind of the price you pay for using a non-server to do server work. It will generally work pretty well, but I find that it does require a bit more babysitting than a rack mount server did.
For me it was important to have ECC RAM and laptops pretty much never have that. My personal recommendation is an old IBM/Lenovo workstation tower as the base. I bought one for $35 on eBay and added $40 of RAM (32GB). A $10 UPS from Goodwill with a $25 battery from Amazon, and whatever hard drives you want. I run Ubuntu and ZFS on it but next time would probably opt for FreeBSD for a nicer OS.
I ran an old Thinkpad as a home router and small home server/NAS device for quite a long time, usually swapping out my old work upgrades every 3 years or so.
They all had onboard gige so it worked fine - native vlan for the inbound Comcast connection, tagged vlans out to a switch for the various LAN connections.
They were from the era of DVD drives so I was able to put an extra HDD in the DVD slot to expand storage with. One model even had a eSATA port.
They worked great. Built-in UPS and they come with a reliable crash cart built-in!
If you don’t need any performance it’s a great backup strategy. If your only way of connecting the drives to the laptop is USB I would be concerned about data integrity if it’s important data.
Individual writes are safe, in my
Experience with thousands of
uSB drives in many configurations, some with 12 2tb drives hanging on multiple USB hubs at the same time.
However, there are disconnects/reconnects every now and then. If you use a standard raid over these usb drives, almost every disconnect/reconnect will trigger a rebuild — and rebuilds take many hours. If you are unlucky enough to have multiple disconnects during a rebuild, you are in trouble.
When I used a laptop as server, the battery became a spicy pillow. I think laptops are not designed to be running continuously and on warmer temperatures than normal.
The UNAS Pro 8 just came out and I'm thinking about getting it, switching away from my aging Synology setup ... only thing I wish it had was a UPS server as my Synology currently serves that purpose to trigger other machines to shut down ...
I believe Synology's UPS monitoring is based on nut-server[1]. In my setup, I am running the server on a separate machine that reads UPS state over USB and Synology is just a client. Maybe UNAS could also just work as a client.
I'm considering doing the same, I guess one would basically just be splitting functions, a dedicated NAS, and a dedicated server for all the functions that Synos tend to perform (generally not very well, but at least with pretty low power usage).
I've been considering "de-enterprising" my home storage stack to save power and noise and gain something a bit more modular. Currently I'm running on an old NAS 1U machine that I bought on eBay for about $300, with a raidz2 of 12x 18TB drives. I have yet to find a good way to get nearly that much storage without going enterprise or spending an absolute fortune.
I'm always interested in these DIY NAS builds, but they also feel just an order of magnitude too small to me. How do you store ~100 TB of content with room to grow without a wide NAS? Archiving rarely used stuff out to individual pairs of disks could work, as could running some kind of cluster FS on cheap nodes (tinyminimicro, raspberry pi, framework laptop, etc) with 2 or 4x disks each off USB controllers. So far none of this seems to solve the problem that is solved quite elegantly by the 1U enterprise box... if only you don't look at the power bill.
I have to imagine that the best NAS build is simply a 6-core or 8-core standard AMD or Intel with a few HBA controllers and maybe 10Gbit SPF+ fiber or something.
"Old server hardware" for $300 is a bit of a variation, in that you're just buying something from 5 years ago so that its cheaper. But if you want to improve power-efficiency, buy a CPU from today rather than an old one.
--------
IIRC, the "5 year old used market" for servers is particularly good because many datacenters and companies opt for a ~5-year upgrade cycle. That means 5-year-old equipment is always being sold off at incredible rates.
Any 5-year-old server will obviously have all the features you need for a NAS (likely excellent connectivity, expandibility, BMS, physical space, etc. etc.). Just you have to put up with power-efficiency specs of 5 years ago.
> How do you store ~100 TB of content with room to grow without a wide NAS?
In the cloud (S3) or on offline (unpowered HDDs or tapes or optical media) I suppose. Most people just don't store that much content.
> So far none of this seems to solve the problem that is solved quite elegantly by the 1U enterprise box... if only you don't look at the power bill.
What kind of power bill are you talking about? I'd expect the drives to be about 10W each steady state (more when spinning up), so 180W. I'd expect a lower-power motherboard/CPU running near idle to be another 40W (or less). If you have a 90% efficient PSU, then maybe 250W in total.
If you're way more than that, you can probably swap out the old enterprisey motherboard/RAM/CPU/PSU for something more modern and do a lot better. Maybe in the same case.
I'm learning 1U is pretty unpleasant though. E.g. I tried an ASRock B650M-HDV/M.2 in a Supermicro CSE-813M. A standard IO panel is higher than 1U. If I remove the IO panel, the motherboard does fit...but the VRM heatsink also was high enough that the top case bows a bit when I put it on. I guess you can get smaller third party VRM heat sinks, but that's another thing to deal with. The CPU cooler options are limited (the Dynatron A42 works, but it's loud when the CPU draws a lot of power). 40mm case fans are also quite loud to move the required airflow. You can buy noctuas or whatever, but they won't really keep it cool. The ones that actually do spin very fast and so are very loud. You must have noticed this too, although maybe you have a spot for the machine where you don't hear the noise all the time.
I'm trying 2U now. I bought and am currently setting up an Innovision AS252-A06 chassis: 8 3.5" hot swap bays, 2U, 520mm depth. (Of course you can have a lot more drives if you go to 2.5" drives, give up hot swap, and/or have room for a deeper chassis.) Less worry about if stuff will fit, more room for airflow without noise.
If you want 100TB, you need a bigger NAS than most, and that makes most of the DIY NAS not so good. 2-4 drives seems to be where DIY shines. These days motherboards often stop at 4x sata, so you'll need a HBA or USB (eww).
Personally, I just don't have that much data, 24TB mirrored for important data is probably enough, and I have my old mirror set avaialable for media like recorded tv and maybe dvds and blu-rays if I can figure out a way to play them that I like better than just putting the discs in the machine.
I admire the courage to store data on refurbished Seagate hard drives. I prefer SSD storage with some backups using cloud cold storage, because I’m not the one replacing the failing hard drives.
I would also prefer having a large number of high capacity SSDs so I could replace my spinning hard drives.
But even the cheapest high capacity SSD deals are still a lot more expensive than hard drive array.
I’ll continue replacing failing hard drives for a few more years. For me that has meant zero replacements over a decade, though I planned for a 5% annual failure rate and have a spare drive in the case ready to go. I could replace a failed drive from the array in the time takes to shut down, swap a cable to the spare drive, and boot up again.
SSDs also need to be examined for power loss protection. The results with consumer drives are mixed and it’s hard to find good info about how common drives behave. Getting enterprise grade drives with guaranteed PLP from large on-onboard capacitors is ideal, but those are expensive. Spinning hard drives have the benefit of using their rotational inertia to power the drive long enough to finish outstanding writes.
You can find cheap used enterprise SSDs on ebay. But the problem is that even the most power efficient enterprise SSD (SATA) idle at like 1w. And given the smaller capacities, you need many more to match a hard drive. In the end HDD might actually consume less power than an all flash array + controllers if you need a large capacity.
Curious, what's the use case for wanting your data backed-up without fail? Is it personal archives or otherwise (business) archive related?
Not to say you shouldn't backup your data, but personally I wouldn't be to affected if one of my personal drives errored out, especially if they contained unused personal files from 10+ years ago (legal/tax/financials are another matter).
Any data I created, paid to license, or put in significant work to gather has to be backed-up with 3-2-1 rule. Stuff I can download or otherwise obtain again is best effort but not mandatory backup.
Mainly I don't want to lose anything that took work to make or get. Personal photos, videos, source code, documents, and correspondence are the highest priority.
Don't RAID these days. Software won rather drastically, likely because CPUs are finally powerful enough to run all those calculations without much of a hassle.
Software solutions like Windows Storage Spaces, ZFS, XFS, unRAID, etc. etc are "just better" than traditional RAID.
Yes, focus on 2x parity drive solutions, such as ZFS's "raidz2", or other such "equivalent to RAID6" systems. But just focus on software solutions that more easily allow you to move hard drives around without tying them to motherboard-slots or other such hardware issues.
I believe RAID5/6 is still experimental (although I believe the main issues were worked out in early 2024), I've seen reports of large arrays being stable since then. It's still recommended to run metadata in raid1/raid1c3.
Might be a bit adventurous for primary storage (though with enough backup and redundancy, why not). But seems perfect for me for backup / cold storage.
Not really. I know that my sleep is worth more than the difference between HDD and SSD prices, and I know the difference between the failure rates and the headache caused by the RMA process, so I buy SSDs.
In essence, what we together are saying is that people with super-sensitive sleep that are also easily upset, and that don't have ultra-high salaries, cannot really afford 18 TB of data (even though they can afford an HDD), and that's true.
Well, again, well done on being able to afford it. I have 24TB array on cheap second hand drives from CEX for about £100 each, using DrivePool - and guess what, if one of them dies I'll just buy another £100 second hand drive. But also guess what - in the 6 years I had this setup, all of these are still in good condition. Paying for SSDs upfront would have been a gigantic financial mistake(imho).
There's a big difference between used as in I just bought this hard drive and have used it for a week in my home server, and used as in refurbished drive after years of hard labor in someone else's server farm
Enterprise drives are way different than anything consumer based. I wouldn't trust a consumer drive used for 2 years, but a true enteprise drive has like millions of hours left of it's life.
Quote from Toshiba's paper on this. [1]
Hard disk drives for enterprise server and storage usage (Enterprise Performance and Enterprise Capacity Drives) have MTTF of up
to 2 million hours, at 5 years warranty, 24/7 operation. Operational temperature range is limited, as the temperature in datacenters
is carefully controlled. These drives are rated for a workload of 550TB/year, which translates into a continuous data transfer rate of
17.5 Mbyte/s[3]. In contrast, desktop HDDs are designed for lower workloads and are not rated or qualified for 24/7 continuous
operation.
From Synology
With support for 550 TB/year workloads1 and rated for a 2.5 million hours mean time to failure (MTTF), HAS5300 SAS drives are built to deliver consistent and class-leading performance in the most intense environments. Persistent write cache technology further helps ensure data integrity for your mission-critical applications.
There isn’t a significant difference between “enterprise” and “consumer” in terms of fundamental characteristics. They have different firmware and warranties, usually disks are tested more methodically.
Max operating range is ~60C for spinning disks and ~70C for SSD. Optimal is <40-45C. The larger agents facilties afaik tend to run as hot as they can.
> drive has like millions of hours left of it's life.
It doesn't apply for the single drive, only for a large number of drives. E.g. if you have 100000 drives (2.4 million hours MTTF) in a server building with the required environmental conditions and maximum workload, be prepared to replace a drive once a day in average.
Drive failure rate versus age is a U-shaped curve. I wouldn't distrust a used drive with healthy performance and SMART parameters.
And you should use some form of redundancy/backups anyway. It's also a good idea to not use all disks from the same batch to avoid correlated failures.
What exactly are these "white label drives"? Aren't these just normal seagate exos drives with SMART information wiped and labels removed? i.e. just a worse used drive.
I never understood why they let Seagate et al do this game about hard drives. If they offer a warranty, then replace the drive to brand new, and shove the recertified, fixed whatever bullshit up your wahzoo.
> Half of tech YouTube has been sponsored by companies like...
It just struck me that the product reviews are a part of the social realm that is barely explored.
Imagine a video website like TikTok or YouTube etc where all videos are organized under products. Priority to those who purchased the product and a category ranked by how many similar products you've purchased.
The thing sort of exists currently in some hard to find corner of TEMU etc but there are no channels or playlists.
They don't have to tell you anything. Just unbox and show what they got.
I just purchased a bicycle chain cleaning device. It was absurdly cheap. The plastic was extruded poorly, it was hard to assemble, it was not entirely obvious how to use it. However! It did the job and it barely got dirty. I expected it to be full of rusty oil both inside and outside but it accumulated just a tiny smudge on the inlet. If anyone made a video it would be a fantastic product.
God, the flood of absolutely useless "review" videos Amazon has incentivized customers to shit all over their site which are nothing more than unboxings are the worst thing about that ecosystem. No thank you.
Alternatively, unknown creators have less incentive to falsely promote or lie. It’s the reason I tend to trust random strangers on Reddit than popular YouTubers who have achieved monetization and sponsorship.
I don’t trust big channels especially, because I assume they have just sold themselves out to the biggest sponsor. Influencers only exist due to campaign deals, where companies try to sneak their ads into your mind by abusing your inclination to trust another human being. All of it is sickening.
In comparison, I’d rather read a general review magazine with a long history. At least they don’t try to trick me into believing they are working out of the goodness of their hearts, and they usually aren’t married to a single big sponsor.
Coincidentally or not, those folks who have more subscribers usually charge more for their consideration. That’s why I generally trust Steve of Gamers Nexus more than other folks, because they don’t do ads except for promoting their own products, so there’s no conflict of interest. On the one hand, Gamers Nexus doesn’t manufacture their own hard drives, but on the other, they publish their methodology and have a reputation to uphold, so I would trust their judgement regarding testing computer hardware more than folks who do engage in outside advertising.
There's Kakaku.com[1] in Japanese Internet for all consumer electronics, Minkara for cars, bookmeter.com for books, and 5ch.net as fallback. It's surprising that there's only Goodreads on English Internet that everyone have heard of...
Hello, author here! It's a nice surprise to notice my own post here, but the timing is unfortunate as I'm shuffling things around on my home server and will accidentally/intentionally take it offline for a bit.
Here's a Wayback Machine copy of the page when that does happen: https://web.archive.org/web/20251006052340/https://ounapuu.e...
I was about to buy a NAS. I find the idea of using an old laptop instead interesting. Especially since it comes with UPS built in.
The author is using a ThinkPad T430.
Any experiences?
The official TrueNAS docs recommend against using USB drives [1]. My understanding is that between the USB controller, flaky connectors and cables, and usb-to-sata bridges of varying quality, there are just too many unknowns to guarantee a reliable experience. For example, I’ve heard that some usb-to-sata controllers will drop commands and not report SMART data. That said, there are of course many people on the internet who have thrown caution to the wind and report that it’s working fine for them.
Personally I’m in the process of building a NAS with an old 9th gen Intel i5. Many mobos support 6 SATA ports and three mirrored 20 TB pairs is enough storage for me. I’m guessing it’ll be a bit more power hungry than a ugreen/synology/etc appliance but there will also be plenty of headroom for running other services.
[1] https://www.truenas.com/docs/core/13.0/gettingstarted/coreha...
I've had the same thing from random disconnects etc from various USB hard drives and SSD's over the years.
Been using like 7 external usb drives with 40-50tb total for a few years with no issues. Not raid, just backing up drive to drive. No controller or drive issues. Mix of seagate and wd 8/12/16gb.
I hate blanket recommendations like this by docs. To me, it just sounds like some guy had a problem a few times and now it's canon. It's like saying "avoid Seagate because their 3tb drives sucked." Well they did, but now they seem to be fine.
What may work anecdotally can't necessarily be used for official recommendations for a large range of users across an unknown range of hardware configurations. If it works for you, that's fine. That isn't sufficient to make a general statement that everybody will be fine using external USB drives, particularly for RAID, especially when people will then make you responsible if something goes wrong for not making sufficiently safe recommendations. You understand that, right?
Is that with ZFS or something else?
Mainly I wouldn't do it because of there's space and SATA ports it seems stupid. Hotter. Worse HW.
Can't really see much good reason to do it tbh except it's in a small hot case which is relatively easy to move around. Maybe if you do occasionally backups and you don't care about scrubbing and redundancy? Otherwise why not shuck them and throw them in a case?
RAID is much different. You can try it over USB, you won't have a good time. TrueNAS is primarily talking about RAID users.
Yes I should have specified that this advice is specific to RAID configurations in NAS applications.
If you're occasionally copying data to an external USB drive, that's totally fine. That's what they were designed for.
The issue is that they were not designed for continuous use, or much more demanding applications like rebuilding/resilvering a drive. It's during these applications that issues occur, which is a double whammy, because it can cause permanent data loss if your USB drive fails during a recovery operation. I did a little more research after posting my last comment and came across this helpful post on the TrueNAS forums going into more depth: https://forums.truenas.com/t/why-you-should-avoid-usb-attach...
YMMV. I have a 4-drive 20TB mdraid10 across two different $50 USB3.0 2-drive enclosures, I've read petabytes off this array with years of uptime and absolutely zero problems. And it runs on one of those $300 off brand NUCs. The 2.5G NIC is the bottleneck on reads.
I own this and it's worth it's weight in gold https://www.supermicro.com/en/products/motherboard/A2SDi-H-T...
Yes. It's pricey but it's never been a problem. It can connect like 12 HDDs with 256GB ram and has 10GBe and runs at a tiny TDP. Has IPMI. Fits in a tiny case.
The only issue I had with this motherboard was that it was difficult to find someone who sold it. Love it
Also I don't see the built-in UPS. The external drives still use external power
That's an amazing board. I had no idea something like this existed.
I don't use a laptop, but I use something fairly adjacent: the Beelink SER6 (https://www.amazon.com/Beelink-4-75GHz-PCIe4-0-Supports-HDMI...), which is basically a gaming laptop converted into a small desktop. For the most part, it has actually been pretty great. It's quiet, has a CPU that is much better than I expected, and a decent enough GPU to do hardware transcoding for Jellyfin without much issue.
I use USB chassis of hard drives to work as the "NAS" part, and it works fairly well, and this box is also my router (using a 10 GbE thunderbolt adapter) though my biggest issue comes with large updates in NixOS.
For reasons that are still not completely clear to me, when I do a very large system update (rebuilding Triton-llvm for Immich seems to really do it), the internal network will consistently cut out until I reboot the machine. I can log in through the external interface with Tailscale and my phone, so the machine itself is fine, but for whatever reason the internal network will die.
And that's kind of the price you pay for using a non-server to do server work. It will generally work pretty well, but I find that it does require a bit more babysitting than a rack mount server did.
For me it was important to have ECC RAM and laptops pretty much never have that. My personal recommendation is an old IBM/Lenovo workstation tower as the base. I bought one for $35 on eBay and added $40 of RAM (32GB). A $10 UPS from Goodwill with a $25 battery from Amazon, and whatever hard drives you want. I run Ubuntu and ZFS on it but next time would probably opt for FreeBSD for a nicer OS.
I ran an old Thinkpad as a home router and small home server/NAS device for quite a long time, usually swapping out my old work upgrades every 3 years or so.
They all had onboard gige so it worked fine - native vlan for the inbound Comcast connection, tagged vlans out to a switch for the various LAN connections.
They were from the era of DVD drives so I was able to put an extra HDD in the DVD slot to expand storage with. One model even had a eSATA port.
They worked great. Built-in UPS and they come with a reliable crash cart built-in!
If you don’t need any performance it’s a great backup strategy. If your only way of connecting the drives to the laptop is USB I would be concerned about data integrity if it’s important data.
Why is USB so bad at data integrity. Doesn't it have error detection/correction? If so, that sounds like a huge design flaw.
Individual writes are safe, in my Experience with thousands of uSB drives in many configurations, some with 12 2tb drives hanging on multiple USB hubs at the same time.
However, there are disconnects/reconnects every now and then. If you use a standard raid over these usb drives, almost every disconnect/reconnect will trigger a rebuild — and rebuilds take many hours. If you are unlucky enough to have multiple disconnects during a rebuild, you are in trouble.
I've had bitflips with USB transfers of 1-10TB. I don't remember the specifics, but my personal confidence in USB is low.
When I used a laptop as server, the battery became a spicy pillow. I think laptops are not designed to be running continuously and on warmer temperatures than normal.
I’ve use an P51 for about a year now with no issues. I initially bought 6bay DAS, but I’ve since moved to pure SSD storage inside the laptop.
You can. It works fine if you know the limitations. An important one is, drives could disconnect, so traditional RAID wouldn't be good.
If you want redundancy, look at something like SnapRAID, http://www.snapraid.it
If you want to combine into a single volume, consider rclone. These remotes specifically are the ones I'm thinking could be useful,
- https://rclone.org/local/
- https://rclone.org/combine/
- https://rclone.org/cache/
Good luck o7
> I was about to buy a NAS.
The UNAS Pro 8 just came out and I'm thinking about getting it, switching away from my aging Synology setup ... only thing I wish it had was a UPS server as my Synology currently serves that purpose to trigger other machines to shut down ...
I believe Synology's UPS monitoring is based on nut-server[1]. In my setup, I am running the server on a separate machine that reads UPS state over USB and Synology is just a client. Maybe UNAS could also just work as a client.
[1]: https://networkupstools.org/
I'm considering doing the same, I guess one would basically just be splitting functions, a dedicated NAS, and a dedicated server for all the functions that Synos tend to perform (generally not very well, but at least with pretty low power usage).
I think they just released some new prosumer ups.
I've been considering "de-enterprising" my home storage stack to save power and noise and gain something a bit more modular. Currently I'm running on an old NAS 1U machine that I bought on eBay for about $300, with a raidz2 of 12x 18TB drives. I have yet to find a good way to get nearly that much storage without going enterprise or spending an absolute fortune.
I'm always interested in these DIY NAS builds, but they also feel just an order of magnitude too small to me. How do you store ~100 TB of content with room to grow without a wide NAS? Archiving rarely used stuff out to individual pairs of disks could work, as could running some kind of cluster FS on cheap nodes (tinyminimicro, raspberry pi, framework laptop, etc) with 2 or 4x disks each off USB controllers. So far none of this seems to solve the problem that is solved quite elegantly by the 1U enterprise box... if only you don't look at the power bill.
I have to imagine that the best NAS build is simply a 6-core or 8-core standard AMD or Intel with a few HBA controllers and maybe 10Gbit SPF+ fiber or something.
"Old server hardware" for $300 is a bit of a variation, in that you're just buying something from 5 years ago so that its cheaper. But if you want to improve power-efficiency, buy a CPU from today rather than an old one.
--------
IIRC, the "5 year old used market" for servers is particularly good because many datacenters and companies opt for a ~5-year upgrade cycle. That means 5-year-old equipment is always being sold off at incredible rates.
Any 5-year-old server will obviously have all the features you need for a NAS (likely excellent connectivity, expandibility, BMS, physical space, etc. etc.). Just you have to put up with power-efficiency specs of 5 years ago.
Dell R500 series is very good for dense storage at low costs if you lean to SATA or NL-SAS
> How do you store ~100 TB of content with room to grow without a wide NAS?
In the cloud (S3) or on offline (unpowered HDDs or tapes or optical media) I suppose. Most people just don't store that much content.
> So far none of this seems to solve the problem that is solved quite elegantly by the 1U enterprise box... if only you don't look at the power bill.
What kind of power bill are you talking about? I'd expect the drives to be about 10W each steady state (more when spinning up), so 180W. I'd expect a lower-power motherboard/CPU running near idle to be another 40W (or less). If you have a 90% efficient PSU, then maybe 250W in total.
If you're way more than that, you can probably swap out the old enterprisey motherboard/RAM/CPU/PSU for something more modern and do a lot better. Maybe in the same case.
I'm learning 1U is pretty unpleasant though. E.g. I tried an ASRock B650M-HDV/M.2 in a Supermicro CSE-813M. A standard IO panel is higher than 1U. If I remove the IO panel, the motherboard does fit...but the VRM heatsink also was high enough that the top case bows a bit when I put it on. I guess you can get smaller third party VRM heat sinks, but that's another thing to deal with. The CPU cooler options are limited (the Dynatron A42 works, but it's loud when the CPU draws a lot of power). 40mm case fans are also quite loud to move the required airflow. You can buy noctuas or whatever, but they won't really keep it cool. The ones that actually do spin very fast and so are very loud. You must have noticed this too, although maybe you have a spot for the machine where you don't hear the noise all the time.
I'm trying 2U now. I bought and am currently setting up an Innovision AS252-A06 chassis: 8 3.5" hot swap bays, 2U, 520mm depth. (Of course you can have a lot more drives if you go to 2.5" drives, give up hot swap, and/or have room for a deeper chassis.) Less worry about if stuff will fit, more room for airflow without noise.
Nah buy the right enterprise gear instead
https://www.supermicro.com/en/products/motherboard/A2SDi-H-T...
I'd really dig a version of this with a Ryzen AI chip and 128gb of ram.
I'm moving to Lenovo tiny m75q series for now due to low idle power and heat generated.
How much TDP and does it have 8+ sata with 10Gbe?
If you want 100TB, you need a bigger NAS than most, and that makes most of the DIY NAS not so good. 2-4 drives seems to be where DIY shines. These days motherboards often stop at 4x sata, so you'll need a HBA or USB (eww).
Personally, I just don't have that much data, 24TB mirrored for important data is probably enough, and I have my old mirror set avaialable for media like recorded tv and maybe dvds and blu-rays if I can figure out a way to play them that I like better than just putting the discs in the machine.
I admire the courage to store data on refurbished Seagate hard drives. I prefer SSD storage with some backups using cloud cold storage, because I’m not the one replacing the failing hard drives.
I would also prefer having a large number of high capacity SSDs so I could replace my spinning hard drives.
But even the cheapest high capacity SSD deals are still a lot more expensive than hard drive array.
I’ll continue replacing failing hard drives for a few more years. For me that has meant zero replacements over a decade, though I planned for a 5% annual failure rate and have a spare drive in the case ready to go. I could replace a failed drive from the array in the time takes to shut down, swap a cable to the spare drive, and boot up again.
SSDs also need to be examined for power loss protection. The results with consumer drives are mixed and it’s hard to find good info about how common drives behave. Getting enterprise grade drives with guaranteed PLP from large on-onboard capacitors is ideal, but those are expensive. Spinning hard drives have the benefit of using their rotational inertia to power the drive long enough to finish outstanding writes.
You can find cheap used enterprise SSDs on ebay. But the problem is that even the most power efficient enterprise SSD (SATA) idle at like 1w. And given the smaller capacities, you need many more to match a hard drive. In the end HDD might actually consume less power than an all flash array + controllers if you need a large capacity.
Curious, what's the use case for wanting your data backed-up without fail? Is it personal archives or otherwise (business) archive related?
Not to say you shouldn't backup your data, but personally I wouldn't be to affected if one of my personal drives errored out, especially if they contained unused personal files from 10+ years ago (legal/tax/financials are another matter).
Any data I created, paid to license, or put in significant work to gather has to be backed-up with 3-2-1 rule. Stuff I can download or otherwise obtain again is best effort but not mandatory backup.
Mainly I don't want to lose anything that took work to make or get. Personal photos, videos, source code, documents, and correspondence are the highest priority.
RAID. Preferably RAID 6. Much, much better to build a system to survive failure than to prevent failure.
Don't RAID these days. Software won rather drastically, likely because CPUs are finally powerful enough to run all those calculations without much of a hassle.
Software solutions like Windows Storage Spaces, ZFS, XFS, unRAID, etc. etc are "just better" than traditional RAID.
Yes, focus on 2x parity drive solutions, such as ZFS's "raidz2", or other such "equivalent to RAID6" systems. But just focus on software solutions that more easily allow you to move hard drives around without tying them to motherboard-slots or other such hardware issues.
> Don't RAID these days. Software won rather drastically
RAID does not mean or imply hardware RAID controllers, which you seem to incorrectly assume.
Software RAID is still 100% RAID.
FYI XFS is not redundant, also RAID usually refers to software RAID these days.
I like btrfs for this purpose since it's extremely easy to setup over cli, but any of the other options mentioned will work.
btrfs RAID is quite infamous for eating your data. Has it been fixed recently?
No. RAID 5/6 is still fundamentally broken and probably won't get fixed
I believe RAID5/6 is still experimental (although I believe the main issues were worked out in early 2024), I've seen reports of large arrays being stable since then. It's still recommended to run metadata in raid1/raid1c3.
RAID0/1/10 has been stable for a while.
I have a dozen refurbished exos disk in my storage machine. Works super! SSD for bigger storage is simply too expensive
Might be a bit adventurous for primary storage (though with enough backup and redundancy, why not). But seems perfect for me for backup / cold storage.
And I prefer to have a healthy bank account balance.
Storing 18TB (let alone with raid) on SSDs is something only those earning Silicon Valley tech wages can afford.
Not really. I know that my sleep is worth more than the difference between HDD and SSD prices, and I know the difference between the failure rates and the headache caused by the RMA process, so I buy SSDs.
In essence, what we together are saying is that people with super-sensitive sleep that are also easily upset, and that don't have ultra-high salaries, cannot really afford 18 TB of data (even though they can afford an HDD), and that's true.
Well, again, well done on being able to afford it. I have 24TB array on cheap second hand drives from CEX for about £100 each, using DrivePool - and guess what, if one of them dies I'll just buy another £100 second hand drive. But also guess what - in the 6 years I had this setup, all of these are still in good condition. Paying for SSDs upfront would have been a gigantic financial mistake(imho).
Every drive is "used" the moment you turn it on.
There's a big difference between used as in I just bought this hard drive and have used it for a week in my home server, and used as in refurbished drive after years of hard labor in someone else's server farm
Enterprise drives are way different than anything consumer based. I wouldn't trust a consumer drive used for 2 years, but a true enteprise drive has like millions of hours left of it's life.
Quote from Toshiba's paper on this. [1]
Hard disk drives for enterprise server and storage usage (Enterprise Performance and Enterprise Capacity Drives) have MTTF of up to 2 million hours, at 5 years warranty, 24/7 operation. Operational temperature range is limited, as the temperature in datacenters is carefully controlled. These drives are rated for a workload of 550TB/year, which translates into a continuous data transfer rate of 17.5 Mbyte/s[3]. In contrast, desktop HDDs are designed for lower workloads and are not rated or qualified for 24/7 continuous operation.
From Synology
With support for 550 TB/year workloads1 and rated for a 2.5 million hours mean time to failure (MTTF), HAS5300 SAS drives are built to deliver consistent and class-leading performance in the most intense environments. Persistent write cache technology further helps ensure data integrity for your mission-critical applications.
[1] https://toshiba.semicon-storage.com/content/dam/toshiba-ss-v...
[2] https://www.synology.com/en-us/company/news/article/HAS5300/...
There isn’t a significant difference between “enterprise” and “consumer” in terms of fundamental characteristics. They have different firmware and warranties, usually disks are tested more methodically.
Max operating range is ~60C for spinning disks and ~70C for SSD. Optimal is <40-45C. The larger agents facilties afaik tend to run as hot as they can.
> drive has like millions of hours left of it's life.
It doesn't apply for the single drive, only for a large number of drives. E.g. if you have 100000 drives (2.4 million hours MTTF) in a server building with the required environmental conditions and maximum workload, be prepared to replace a drive once a day in average.
datablocks.dev has a page explaining what white label and recertified disks are [1]. Those are not disks used for years under heavy load.
1: https://datablocks.dev/blogs/news/white-label-vs-recertified...
Drive failure rate versus age is a U-shaped curve. I wouldn't distrust a used drive with healthy performance and SMART parameters.
And you should use some form of redundancy/backups anyway. It's also a good idea to not use all disks from the same batch to avoid correlated failures.
Returns are known bads.
What exactly are these "white label drives"? Aren't these just normal seagate exos drives with SMART information wiped and labels removed? i.e. just a worse used drive.
Trying to think of reasons why the manufacturer wouldn't want their name on them and none of them are good. And for not even much of a discount.
Weren't shucked drives (removed from enclosures) referred to as White label drives at one point?
I was hoping for a full text dump of the SMART data from the drives.
I never understood why they let Seagate et al do this game about hard drives. If they offer a warranty, then replace the drive to brand new, and shove the recertified, fixed whatever bullshit up your wahzoo.
These drives are very likely refurbs that are unofficial.
White labeling avoids lawsuits.
OT
> Half of tech YouTube has been sponsored by companies like...
It just struck me that the product reviews are a part of the social realm that is barely explored.
Imagine a video website like TikTok or YouTube etc where all videos are organized under products. Priority to those who purchased the product and a category ranked by how many similar products you've purchased.
The thing sort of exists currently in some hard to find corner of TEMU etc but there are no channels or playlists.
The reason you don’t see videos arranged by product is because everyone knows not to trust unknown creators telling you how great a product is.
Viewers want to see opinions from specific people they’ve come to trust, not the first video that comes up for a product.
They don't have to tell you anything. Just unbox and show what they got.
I just purchased a bicycle chain cleaning device. It was absurdly cheap. The plastic was extruded poorly, it was hard to assemble, it was not entirely obvious how to use it. However! It did the job and it barely got dirty. I expected it to be full of rusty oil both inside and outside but it accumulated just a tiny smudge on the inlet. If anyone made a video it would be a fantastic product.
God, the flood of absolutely useless "review" videos Amazon has incentivized customers to shit all over their site which are nothing more than unboxings are the worst thing about that ecosystem. No thank you.
1. You could be that anyone.
2. The world is filled to the brim with videos about "fantastic products".
Alternatively, unknown creators have less incentive to falsely promote or lie. It’s the reason I tend to trust random strangers on Reddit than popular YouTubers who have achieved monetization and sponsorship.
I don’t trust big channels especially, because I assume they have just sold themselves out to the biggest sponsor. Influencers only exist due to campaign deals, where companies try to sneak their ads into your mind by abusing your inclination to trust another human being. All of it is sickening.
In comparison, I’d rather read a general review magazine with a long history. At least they don’t try to trick me into believing they are working out of the goodness of their hearts, and they usually aren’t married to a single big sponsor.
Online reviews are broken beyond repair.
>I’d rather read a general review magazine with a long history.
Do any of these still exist?
Coincidentally or not, those folks who have more subscribers usually charge more for their consideration. That’s why I generally trust Steve of Gamers Nexus more than other folks, because they don’t do ads except for promoting their own products, so there’s no conflict of interest. On the one hand, Gamers Nexus doesn’t manufacture their own hard drives, but on the other, they publish their methodology and have a reputation to uphold, so I would trust their judgement regarding testing computer hardware more than folks who do engage in outside advertising.
There's Kakaku.com[1] in Japanese Internet for all consumer electronics, Minkara for cars, bookmeter.com for books, and 5ch.net as fallback. It's surprising that there's only Goodreads on English Internet that everyone have heard of...
1: https://review.kakaku.com/review/K0001682323/ | https://review-kakaku-com.translate.goog/review/K0001682323/...