Over the past few weeks, the host I’ve been with for over 3 years, OVH, announced a rather large price increase of 20% because of Brexit – the current universal excuse to squeeze the customer for more cash. This change has sent the price of my dedicated server solution with them to over £45 a month. Doing some napkin-calculation gave me £18 a month in extra power to run a small server locally. So I’ve decided to bring the hosting solution back to my local network & run from my domestic internet link, which at 200Mbit/s DL & 20Mbit/s UL should be plenty fast enough to handle the modest levels of traffic I usually get.
Obviously, some hardware was required for this, so I obtained this beauty cheap on eBay:
This is a Gen 8 HP Proliant Microserver, very small & quiet, perfect for the job. This came with 4GB of RAM installed from the factory, and a Celeron G1610T running at 2.3GHz. Both are a little limited, so some upgrades will be made to the system.
4 SATA drive bays are located behind the magnetically-locked front door, there’s a 250GB boot disk in here along with a pair of 500GB disks in RAID1 to handle the website files & databases. For my online file hosting site, the server has a backend NFS link direct to Volantis – my 28TB storage server. This arrangement keeps the large file storage side of things off the web server disks & on a NAS, where it should be.
First thing is a RAM upgrade to the full supported capacity of 16GB. This being a Proliant server machine, doesn’t take anything of a standard flavour, it’s requirements are DDR3-10600E or DDR3-12800E (the E in here being ECC). This memory is both eye-wateringly expensive & difficult to find anywhere in stock. It’s much cheaper & easier to find the ECC Registered variety, but alas this isn’t compatible.
Over the past 48 hours or so, I’ve been migrating everything over to the new baby server, with a couple of associated teething problems, but everything seems to have gone well so far. The remaining job to get everything running as it should is an external mail relay – sending any kind of email from a dynamic IP / domestic ISP usually gets it spam binned by the big providers instantly, regardless of it actually being spam or not – more to come on that setup & configuring postfix to use an external SMTP relay server soon!
If anyone does find something weird going on with the blog, do let me know via the contact page or comments!
For some time now I’ve been running a large disk array to store all the essential data for my network. The current setup has 10x 4TB disks in a RAID6 array under Linux MD.
Up until now the disks have been running in external Orico 9558U3 USB3 drive bays, through a PCIe x1 USB3 controller. However in this configuration there have been a few issues:
Congestion over the USB3 link. RAID rebuild speeds were severely limited to ~20MB/s in the event of a failure. General data transfer was equally as slow.
Drive dock general reliability. The drive bays are running a USB3 – SATA controller with a port expander, a single drive failure would cause the controller to reset all disks on it’s bus. Instead of losing a single disk in the array, 5 would disappear at the same time.
Cooling. The factory fitted fans in these bays are total crap – and very difficult to get at to change. A fan failure quickly allows the disks to heat up to temperatures that would cause failure.
Upgrade options difficult. These bays are pretty expensive for what they are, and adding more disks to the USB3 bus would likely strangle the bandwidth even further.
Disk failure difficult to locate. The USB3 interface doesn’t pass on the disk serial number to the host OS, so working out which disk has actually failed is difficult.
To remedy these issues, a proper SATA controller solution was required. Proper hardware RAID controllers are incredibly expensive, so they’re out of the question, and since I’m already using Linux MD RAID, I didn’t need a hardware controller anyway.
A quick search for suitable HBA cards showed me the IOCrest 16-port SATAIII controller, which is pretty low cost at £140. This card breaks out the SATA ports into standard SFF-8086 connectors, with 4 ports on each. Importantly the cables to convert from these server-grade connectors to standard SATA are supplied, as they’re pretty expensive on their own (£25 each).
This card gives me the option to expand the array to 16 disks eventually, although the active array will probably be kept at 14 disks with 2 hot spares, this will give a total capacity of 48TB.
Here’s the card installed in the host machine, with the array running. One thing I didn’t expect was the card to be crusted with activity LEDs. There appears to be one LED for each pair of disks, plus a couple others which I would expect are activity on the backhaul link to PCIe. (I can’t be certain, as there isn’t any proper documentation anywhere for this card. It certainly didn’t come with any ;)).
I’m not too impressed with the fan that’s on the card – it’s a crap sleeve bearing type, so I’ll be keeping a close eye on this for failure & will replace with a high quality ball-bearing fan when it finally croaks. The heatsink is definitely oversized for the job, with nothing installed above the card barely gets warm, which is definitely a good thing for life expectancy.
Update 10/02/17 – The stock fan is now dead as a doornail after only 4 months of continuous operation. Replaced with a high quality ball-bearing 80mm Delta fan to keep things running cool. As there is no speed sense line on the stock fan, the only way to tell it was failing was by the horrendous screeching noise of the failing bearings.
Above is the final HBA installed in the PCIe x1 slot above – a parallel SCSI U320 card that handles the tape backup drives. This card is very close to the cooling fan of the SATA card, and does make it run warmer, but not excessively warm. Unfortunately the card is too long for the other PCIe socket – it fouls on the DIMM slots.
The tape drives are LTO2 300/600GB for large file backup & DDS4 20/40GB DAT for smaller stuff. These were had cheap on eBay, with a load of tapes. Newer LTO drives aren’t an option due to cost.
The main disk array is currently built as 9 disks in service with a single hot spare, in case of disk failure, this gives a total size after parity of 28TB:
Linux MD Detail
Creation Time:Wed Mar1116:01:012015
Used Dev Size:3906887360(3725.90GiB4000.65GB)
Update Time:Mon Nov1414:28:592016
Number Major Minor RaidDevice State
The disks used are Seagate ST4000DM000 Desktop HDDs, which at this point have ~15K hours on them, and show no signs of impending failure.
Here’s a screenshot with the disk array fully loaded running over USB3. The aggregate speed on the md0 device is only 21795KB/s. Extremely slow indeed.
This card is structured similarly to the external USB3 bays – a PCI Express bridge glues 4 Marvell 9215 4-port SATA controllers into a single x8 card. Bus contention may become an issue with all 16 ports used, but as far with 9 active devices, the performance increase is impressive. Adding another disk to the active array would certainly give everything a workout, as rebuilding with an extra disk will hammer both read from the existing disks & will write to the new.
With all disks on the new controller, I’m sustaining read speeds of 180MB/s. (Pulling data off over the network). Write speeds are always going to be pretty pathetic with RAID6, as parity calculations have to be done. With Linux MD, this is done by the host CPU, which is currently a Core2Duo E7500 at 2.96GHz, with this setup, I get 40-60MB/s writes to the array with large files.
Since I don’t have a suitable case with built in drive bays, (again, they’re expensive), I’ve had to improvise with some steel strip to hold the disks in a stack. 3 DC-DC converters provides the regulated 12v & 5v for the disks from the main unregulated 12v system supply. Both the host system & the disks run from my central battery-backed 12v system, which acts like a large UPS for this.
The SATA power splitters were custom made, the connectors are Molex 67926-0001 IDC SATA power connectors, with 18AWG cable to provide the power to 4 disks in a string.
These require the use of a special tool if you value your sanity, which is a bit on the expensive side at £25+VAT, but doing it without is very difficult. You get a very well made tool for the price though, the handle is anodised aluminium & the tool head itself is a 300 series stainless steel.
To do some upgrades to my NAS, I needed some SATA power adaptors, to split the PSU out to the planned 16 disk drives. eBay has these for very little money, however there’s a good reason for them being cheap.
The marking on the wire tells me it’s 18AWG, which should be good for 9.5A at an absolute maximum. However these adaptors are extremely light.
Here’s the cheapo eBay wire compared to proper 18AWG wire. The cores in the eBay adaptor are tiny, I’d guess about 24AWG, only good for about 3A. As disk drives pull about 2A from the +12v rail on startup to spin the platters up to speed, this thin wire is going to cause quite the volt drop & possibly prevent the disk from operating correctly.