A while back I installed a few 10GB nics running over CAT6, and have enjoyed the higher throughput to my FreeNAS box.
The FreeNAS box is running a bunch of spinners in RAIDZ, so while the sequential read/write have been pretty good, the 4k read/write have been pretty lousy. For the longest time I didn’t know if the low 4k file speed limitation was due to the fact that the array is based on spinning disks or because the CAT6 protocol has too much overhead and limits the throughput of small files.

Since I have multiple workstations connected to the NAS, what I wanted was the ability to move the SSD storage off the workstations and have it all centralized. No more copying projects to local storage for maximum access speed for data processing.

The RJ45 10GB solution maxes out the HDD array in sequential read/writes at about 350 MB/s, which makes sense with 5 x 10 TB spinning disks. While that’s not SSD speed, its better then 1GB connectivity. However copying small files has always sucked.

The plan was to buy a bunch of SSDs and throw them in a RAIDz array but i was not sure if my network would be able to take full advantage of the SSD speeds in 4k access. I had read that DAC connections have lower latency, but didn’t have any on hand to try out, until now.

Today I installed a couple Mellanox ConnectX-2 cards to test out the speed over DAC connections.

WOW what an improvement! 

I also tested the speed of local SSD and nvme storage to have a base line.

The test was done connecting one of the workstations to a server through 1GB CAT5, 10GB CAT6 (ASUS XG-C100C) and 10GB SPF+ DAC (Mellanox ConnectX-2). On the server side, I used a RAM disk to eliminate any bottlenecks, and test purely network speeds.

There’s a few more tests to run, but it looks like the RJ45 (CAT6) connection for 10GB connectivity is lacking. And while the DAC connection is 4 times faster for 4k read/writes than the RJ45 connection, I find it strange that the 1GB connection can put out double the 4k throughput.

I also find it strange that the local SSDs have higher write speeds then read speeds with 4k file access.