I did some exploration over the past few months on storage for my home
lab (This post is not about number crunching different setups).
already have a FreeNAS system doing mass storage (8T), mostly media. I
added an additional pool of 700G of Flash. I wanted to compare VMware
(ESXi) performance between NFS and iSCSI (I already have some ideas of
what performance is like for VMware and Fiber Channel from work). After
a little testing, and reading, I concluded that between VMware NFS doing
extra checks to ensure data integrity, and ZFS doing extra checks to
ensure data integrity, things would not speed up enough to be useful. I
was stuck around 1-200Mbit.
iSCSI, is it worth it?
I next turned to iSCSI. As a rule, I avoid iSCSI because the idea of
running a UNIX block device over a best-effort protocol is not a recipe
for success. On FreeNAS with ZFS, I can export a zvol of block space
over iSCSI and let VMware manage it completely. In theory, this would be
faster. It turns out that in this case at least, iSCSI was significantly
faster. I’ve seen speeds up to 850Mbit between ESXi and FreeNAS. The
FreeNAS has 2x 1G links configured for iSCSI, and the ESXi box has 2x 1G
links configured for iSCSI (I did this because I actually have 2 ESXi
boxes, and I didn’t want them to bottleneck on a single 1G link to
If you aren’t familiar with the iSCSI docs for both
VMware, and FreeNAS,
you should do a little reading.
Here are the Juniper config bits (based off JunOS 13.2):
firstname.lastname@example.org> show configuration interfaces ge-0/0/42 | display setset interfaces ge-0/0/42 description "FreeNAS Storage 1"set interfaces ge-0/0/42 mtu 9216set interfaces ge-0/0/42 unit 0 family ethernet-switching
email@example.com> show configuration interfaces ge-0/0/43 | display setset interfaces ge-0/0/43 description "FreeNAS Storage 1"set interfaces ge-0/0/43 mtu 9216set interfaces ge-0/0/43 unit 0 family ethernet-switching
firstname.lastname@example.org> show vlans storage1
Name Tag Interfaces
email@example.com> show vlans storage2
Name Tag Interfaces
Here are the screenshots showing the config of both the FreeNAS and ESXi
(through vCenter VCSA):
I’ve got 2 vmkernel interfaces for iSCSI configured, on different
networks, using jumbo frames. This gives me multi-pathing, for
additional throughput, in addition to making it fault-tolerant. Coupled
with using SSDs as the backing store gives me reasonable VM performance,
even when I need to do things like source code compiles.
cryptomonkeys.com by Louis Kowolowski is licensed under a Creative Commons International License. ·