StoneFly i4000

By Jack Fegreus
published: Wednesday, February 20 2008

iSCSI Concentration Advances System Virtualization

Maximizing the effectiveness of multiple ESX servers, a SAN for storage resources is necessary; moreover, the advanced features of VMFS, make iSCSI over an Ethernet LAN an ideal way to share storage resources and simplify storage management.

Via StoneFusion , a specialized OS built on the Linux kernel, the StoneFly iSCSI Storage Concentrator integrates the power of an iSCSI router with extensive storage management services. As a result, this StoneFly appliance presents IT with an exceptional mechanism for extending the benefits of any existing Fibre Channel SAN to a much broader base of clients. Not the least of these extended clients are virtual machines (VMs) running VMware Virtual Infrastructure (VI).

IT can quickly install a Stonefly Storage Concentrator utilizing existing Ethernet and FC infrastructure and then leverage the concentrator's storage-provisioning engine to provide advanced storage management, business continuity, and disaster recovery functions. In particular, StoneFusion provides robust storage virtualization, synchronous and asynchronous mirroring, snapshots, and active/active clustering of concentrators.

With regard to IT management costs, system and storage virtualization together share the spotlight in a 2006 McKinsey survey of senior IT executives. The biggest driver of IT costs is not the acquisition of resources, but rather the management of those resources. The general rule of thumb is that operating costs for managing storage on a per-gigabyte basis are three to ten times greater than the capital costs of storage acquisition.

What makes virtualization a top-of-mind proposition for CIOs today is the ability of virtual devices to be isolated from the constraints of physical limitations. By separating function from physical implementation, IT can manage that resource as a generic device based on its function. That means system administrators can narrow their operations focus from a plethora of proprietary devices to a limited number of generic resource pools. As a result, SAN technology has long been regarded as the premier means of consolidating storage resources and streamlining management in large data centers.

What's more, storage virtualization in a VI environment is a much simpler proposition when compared to storage virtualization in an environment with traditional physical servers. That’s because commercial operating systems, such as Microsoft Windows and Linux, assume exclusive ownership of storage volumes. On the other hand, the file system for VMware ESX, dubbed VMFS, eliminates the need for exclusive volume ownership by handling distributed file locking between systems.

VMFS has a built-in distributed locking mechanism (DLM) that avoids the massive overhead that a DLM typically imposes: VMFS simple treats each disk volume as a single-file image in a way that is loosely analogous to an ISO-formatted CDROM. When a VM’s OS mounts a disk, it opens a disk-image file; VMFS locks that file; and the VM's OS gains exclusive ownership of the disk volume with a plethora of files.

That opens the door to using iSCSI to extend the benefits of physical and functional separation via a cost-effective lightweight SAN. That’s all the more important given that storage virtualization is a necessary prerequisite to derive the full set of benefits from system virtualization in a VI environment,. For both a VM and its data, the issues of availability and mobility play an important role in daily operational tasks and long-term strategic plans.

For cost-conscious IT decision makers, the StoneFly Storage Concentrator adds another important advantage: the ability to cut operating costs. Via the StoneFusion storage virtualization engine, system administrators are guided through such provisioning and storage management tasks as the creation of volume copies and snapshots and even the configuration of synchronous and asynchronous mirrors. As a result, a system administrator servicing an iSCSI client can directly handle the labor-intensive storage management tasks that would normally require coordination with a storage administrator.

To assess the Stonefly Storage Concentrator i4000, VSM Labs set up two test scenarios. In the initial scenario, we concentrated on determining performance parameters for traditional physical servers. In this scenario, we ran Windows Server 2003 SP2 and Novel SUSE® Linux Enterprise Server (SLES) 10 SP1 on an HP Proliant ML350 G3 server. This server sported a 2.4GHz Xeon processor, 2GB of RAM, and an embedded Gigabit Ethernet TOE. We also installed a QLogic 4050 hardware iSCSI HBA.

In our second scenario, we used our initial test results for the HP ProLiant ML 350 as a template for server consolidation. Utilizing two 4-way servers running ESX 3.0.1, VSM Labs tested iSCSI performance on an ESX host server in support of VM datastores that were hosting virtual work volumes. These tests were done in the context of replacing an HP Proliant ML350 G3 server with a VM. In addition, we tested the volume copy and advanced image management functionality of StoneFusion in our VI environment. Those tests, we designed to assess StoneFusion as a means of enhancing the distribution of VM operating systems from templates and bolstering business continuity for disaster recovery.

Along with our Stonefly i4000 iSCSI Storage Concentrator on the iSCSI side of our SAN fabric, we employed a NETGEAR level 3 managed Gigabit Ethernet switch and several QLogic 4050 iSCSI HBAs. By using the QLogic iSCSI HBA, we were able to maximize throughput from the StoneFly i4000 by eliminating all overhead associated with iSCSI packet processing.

On the Fibre Channel side of our fabric, we utilized a QLogic SANbox 9200 switch, an nStor 4540 storage array, and an IBM DS4100 storage array. We chose the IBM TotalStorage DS4100 as the primary array for providing backend storage for two reasons: its large storage capacity and its robust I/O caching capability. Using low-cost high-capacity SATA drives, we were able to configure our IBM DS4100 array with 3.2TB of storage: From that pool, we assigned 1.6TB to the StoneFly i4000 in bulk via a single LUN.

For our tests, rapid response to excessively high numbers of I/O operations per second (IOPS) would trump capacity. That's because our oblLoad benchmark generates high numbers of IOPS to stress all components of a SAN fabric. With respect to our analysis, the IBM DS4100 provides an excellent balance of capacity with I/O responsiveness through two independent controllers, each of which features a highly configurable 1GB cache and dual 2Gbit FC ports.

By performing all partitioning and management functions for virtual storage volumes on the iSCSI concentrator instead of the FC array, VSM Labs was able to leverage key capabilities of StoneFusion to reduce operating costs by enabling system administrators to carry out tasks that normally require co-ordination with a storage administrator. In particular, we were able to consolidate storage from multiple FC arrays into a pool that could be managed from the StoneFly i4000. More importantly, we were able to configure logical volumes—dubbed resource targets in the iSCSI vernacular—and export them to client systems without any regard for the sources of the blocks within the pool. Nonetheless, to maintain consistency in benchmark performance—highly dependent on the disk drive characteristics, controller caching, and RAID configuration—VSM Labs created all volumes that would be used for performance benchmarking explicitly with disk blocks imported via the 1.6TB LUN from the DS4100 array.

We began testing on an HP Proliant ML350 G3 server running Windows Server 2003 and using both the Qlogic iSCSI HBA and the Microsoft software iSCSI initiator. Like the Microsoft initiator, the Qlogic iSCSI HBA supports iSNS, so it too will discover the StoneFly i4000 automatically. What's more, the QLogic iSCSI HBA off loads all iSCSI packet processing—a TOE only off loads the processing of the TCP packets that encapsulate the SCSI command packets—and thereby provides a distinct edge in processing IOPS. This is very significant for maximizing performance of the StoneFly i4000, which was able to sustain a load of 10,000 IOPS with 8KB data requests.

The IOPS throughput patterns for oblLoad using the QLogic HBA and the server's embedded TOE with the Microsoft initiator were remarkably similar; however, absolute performance measured in total IOPS was distinctly higher for the QLogic iSCSI HBA. This was especially true for small numbers of daemons, which coincides with the time when the host is most sensitive to changes in overhead. With more than 12 daemons, the difference in the number of IOPS completed varied by less than 2%.

A very different picture surfaced on Linux for transaction processing. IOPS performance for iSCSI on SLES10—even with the QLogic iSCSI HBA—trailed iSCSI performance on Windows Server 2003 by an order of magnitude. This is a function of the way Linux bundles I/O and has nothing to do with the StoneFly i4000. It is, however, a condition that the StoneFly i4000 can exploit. On the other hand, the StoneFusion OS is tuned for high data throughput. As a result, when we ran oblLoad with 64KB I/O requests—used in multi-dimensional business intelligence application scenarios—we measured the same level of IOPS while moving 8 times more data.

That ability to deliver high data throughput levels is particularly important in supporting high-end multimedia applications, especially when dealing with streaming video. Both Linux and Windows client systems were able to stream large multi-gigabyte files sequentially at wire—1Gbps—speed through the StoneFly i4000.

In the final phase of testing of the StoneFly i4000, VSM Labs utilized two quad-processor servers to run a VMware Infrastructure 3 environment. This advanced third-generation platform virtualizes an entire IT infrastructure including servers, storage, and networks. For the VSM Labs test scenario, we focused our attention on the problem of consolidating four servers along the lines of our HP ProLiant ML350 G3 system on each of our 4-way servers.

The primary means by which VMware ESX Server provides access to virtual storage volumes is by encapsulating a VM disk in a VMFS datastore in a way that is analogous to a CD-ROM image file. The VM disk is a single large VMFS file that is presented to the VM's OS as a SCSI disk drive, which contains a file system with many individual files. The OS of the VM issues I/O commands to what appears to be a local SCSI drive connected to a local SCSI controller. In practice, the block read/write requests are passed to the VMkernel where a physical device driver, such as the driver for the QLogic iSCSI HBA, forwards the read/write requests and directs them to the actual physical hardware device.

With a DLM, a datastore can contain multiple VM disk files that are accessed by multiple ESX Servers. That scheme can put I/O loads on a VMFS volume that are significantly higher than the loads on a disk volume in a single-host, single-operating-system environment. To meet those loads, VMFS has been tuned as a high-performance file system for storing large, monolithic virtual disk files. Tuning an array for a particular application becomes irrelevant when using a VM disk file.

The alternative to using VMFS is to use a raw LUN formatted with a native file system associated with the virtual machine (VM). Using a raw device as though it were a VMFS-hosted file requires a VMFS-hosted pointer file to redirect I/O requests from a VMFS volume to the raw LUN. This scheme is dubbed Raw Device Mapping (RDM). What drives the use of an RDM scenario is the need to share data with external physical machines.

While VSM Labs ran functionality tests of RDM volumes, we chose to utilize unique VMware datastores to encapsulate single virtual volumes in our benchmark tests. Given that the default block size for VMFS is 1MB, we followed two fundamental rules of thumb in provisioning backend storage for the StoneFly i4000:

1. Put as many spindles into the underlying FC array as possible.
2. Make the FC array's stripe size as large as possible.

In particular, we utilized 7-drive arrays with a stripe size of 256KB—the default for high-end UNIX systems—in the IBM DS4100 storage system. With our storage system sporting two independent disk controllers with a 1-GB cache, we garnered a significant boost in our IOPS performance tests by exploiting read-ahead track caching.

We began testing iSCSI performance on a VMware ESX Server with virtual machines running Windows Server 2003 SP2. With a 50GB datastore mounted via the QLogic HBA, the number of IOPS completed by oblLoad was virtually identical to the number completed on our base HP Proliant ML350 server system running Windows Server 2003 SP2. Without the iSCSI HBA, peak IOPS performance fell by about 40%.

The most extraordinary results, however, occurred when we ran SUSE Linux Enterprise Server (SLES) 10 SP1 within a VM. In this case, IOPS performance improved with both the QLogic iSCSI HBA and with the VMware iSCSI initiator, as compared to running a physical server. With a VM running SLES, however, IOPS performance was propelled well beyond what we had measured with a physical machine. While the basic pattern for IOPS throughput remained the same, the net performance result was a throughput level that was often on a scale showing an absolute increase in performance on the order of 200-to-250% higher for any given number of oblLoad disk daemons.

While the StoneFly i4000 provided exceptional performance, it was the added provisioning features of StoneFusion that made the biggest impact in managing a VI environment. Since the prime goal of systems virtualization is to maximize resource utilization, multiple VMs will be running on a host server at any instance in time. To avoid the overhead of installing multiple instances of an OS, VMware supports the concept of creating an OS installation template and then cloning that template the next time that the OS is to be installed.

In a VI environment, the creation of templates is handled by the VMware Virtual Center software, which requires a separate system running Windows Server along with a commercial database, such as SQL Server or Oracle, to keep track of all disk images. Similar functionality can be leveraged using the StoneFly i4000 Storage Concentrator through the StoneFusion image management functions for volumes. While best practices call for maintaining offline template volumes for this task, we were able to use any volume at any time, provided that we were able take that volume offline.

To clone a volume image, we first needed to shutdown all VMs running on that virtual volume and close any iSCSI sessions that were open for that volume with any ESX servers. Once this was done, we could begin the rather simple process of adding a mirror image to the volume, which is normally done to provide for high availability in either a disaster/recovery or a backup scenario.

The creation of a mirror is a remarkably fast and efficient process under StoneFusion. We monitored the FC switch port that was connected to the StoneFly i4000 during the process. Read and write data throughput remained fully synchronized during the process—each at a pace of 45MB per second. Att that rate, the process of generating an OS clone complete with any additional software applications was merely a matter of minutes.

We could then add the cloned VM to the pool of virtual machines on each ESX server. On powering on the new VM for the first time, the ESX server would recognize that this VM had an existing identifier and would request confirmation that it should either retain or create a new ID for this VM. Once that was completed, we were done with the process of creating a new VM.