Aug 24 2019 August 24, 2019 August 24, 2019 NoComment by Stefan

Deploy Azure Stack HCI on a two node cluster with nested resiliency using HPE servers

Posted in Azure Stack HCI, Storage Spaces Direct, Windows Server 2019

With Windows Server 2019, Storage Spaces Direct has been improved. It now supports two node clusters with nested resiliency that enables the cluster to withstand multiple hardware failures without the loss of storage. This is a great solution for small businesses that cannot yet move to the cloud but want to have the benefits of Windows Server 2019.

You can find more about Nested Resiliency here: https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/nested-resiliency

So what is Azure Stack HCI?

This is how Microsoft describes it:

‘Azure Stack HCI is a hyper-converged Windows Server 2019 cluster that uses validated hardware to run virtualized workloads on-premises. Azure Stack HCI solutions use Microsoft-validated hardware to ensure optimal performance and reliability, and include support for technologies such as NVMe drives, persistent memory, and remote-direct memory access (RDMA) networking.’

So basically it’s Storage Spaces Direct running on Windows Server 2019 Datacenter using Microsoft validated hardware.

The Hardware

Several hardware vendors have certified solutions for Azure Stack HCI. I recently did a deployment using HPE servers. HPE has published several sample BoM’s for All Flash and Hybrid setups. You can find them here: https://www.hpe.com/us/en/alliance/microsoft/azurestackhci.html

I used two HPE DL380 Gen10 servers. Each node has 6 x 1.9TB SSD drives for Storage Spaces Direct and two drives for the OS. Even though the new HPE controllers support a mixed mode of RAID and HBA mode it’s important to use a dedicated storage controller for Storage Spaces otherwise the solution is not supported. So I have added a separate controller for the OS drives and will use the internal controller for Storage Spaces.

Here’s the complete list of hardware per node:

  • 1 x 826566-B21 – HPE DL380 Gen10 5118 2P 64G 8SFF Svr
  • 2 x 815100-B21 – HPE 32GB 2Rx4 PC4-2666V-R Smart Kit
  • 1 x 826687-B21 – HPE DL38X Gen10 Premium 2SFF HDD Kit
  • 1 x 804394-B21 – HPE E208i-p SR Gen10 raid controller
  • 2 x 870753-B21 – 300GB 15K 12G SAS SFF HDD
  • 1 x 804338-B21 – HPE Smart Array P816i-a SR Gen10 Controller
  • 6x P07930-B21 – HPE 1.92TB SATA 6G MIXED USE SFF

Basic installation

So first we configured RAID1 for the OS disks and installed Windows Server 2019 Datacenter. Added the HPE drivers and firmware updates through SPP.

Rename the VM and Management network adapters and create a Team.

Rename-NetAdapter -Name "Embedded LOM 1 Port 1" -NewName "ManagementAdapter1"
Rename-NetAdapter -Name "Embedded LOM 1 Port 2" -NewName "ManagementAdapter2"
Rename-NetAdapter -Name "Embedded LOM 1 Port 3" -NewName "ManagementAdapter3"
Rename-NetAdapter -Name "Embedded LOM 1 Port 4" -NewName "ManagementAdapter4"
 
New-NetLbfoTeam -Name Management -TeamMembers ManagementAdapter1,ManagementAdapter2,ManagementAdapter3,ManagementAdapter4 -TeamingMode SwitchIndependent -LoadBalancingAlgorithm Dynamic

Set the management IP address.

New-NetIPAddress -InterfaceAlias "Management" -IPAddress "192.168.0.10" -PrefixLength "24" -DefaultGateway "192.168.0.1"
Set-DnsClientServerAddress -InterfaceAlias "Management" -ServerAddresses "192.168.0.100"

Rename the storage adapters and configure ip addresses.

Rename-NetAdapter -Name "LOM 1 Port 1" -NewName "Cluster1"
Rename-NetAdapter -Name "LOM 1 Port 2" -NewName "Cluster2"

New-NetIPAddress -InterfaceAlias "Cluster1" -IPAddress 10.10.10.10 -PrefixLength 24
Set-NetIPInterface -InterfaceAlias "Cluster1" -Dhcp Disabled
New-NetIPAddress -InterfaceAlias "Cluster2" -IPAddress 10.10.11.10 -PrefixLength 24
Set-NetIPInterface -InterfaceAlias "Cluster2" -Dhcp Disabled

Set the interface metrics.

Set-NetIPInterface -InterfaceAlias "Management" -InterfaceMetric 1
Set-NetIPInterface -InterfaceAlias Cluster1 -InterfaceMetric 2
Set-NetIPInterface -InterfaceAlias Cluster2 -InterfaceMetric 2

Network

Since this is a 2-node setup we won’t be using switches to connect the nodes. Instead we use direct attach cables using the 2-port 25Gb adapters. We will be using RMDA. The HPE 640FLR-SFP28 adapter is in fact a Mellanox adapter which uses RoCE so we need to configure DCB. I won’t go into the specifics of configuring DCB in this post but here’s the PowerShell code.

#Remove any existing configuration
Remove-NetQosPolicy -Confirm:$False
Remove-NetQosTrafficClass -Confirm:$False
Disable-NetQosFlowControl -Priority 0,1,2,3,4,5,6,7

#Disable DCBx
Set-NetQosDcbxSetting -InterfaceAlias Cluster1 –Willing $false -Confirm:$False
Set-NetQosDcbxSetting -InterfaceAlias Cluster2 –Willing $false -Confirm:$False

#Create QoS Policies
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

#Enable QoS
Enable-NetQosFlowControl -Priority 3,7

#Define QoS Traffic classes
New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 80 -Algorithm ETS 
New-NetQosTrafficClass "Cluster" -Priority 7 -BandwidthPercentage 1 -Algorithm ETS 

#Disable Flow Control
Set-NetAdapterAdvancedProperty -Name "Cluster1" -RegistryKeyword "*FlowControl" -RegistryValue 0
Set-NetAdapterAdvancedProperty -Name "Cluster2" -RegistryKeyword "*FlowControl" -RegistryValue 0 

#Enable QoS on the network adapters
Get-NetAdapterQos -Name "Cluster1","Cluster2" | Enable-NetAdapterQos

#Enable RDMA on the network adapters
Get-NetAdapterRdma Cluster1,Cluster2 | Enable-NetAdapterRDMA

Cluster Configuration

Install the necessary roles on both nodes.

# Install Roles/Features
$ServerList = "Server1", "Server2"
$FeatureList = "Hyper-V", "Failover-Clustering", "Data-Center-Bridging", "RSAT-Clustering-PowerShell", "Hyper-V-PowerShell", "FS-FileServer"

Invoke-Command ($ServerList) {
    Install-WindowsFeature -Name $Using:Featurelist
}

Perform Cluster validation, make sure all tests are green.

Test-Cluster –Node Server1, Server2 –Include "Storage Spaces Direct", "Inventory", "Network", "System Configuration"

Create the cluster.

New-Cluster –Name S2DCluster -Node Server1, Server2 –NoStorage –StaticAddress 192.168.0.11

Add cluster witness. We will be using a Cloud witness on Azure.

Set-ClusterQuorum -CloudWitness -AccountName <Storage Account Name> -AccessKey <Access Key>

Enable Storage Spaces Direct.

Enable-ClusterStorageSpacesDirect

Optionally configure CSV Cache size.

$ClusterName = "S2DCluster"
$CSVCacheSize = 8192 #Size in MB
(Get-Cluster $ClusterName).BlockCacheSize = $CSVCacheSize

Nested Resiliency

Since this is a 2-node setup we will be using Nested Resiliency so that it can sustain multiple hardware failures at the same time. There are two options for Nested Resiliency:

Nested two-way mirror. Two-way mirroring within the server and two-way mirroring between the servers.

Nested mirror-accelerated parity. Combines two-way mirroring with nested parity.

For this setup we chose nested two-way mirror to maximize performance and resiliency.

Define the nested two-way mirror storage tier.

New-StorageTier -StoragePoolFriendlyName S2D* -FriendlyName NestedMirror -ResiliencySettingName Mirror -MediaType SSD -NumberOfDataCopies 4

Create the volumes.

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Mirror1 -StorageTierFriendlyNames NestedMirror -StorageTierSizes 2TB
New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Mirror2 -StorageTierFriendlyNames NestedMirror -StorageTierSizes 2TB

Performance

To test the performance I created two CSVs one for each server. I deployed 10 Windows Server 2016 core VM’s divided across the nodes. I used VMFleet and DiskSpd to perform the following tests.

First test 566K IOPS / 0,187 Read Latency / 0,449 Write Latency

  • 10VM’s
  • 4k IO
  • 100% Read / 0% Writes
  • 10GB Test file
  • CSV Cache disabled

Second test 195K IOPS / 0,212 Read Latency / 0,263 Write Latency

  • 10VM’s
  • 4k IO
  • 70% Read / 30% Writes
  • 10GB Test file
  • CSV Cache disabled

As you can see there’s a clear drop in IOPS when doing 30% Writes. This is due to the fact that we have 4 copies using Nested two-way mirror.

Summary

When choosing Azure Stack HCI make sure you select validated hardware solutions from the vendors. Read all their documentation to prevent any problems. The new feature Nested Resiliency that comes with Windows Server 2019 is a great solution for small 2-node deployments. Usable space is less than with a normal two-way mirror and write IOPS are lower but you get better availability.