Aug 24 2019 August 24, 2019 August 24, 2019 NoComment by Stefan

Deploy Azure Stack HCI on a two node cluster with nested resiliency using HPE servers

Posted in Azure Stack HCI, Storage Spaces Direct, Windows Server 2019

With Windows Server 2019, Storage Spaces Direct has been improved. It now supports two node clusters with nested resiliency that enables the cluster to withstand multiple hardware failures without the loss of storage. This is a great solution for small businesses that cannot yet move to the cloud but want to have the benefits of Windows Server 2019.

You can find more about Nested Resiliency here: https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/nested-resiliency

So what is Azure Stack HCI?

This is how Microsoft describes it:

‘Azure Stack HCI is a hyper-converged Windows Server 2019 cluster that uses validated hardware to run virtualized workloads on-premises. Azure Stack HCI solutions use Microsoft-validated hardware to ensure optimal performance and reliability, and include support for technologies such as NVMe drives, persistent memory, and remote-direct memory access (RDMA) networking.’

So basically it’s Storage Spaces Direct running on Windows Server 2019 Datacenter using Microsoft validated hardware.

The Hardware

Several hardware vendors have certified solutions for Azure Stack HCI. I recently did a deployment using HPE servers. HPE has published several sample BoM’s for All Flash and Hybrid setups. You can find them here: https://www.hpe.com/us/en/alliance/microsoft/azurestackhci.html

I used two HPE DL380 Gen10 servers. Each node has 6 x 1.9TB SSD drives for Storage Spaces Direct and two drives for the OS. Even though the new HPE controllers support a mixed mode of RAID and HBA mode it’s important to use a dedicated storage controller for Storage Spaces otherwise the solution is not supported. So I have added a separate controller for the OS drives and will use the internal controller for Storage Spaces.

Here’s the complete list of hardware per node:

  • 1 x 826566-B21 – HPE DL380 Gen10 5118 2P 64G 8SFF Svr
  • 2 x 815100-B21 – HPE 32GB 2Rx4 PC4-2666V-R Smart Kit
  • 1 x 826687-B21 – HPE DL38X Gen10 Premium 2SFF HDD Kit
  • 1 x 804394-B21 – HPE E208i-p SR Gen10 raid controller
  • 2 x 870753-B21 – 300GB 15K 12G SAS SFF HDD
  • 1 x 804338-B21 – HPE Smart Array P816i-a SR Gen10 Controller
  • 6x P07930-B21 – HPE 1.92TB SATA 6G MIXED USE SFF

Basic installation

So first we configured RAID1 for the OS disks and installed Windows Server 2019 Datacenter. Added the HPE drivers and firmware updates through SPP.

Rename the VM and Management network adapters and create a Team.

Rename-NetAdapter -Name "Embedded LOM 1 Port 1" -NewName "ManagementAdapter1"
Rename-NetAdapter -Name "Embedded LOM 1 Port 2" -NewName "ManagementAdapter2"
Rename-NetAdapter -Name "Embedded LOM 1 Port 3" -NewName "ManagementAdapter3"
Rename-NetAdapter -Name "Embedded LOM 1 Port 4" -NewName "ManagementAdapter4"
 
New-NetLbfoTeam -Name Management -TeamMembers ManagementAdapter1,ManagementAdapter2,ManagementAdapter3,ManagementAdapter4 -TeamingMode SwitchIndependent -LoadBalancingAlgorithm Dynamic

Set the management IP address.

New-NetIPAddress -InterfaceAlias "Management" -IPAddress "192.168.0.10" -PrefixLength "24" -DefaultGateway "192.168.0.1"
Set-DnsClientServerAddress -InterfaceAlias "Management" -ServerAddresses "192.168.0.100"

Rename the storage adapters and configure ip addresses.

Rename-NetAdapter -Name "LOM 1 Port 1" -NewName "Cluster1"
Rename-NetAdapter -Name "LOM 1 Port 2" -NewName "Cluster2"

New-NetIPAddress -InterfaceAlias "Cluster1" -IPAddress 10.10.10.10 -PrefixLength 24
Set-NetIPInterface -InterfaceAlias "Cluster1" -Dhcp Disabled
New-NetIPAddress -InterfaceAlias "Cluster2" -IPAddress 10.10.11.10 -PrefixLength 24
Set-NetIPInterface -InterfaceAlias "Cluster2" -Dhcp Disabled

Set the interface metrics.

Set-NetIPInterface -InterfaceAlias "Management" -InterfaceMetric 1
Set-NetIPInterface -InterfaceAlias Cluster1 -InterfaceMetric 2
Set-NetIPInterface -InterfaceAlias Cluster2 -InterfaceMetric 2

Network

Since this is a 2-node setup we won’t be using switches to connect the nodes. Instead we use direct attach cables using the 2-port 25Gb adapters. We will be using RMDA. The HPE 640FLR-SFP28 adapter is in fact a Mellanox adapter which uses RoCE so we need to configure DCB. I won’t go into the specifics of configuring DCB in this post but here’s the PowerShell code.

#Remove any existing configuration
Remove-NetQosPolicy -Confirm:$False
Remove-NetQosTrafficClass -Confirm:$False
Disable-NetQosFlowControl -Priority 0,1,2,3,4,5,6,7

#Disable DCBx
Set-NetQosDcbxSetting -InterfaceAlias Cluster1 –Willing $false -Confirm:$False
Set-NetQosDcbxSetting -InterfaceAlias Cluster2 –Willing $false -Confirm:$False

#Create QoS Policies
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

#Enable QoS
Enable-NetQosFlowControl -Priority 3,7

#Define QoS Traffic classes
New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 80 -Algorithm ETS 
New-NetQosTrafficClass "Cluster" -Priority 7 -BandwidthPercentage 1 -Algorithm ETS 

#Disable Flow Control
Set-NetAdapterAdvancedProperty -Name "Cluster1" -RegistryKeyword "*FlowControl" -RegistryValue 0
Set-NetAdapterAdvancedProperty -Name "Cluster2" -RegistryKeyword "*FlowControl" -RegistryValue 0 

#Enable QoS on the network adapters
Get-NetAdapterQos -Name "Cluster1","Cluster2" | Enable-NetAdapterQos

#Enable RDMA on the network adapters
Get-NetAdapterRdma Cluster1,Cluster2 | Enable-NetAdapterRDMA

Cluster Configuration

Install the necessary roles on both nodes.

# Install Roles/Features
$ServerList = "Server1", "Server2"
$FeatureList = "Hyper-V", "Failover-Clustering", "Data-Center-Bridging", "RSAT-Clustering-PowerShell", "Hyper-V-PowerShell", "FS-FileServer"

Invoke-Command ($ServerList) {
    Install-WindowsFeature -Name $Using:Featurelist
}

Perform Cluster validation, make sure all tests are green.

Test-Cluster –Node Server1, Server2 –Include "Storage Spaces Direct", "Inventory", "Network", "System Configuration"

Create the cluster.

New-Cluster –Name S2DCluster -Node Server1, Server2 –NoStorage –StaticAddress 192.168.0.11

Add cluster witness. We will be using a Cloud witness on Azure.

Set-ClusterQuorum -CloudWitness -AccountName <Storage Account Name> -AccessKey <Access Key>

Enable Storage Spaces Direct.

Enable-ClusterStorageSpacesDirect

Optionally configure CSV Cache size.

$ClusterName = "S2DCluster"
$CSVCacheSize = 8192 #Size in MB
(Get-Cluster $ClusterName).BlockCacheSize = $CSVCacheSize

Nested Resiliency

Since this is a 2-node setup we will be using Nested Resiliency so that it can sustain multiple hardware failures at the same time. There are two options for Nested Resiliency:

Nested two-way mirror. Two-way mirroring within the server and two-way mirroring between the servers.

Nested mirror-accelerated parity. Combines two-way mirroring with nested parity.

For this setup we chose nested two-way mirror to maximize performance and resiliency.

Define the nested two-way mirror storage tier.

New-StorageTier -StoragePoolFriendlyName S2D* -FriendlyName NestedMirror -ResiliencySettingName Mirror -MediaType SSD -NumberOfDataCopies 4

Create the volumes.

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Mirror1 -StorageTierFriendlyNames NestedMirror -StorageTierSizes 2TB
New-Volume -StoragePoolFriendlyName S2D* -FriendlyName Mirror2 -StorageTierFriendlyNames NestedMirror -StorageTierSizes 2TB

Performance

To test the performance I created two CSVs one for each server. I deployed 10 Windows Server 2016 core VM’s divided across the nodes. I used VMFleet and DiskSpd to perform the following tests.

First test 566K IOPS / 0,187 Read Latency / 0,449 Write Latency

  • 10VM’s
  • 4k IO
  • 100% Read / 0% Writes
  • 10GB Test file
  • CSV Cache disabled

Second test 195K IOPS / 0,212 Read Latency / 0,263 Write Latency

  • 10VM’s
  • 4k IO
  • 70% Read / 30% Writes
  • 10GB Test file
  • CSV Cache disabled

As you can see there’s a clear drop in IOPS when doing 30% Writes. This is due to the fact that we have 4 copies using Nested two-way mirror.

Summary

When choosing Azure Stack HCI make sure you select validated hardware solutions from the vendors. Read all their documentation to prevent any problems. The new feature Nested Resiliency that comes with Windows Server 2019 is a great solution for small 2-node deployments. Usable space is less than with a normal two-way mirror and write IOPS are lower but you get better availability.


Apr 21 2015 April 21, 2015 May 1, 2015 NoComment by Stefan

SCCM 2012 Client Push including cumulatieve updates

Posted in SCCM

We are using Client Push to deploy SCCM clients. We are also using the AllowedMPs registry setting to force the client to use a certain Management Point. This setting was introduced in CU3. However the client push always installs the RTM version which doesn’t have the AllowedMPs option. For more information on the AllowedMPs option read this blog: http://blogs.technet.com/b/jchalfant/archive/2014/09/22/management-point-affinity-added-in-configmgr-2012-r2-cu3.aspx

In our environment this resulted in clients that are installed but unable to communicate as it uses a random MP. After a lot of searching I found there’s an (perhaps unsupported) way of including updates in the push installation. And it’s very simple!

On the site server locatie the client folder for instance D:\Program Files\Microsoft Configuration Manager\Client. Create a folder called ClientPatch in the i386 and x64 folder. Copy the respective updates into the ClientPatch folders. That’s all there is to it. CCMSetup will install the RTM client and apply the hotfixes located in the ClientPatch folder. Like I said very simple!


Tags:

Apr 13 2015 April 13, 2015 May 1, 2015 2Comments by Stefan

How to update drive firmware when using Storage Spaces

Posted in Storage Spaces, Windows 2012 R2

I’m currently building a storage solution based on Windows Storage Spaces. I’m using Seagate Enterprise SAS and SSD drives. It’s best practise to install the latest firmware before configuring storage spaces. But what if you have a running setup and there’s a critical firmware update available? Storage Spaces hides the physical drives as the drives are part of the pool on which the virtual disks are created. It’s similar to a traditional RAID solution where accesss to the physical drive is not possible. With traditional RAID solutions however the vendor usually provides a method for upgrading drive firmware while the system is running. Storage Spaces doesn’t have this feature. Most likely because the underlaying hardware can be from different vendors making it hard to create this kind of feature in Storage Spaces. Of course there’s the option of bringing down the system and booting with a DOS or Linux distribution but what you really want is a way to accomplish this without any downtime. In this post I will explain how I did this with my setup using Seagate drives.

Read More

Tags:

Dec 28 2013 December 28, 2013 December 28, 2013 NoComment by Stefan

Making a serial port available on Hyper-V VMs

Posted in Hyper-V

I have been looking for a way to make a serial port available on one of my VMs running on Hyper-V. Hyper-V exposes serial ports as named pipes to the VM. I found this program PipeToCom that runs a service on the VM which creates as many named pipe serial ports as you want.

It can be downloaded here: https://github.com/albertjan/PipeToCom


Tags:

Nov 16 2013 November 16, 2013 November 16, 2013 15Comments by Stefan

Create Office 365 mailboxes while On Premise mailbox still exists

Posted in FIM, Office 365

There are situations in which you want to create a office 365 mailbox on a AD synchronized account that still has an on premises mailbox. For instance when using 3rd party migration tools that require the office 365 mailbox to exist. Normally when assigning a office 365 exchange license to a user a mailbox is automatically created. When this user still has an on premises mailbox office 365 will not create a mailbox. A mailbox will only be created when the microsoft migration tools are used. Office 365 checks the existence of the msExchMailboxGuid attribute to determine whether an on premise mailbox exists.

PERFORM THESE STEPS AT YOUR OWN RISK!

We can alter the directory sychronisation so that the msExchMailboxGuid is no longer synchronised to Office 365. This can be done by performing the following:

  1. On the computer thats running the directory synchronisation tool.
  2. Navigate to C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\UIShell
  3. Launch the application miisclient.exe
  4. Click on the management agents tab
  5. Open the properties of the Active Directory Connector
  6. Go to Configure Attribute Flow
  7. Expand Object Type: user in the Configure Attribute Flow section
  8. Scroll down to msExchMailboxGuid
  9. Click the Delete button to remove the mapping
  10. Click Ok to save the changes

If you already have synchronized objects in Office 365 these will still have the msExchMailboxGUID populated in Office 365. Since there is no way to use ADSIEdit on Office 365 objects we will have to do this with the directory synchronization tool.

The Azure Directory Synchronization tool is actually an installation of FIM (Forefront Identity Manager). I will not go into the specifics of how FIM works in this article but I will explain how to remove the msExchMailboxGUID value. FIM uses a metaverse to store the objects and attributes that are synchronized to Office 365. The metaverse is stored in a SQL database. We have removed the msExchMailboxGUID from the Active Directory Connector so it no longer populates the metaverse however for existing objects the value is still populated and will need to be removed. To do so follow these steps on the computer that’s running FIM:

  1. Install the SQL Express admin studio.
  2. Perform the following query:SELECT        userPrincipalName, msExchMailboxGuid
    FROM            mms_metaverse
    WHERE        (msExchMailboxGuid IS NOT NULL)This should return the user objects that still have the msExchMailboxGUI attribute populated.
  3. Perform the following Query to remove the msExchMailboxGUI from all users:
    UPDATE mms_metaverse SET msExchMailboxGuid = NULL WHERE (msExchMailboxGuid IS NOT NULL)
  4. Now that this is done we need to synchronize the changes to Office 365.
  5. Navigate to C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\UIShell
  6. Launch the application miisclient.exe
  7. Click on the management agents tab
  8. Right-click on the Windows Azure Active Directory Connector and click Run.
  9. Perform a Full Import Full Sync.
  10. When the Full Import is finished perform an Export.
  11. Assign licenses to the accounts in Office 365. This should create the mailboxes.

This completes the procedure. Make sure that if you have to reinstall the Directory Sync tool onto another system to apply step 1 to 10 before running it.

 

 

 


Tags:

May 20 2013 May 20, 2013 May 20, 2013 NoComment by Stefan

Installing drivers on Windows 2012 Server Core

Posted in Hyper-V, Server Core

I needed to install a USB device on my HyperV host for which drivers had to be installed. Here’s how to do it:

  1. First copy the drivers to a temporary folder on the server core host. In my case C:\Install\FTDI.
  2. Open a command prompt on the server core host.
  3. In the directory with the drivers type the following command: pnputil -i -a *.inf
  4. As you can see the drivers have been installed successfully:

    Microsoft PnP Utility
    Processing inf :            ftdibus.inf
    Successfully installed the driver on a device on the system.
    Driver package added successfully.
    Published name :            oem2.inf
    Processing inf :            ftdiport.inf
    Successfully installed the driver on a device on the system.
    Driver package added successfully.
    Published name :            oem3.inf
    Total attempted:              2
    Number successfully imported: 2

    Microsoft PnP Utility
    Processing inf :            ftdibus.infSuccessfully installed the driver on a device on the system.Driver package added successfully.Published name :            oem2.inf
    Processing inf :            ftdiport.infSuccessfully installed the driver on a device on the system.Driver package added successfully.Published name :            oem3.inf

    Total attempted:              2Number successfully imported: 2

  5. Now connect the USB device and check whether it is recognized correctly in the system event log:
    Driver Management concluded the process to install driver ftdiport.inf_amd64_66f12493ff5adfb9\ftdiport.inf for Device Instance ID FTDIBUS\VID_0403+PID_6001+AE01CSYUA\0000 with the following status: 0x0.

Tags:

Dec 31 2012 December 31, 2012 November 16, 2013 1Comment by Stefan

Remote management of a standalone Hyper-V 2012 Server

Posted in Hyper-V

This post mainly focuses on managing a standalone Hyper-V 2012 server with MMC tools. Hyper-V server is basically a server core installation. Server core can be managed using the command line through a remote desktop session but most tasks can be done remotely using the standard (MMC based) management tools.

  1. Since this is a standalone server we have to assume there is no DNS entry for this server. So first we need to add an entry in the hosts file on the workstation from where we want to manage the HyperV server.Edit the C:\windows\system32\drivers\etc\hosts file (you may need to give yourself permissions to edit the file).
    Add the ip address and name of the HyperV server and save the hosts file.
  2. Now we need to adjust the firewall on the Hyper-V server and the remote computer for remote administration to work.On the server execute the following command:netsh advfirewall firewall set rule group=”remote administration” new enable=yesYou may receive the following error: “No rules match the specified criteria”. The cause of the error seems to be that the “remote administration group” doesn’t exist. In this case run the following command first. This will create the rule group:

    netsh firewall set service type=remoteadmin mode=enable

    You will receive the following warning but the command is successful:IMPORTANT: Command executed successfully.
    However, “netsh firewall” is deprecated;
    use “netsh advfirewall firewall” instead.
    For more information on using “netsh advfirewall firewall” commands
    instead of “netsh firewall”, see KB article 947709
    at http://go.microsoft.com/fwlink/?linkid=121488

    Now repeat the command and it will succeed:

    netsh advfirewall firewall set rule group=”remote administration” new enable=yes

    Updated 3 rule(s).

    Ok.

  3. To allow access to disk management remotely run the following command on the server and the remote computer:

    netsh advfirewall firewall set rule group=”Remote Volume Management” new enable=yes
  4. Since the Hyper-V Server is not a member of a domain, establish alternate credentials to use to connect to the Hyper-V server by typing the following command at a command prompt on the remote computer:

    cmdkey /add:<ServerName> /user:<UserName> /pass:<password>
  5. In order to be able to use Server Manager we need to modify the TrustedHosts list for the WinRM client on the remote computer:

    Set-Item WSMAN:\localhost\client\trustedhosts -value YOURSERVERNAME -concatenate

Now you should be able to administer Hyper-V remotely using MMC based management tools. You can download the RSAT (remote server administration tools) for your client OS here: http://www.microsoft.com/en-us/download/search.aspx?q=remote+server+administration+tools

 


Tags: