How to Identify I/O Intensive VMs on Hyper-V Hosts Using Performance Counters and PowerShell


2 views

When managing Hyper-V environments with multiple VMs (typically 20-50 per host), identifying I/O-intensive virtual machines becomes critical for performance troubleshooting. The Windows System process aggregates all Hyper-V storage operations, making traditional tools like Process Explorer or Task Manager inadequate for VM-level diagnostics.

Method 1: Hyper-V Performance Counters

The most precise way involves leveraging Hyper-V's built-in performance counters. Run this PowerShell command to list relevant counters:

Get-Counter -ListSet "Hyper-V Virtual Storage Device" | 
Select-Object -ExpandProperty Counter

Method 2: Real-Time Monitoring with PowerShell

This script provides real-time VM disk activity monitoring:

$counterPath = "\Hyper-V Virtual Storage Device(*)\Write Bytes/sec"
$sampleInterval = 5 # seconds

while($true) {
    $counters = (Get-Counter -Counter $counterPath).CounterSamples
    $activeVMs = $counters | 
        Where-Object { $_.CookedValue -gt 0 } |
        Sort-Object -Property CookedValue -Descending |
        Select-Object InstanceName, CookedValue
    
    Clear-Host
    Write-Host "Top I/O Intensive VMs - $(Get-Date)"
    $activeVMs | Format-Table -AutoSize
    Start-Sleep -Seconds $sampleInterval
}

Historical Analysis with PerfMon

Configure a Data Collector Set with these key counters:

  • \Hyper-V Virtual Storage Device(*)\Read Bytes/sec
  • \Hyper-V Virtual Storage Device(*)\Write Bytes/sec
  • \Hyper-V Virtual Storage Device(*)\Total IOPS

Storage-Specific Troubleshooting

For CSV environments, add these counters:

"\Cluster Storage Hybrid Disks(*)\Bytes Read/sec"
"\Cluster Storage Hybrid Disks(*)\Bytes Written/sec"

When Performance Counters aren't available (e.g., Server Core):

# Check VM disk latency metrics
Get-VM | Get-VMHardDiskDrive | 
Select-Object VMName, ControllerType, ControllerNumber, ControllerLocation,
@{Name="Latency";Expression={(Get-VHD $_.Path).DiskIdentifier}}

Sample output from our monitoring script:

InstanceName                          CookedValue
------------                          -----------
VM07_HardDisk_1                        157286400
VM12_HardDisk_0                         52428800
VM03_HardDisk_2                         10485760

This clearly shows VM07's first virtual disk is generating ~150MB/s of writes.

  • Consider separating high-I/O VMs onto dedicated physical disks
  • Review VM virtual disk configurations (fixed vs dynamic disks)
  • Check for proper integration services installation
  • Evaluate storage QoS policies for critical workloads

When managing Hyper-V hosts with multiple VMs (typically 20-30 per host), administrators often encounter periods of sustained high disk utilization without clear visibility into which specific VM is causing the bottleneck. The native Windows tools like Process Explorer simply attribute all Hyper-V related I/O to the "System" process, making troubleshooting frustratingly opaque.

Here are three technical approaches to pinpoint I/O-intensive VMs:

# PowerShell: Get VM disk activity metrics
Get-VM | Get-VMHardDiskDrive | ForEach-Object {
    $stats = Get-VMResourcePool -Name "PhysicalDisk" | Get-VMResourcePoolStatistics
    [PSCustomObject]@{
        VMName = $_.VMName
        DiskPath = $_.Path
        ReadIOPS = $stats.AllocationMetricValue.ReadIops
        WriteIOPS = $stats.AllocationMetricValue.WriteIops
        Latency = $stats.AllocationMetricValue.AverageLatency
    }
} | Sort-Object WriteIOPS -Descending

Configure these essential Hyper-V performance counters:

  • Hyper-V Virtual Storage Device - Read Bytes/sec (Instance = VM name)
  • Hyper-V Virtual Storage Device - Write Bytes/sec (Instance = VM name)
  • Hyper-V Virtual IDE Controller - Storage Error Count

For deep analysis, use Event Tracing for Windows to capture disk I/O events:

# Capture Hyper-V storage events for 60 seconds
logman create trace HyperVStorageTrace -ow -o C:\Traces\HyperVStorage.etl -p "Microsoft-Windows-Hyper-V-VMMS" 0x8000 -f bincirc -max 500 -nb 16 16 -bs 1024
logman start HyperVStorageTrace
Start-Sleep -Seconds 60
logman stop HyperVStorageTrace

# Parse with PowerShell
Get-WinEvent -Path C:\Traces\HyperVStorage.etl | Where-Object {
    $_.Id -eq 21032 -or $_.Id -eq 21033
} | Select-Object TimeCreated, @{n="VM";e={$_.Properties[0].Value}},
  @{n="IOType";e={$_.Properties[3].Value}},
  @{n="SizeKB";e={[math]::Round($_.Properties[4].Value/1KB,2)}}

For Server Core installations where Resmon isn't available:

  1. Configure Performance Monitor to log to a file
  2. Use PowerShell remoting to collect data from multiple hosts
  3. Deploy Azure Monitor or SCOM for enterprise-scale monitoring

A case study where we identified a SQL Server VM causing storage latency:

# Sample output from our troubleshooting
VMName    DiskPath                     ReadIOPS WriteIOPS Latency
------    --------                     -------- --------- -------
SQLVM01   E:\VHDs\SQLData_Disk4.vhdx   1856     3428      45.7
WEBVM12   C:\VHDs\WebApp_Disk1.vhdx    892      567       12.3
DCVM05    C:\VHDs\DomainCtrl.vhdx      324      211       8.2

The data clearly showed SQLVM01 was responsible for 68% of write operations with concerning latency spikes during business hours. We resolved this by migrating the VM to faster storage and optimizing SQL Server configurations.