When managing Hyper-V environments with multiple VMs (typically 20-50 per host), identifying I/O-intensive virtual machines becomes critical for performance troubleshooting. The Windows System process aggregates all Hyper-V storage operations, making traditional tools like Process Explorer or Task Manager inadequate for VM-level diagnostics.
Method 1: Hyper-V Performance Counters
The most precise way involves leveraging Hyper-V's built-in performance counters. Run this PowerShell command to list relevant counters:
Get-Counter -ListSet "Hyper-V Virtual Storage Device" | Select-Object -ExpandProperty Counter
Method 2: Real-Time Monitoring with PowerShell
This script provides real-time VM disk activity monitoring:
$counterPath = "\Hyper-V Virtual Storage Device(*)\Write Bytes/sec" $sampleInterval = 5 # seconds while($true) { $counters = (Get-Counter -Counter $counterPath).CounterSamples $activeVMs = $counters | Where-Object { $_.CookedValue -gt 0 } | Sort-Object -Property CookedValue -Descending | Select-Object InstanceName, CookedValue Clear-Host Write-Host "Top I/O Intensive VMs - $(Get-Date)" $activeVMs | Format-Table -AutoSize Start-Sleep -Seconds $sampleInterval }
Historical Analysis with PerfMon
Configure a Data Collector Set with these key counters:
- \Hyper-V Virtual Storage Device(*)\Read Bytes/sec
- \Hyper-V Virtual Storage Device(*)\Write Bytes/sec
- \Hyper-V Virtual Storage Device(*)\Total IOPS
Storage-Specific Troubleshooting
For CSV environments, add these counters:
"\Cluster Storage Hybrid Disks(*)\Bytes Read/sec" "\Cluster Storage Hybrid Disks(*)\Bytes Written/sec"
When Performance Counters aren't available (e.g., Server Core):
# Check VM disk latency metrics Get-VM | Get-VMHardDiskDrive | Select-Object VMName, ControllerType, ControllerNumber, ControllerLocation, @{Name="Latency";Expression={(Get-VHD $_.Path).DiskIdentifier}}
Sample output from our monitoring script:
InstanceName CookedValue ------------ ----------- VM07_HardDisk_1 157286400 VM12_HardDisk_0 52428800 VM03_HardDisk_2 10485760
This clearly shows VM07's first virtual disk is generating ~150MB/s of writes.
- Consider separating high-I/O VMs onto dedicated physical disks
- Review VM virtual disk configurations (fixed vs dynamic disks)
- Check for proper integration services installation
- Evaluate storage QoS policies for critical workloads
When managing Hyper-V hosts with multiple VMs (typically 20-30 per host), administrators often encounter periods of sustained high disk utilization without clear visibility into which specific VM is causing the bottleneck. The native Windows tools like Process Explorer simply attribute all Hyper-V related I/O to the "System" process, making troubleshooting frustratingly opaque.
Here are three technical approaches to pinpoint I/O-intensive VMs:
# PowerShell: Get VM disk activity metrics
Get-VM | Get-VMHardDiskDrive | ForEach-Object {
$stats = Get-VMResourcePool -Name "PhysicalDisk" | Get-VMResourcePoolStatistics
[PSCustomObject]@{
VMName = $_.VMName
DiskPath = $_.Path
ReadIOPS = $stats.AllocationMetricValue.ReadIops
WriteIOPS = $stats.AllocationMetricValue.WriteIops
Latency = $stats.AllocationMetricValue.AverageLatency
}
} | Sort-Object WriteIOPS -Descending
Configure these essential Hyper-V performance counters:
- Hyper-V Virtual Storage Device - Read Bytes/sec (Instance = VM name)
- Hyper-V Virtual Storage Device - Write Bytes/sec (Instance = VM name)
- Hyper-V Virtual IDE Controller - Storage Error Count
For deep analysis, use Event Tracing for Windows to capture disk I/O events:
# Capture Hyper-V storage events for 60 seconds
logman create trace HyperVStorageTrace -ow -o C:\Traces\HyperVStorage.etl -p "Microsoft-Windows-Hyper-V-VMMS" 0x8000 -f bincirc -max 500 -nb 16 16 -bs 1024
logman start HyperVStorageTrace
Start-Sleep -Seconds 60
logman stop HyperVStorageTrace
# Parse with PowerShell
Get-WinEvent -Path C:\Traces\HyperVStorage.etl | Where-Object {
$_.Id -eq 21032 -or $_.Id -eq 21033
} | Select-Object TimeCreated, @{n="VM";e={$_.Properties[0].Value}},
@{n="IOType";e={$_.Properties[3].Value}},
@{n="SizeKB";e={[math]::Round($_.Properties[4].Value/1KB,2)}}
For Server Core installations where Resmon isn't available:
- Configure Performance Monitor to log to a file
- Use PowerShell remoting to collect data from multiple hosts
- Deploy Azure Monitor or SCOM for enterprise-scale monitoring
A case study where we identified a SQL Server VM causing storage latency:
# Sample output from our troubleshooting
VMName DiskPath ReadIOPS WriteIOPS Latency
------ -------- -------- --------- -------
SQLVM01 E:\VHDs\SQLData_Disk4.vhdx 1856 3428 45.7
WEBVM12 C:\VHDs\WebApp_Disk1.vhdx 892 567 12.3
DCVM05 C:\VHDs\DomainCtrl.vhdx 324 211 8.2
The data clearly showed SQLVM01 was responsible for 68% of write operations with concerning latency spikes during business hours. We resolved this by migrating the VM to faster storage and optimizing SQL Server configurations.