In a fully virtualized VMware infrastructure with features like vMotion, HA, and DRS, VMs can reside on any host in the cluster. This dynamic nature creates challenges when you need to maintain strict startup sequences across the entire datacenter after a power outage or maintenance event.

While ESXi hosts allow configuring VM startup orders per host (using the "Virtual Machine Startup/Shutdown" settings in the host configuration), this approach doesn't scale well for:

• VMs that might migrate between hosts
• Dependencies spanning multiple hosts
• Complex multi-tier applications

Here's a PowerCLI script that implements a global startup sequence by checking VM dependencies before starting each VM:

# Connect to vCenter
Connect-VIServer -Server your-vcenter.example.com

# Define your startup order with dependencies
$vmStartupOrder = @(
    @{Name="DC1"; DependsOn=@()},
    @{Name="DC2"; DependsOn=@()},
    @{Name="SQL-Cluster"; DependsOn=@("DC1","DC2")},
    @{Name="App-Server"; DependsOn=@("SQL-Cluster")}
)

# Function to check if a VM is ready
function Test-VMReady {
    param($vmName)
    $vm = Get-VM -Name $vmName
    return ($vm.PowerState -eq "PoweredOn" -and 
            (Get-VMQuestion -VM $vm | Where-Object {$_.Text -like "*tools*"} | Measure-Object).Count -eq 0)
}

# Main startup sequence
foreach ($vmConfig in $vmStartupOrder) {
    $vm = Get-VM -Name $vmConfig.Name
    if ($vm.PowerState -ne "PoweredOn") {
        # Check dependencies
        $dependenciesMet = $true
        foreach ($dep in $vmConfig.DependsOn) {
            if (-not (Test-VMReady $dep)) {
                Write-Host "Waiting for dependency $dep..."
                $dependenciesMet = $false
                break
            }
        }
        
        if ($dependenciesMet) {
            Start-VM -VM $vm -Confirm:$false
            # Add delay if needed
            Start-Sleep -Seconds 30
        }
    }
}

For more sophisticated solutions, you can leverage the vSphere API through:

• vSphere Automation SDKs (Python, Java, etc.)
• REST API endpoints in vSphere 7.0+
• Integration with orchestration tools like Ansible or Terraform

When vCenter runs as a VM, you need special handling:

1. Configure host-local startup for vCenter and its dependencies
2. Use host profiles to maintain this configuration across all hosts
3. Consider running a secondary vCenter instance for management

Other strategies to consider:

• VMware Site Recovery Manager for planned failovers
• Custom alarms and actions in vCenter
• Third-party solutions like Turbonomic or vRealize Orchestrator

In modern VMware environments where VMs can migrate between hosts via vMotion, traditional per-host startup orders defined in /etc/vmware/hostd/vmAutoStart.xml become inadequate. We need a datacenter-wide orchestration that respects service dependencies regardless of physical host location.

// Sample PowerCLI to create dependency-based startup groups
$startupGroup1 = New-ClusterGroup -Name "Tier1-Core" -Cluster (Get-Cluster "ProdCluster")
$startupGroup2 = New-ClusterGroup -Name "Tier2-Database" -Cluster (Get-Cluster "ProdCluster")

Add-ClusterGroupMember -ClusterGroup $startupGroup1 -VM (Get-VM "DC01","DC02","vCenterVM")
Add-ClusterGroupMember -ClusterGroup $startupGroup2 -VM (Get-VM "SQL-PROD","Oracle-PROD")

New-ClusterGroupDependency -ClusterGroup $startupGroup2 -Dependency $startupGroup1

For complex scenarios where built-in features fall short, we can leverage the vSphere Automation API:

// Python example using pyVmomi to implement global startup sequencing
from pyVmomi import vim, vmodl

def create_global_startup_order(content, vm_list):
    startup_manager = content.vmProvisioningChecker
    spec = vim.vm.StartupSpec()
    
    for idx, vm in enumerate(vm_list):
        vm_config = vim.vm.StartupConfig()
        vm_config.vm = vm
        vm_config.startOrder = idx * 5  # 5 second intervals
        vm_config.startAction = 'powerOn'
        vm_config.waitForHeartbeat = True if "DB" in vm.name else False
        
        spec.config.append(vm_config)
    
    startup_manager.UpdateStartupOrder(spec)

When vCenter runs as a VM, we need a bootstrap sequence:

1. Configure ESXi host's vmware-vm-autostart service to start vCenter first
2. Set vCenter's HA restart priority to "High"
3. Implement retry logic in startup scripts for dependent services

For a financial services client with 500+ VMs, we implemented this solution:

# PowerShell workflow for tiered startup
$tiers = @{
    "Tier0" = @("vcsa-01","dc-01","dc-02","nsx-manager")
    "Tier1" = @("sql-cluster","oracle-rac")
    "Tier2" = @("app-server-*","web-server-*")
}

Start-VM -VM $tiers["Tier0"] -Confirm:$false
Wait-VM -VM $tiers["Tier0"] -Heartbeat -TimeoutSec 300

foreach($vm in $tiers["Tier1"]) {
    Start-VM -VM $vm
    Wait-Tools -VM $vm -TimeoutSeconds 600
}

Start-VM -VM $tiers["Tier2"] -RunAsync

Implement health checks between tiers using simple TCP port checks:

# Bash script for dependency verification
check_mssql_ready() {
    while ! nc -z $SQL_SERVER 1433; do
        sleep 5
        logger "Waiting for SQL Server at $SQL_SERVER..."
    done
}

check_domain_ready() {
    until ldapsearch -x -H ldap://$DC_SERVER -s base; do
        sleep 10
    done
}