When working with Solaris 10 systems, administrators often encounter confusing memory statistics. Let's examine a real-world scenario where multiple tools report different memory usage:

# Top output shows:
Memory: 16G phys mem, 672M free mem, 2048M total swap, 2023M free swap

# Prstat shows much lower RSS usage:
NPROC USERNAME  SWAP   RSS MEMORY
   45 orbixadm 1449M 1592M   9.7%
   48 root      146M  160M   1.0%
   ... (total RSS doesn't reach 15GB)

The key to resolving this discrepancy lies in understanding how Solaris manages memory differently from other Unix-like systems:

1. Kernel Memory Usage: The ::memstat output reveals 82% of memory is used by the kernel
2. ZFS ARC Cache: Solaris uses available RAM for filesystem caching aggressively
3. Memory Accounting: Some memory isn't accounted in process RSS but is actively used

Here are the essential commands for accurate memory analysis on Solaris 10:

# Comprehensive memory breakdown
echo ::memstat | mdb -k

# ZFS ARC cache size
kstat zfs::arcstats:size

# Process memory sorted by RSS
prstat -a -s rss

# Traditional memory stats
vmstat -p

For regular monitoring, create this shell script (memcheck.sh):

#!/bin/sh
echo "===== Memory Summary ====="
echo "Physical Memory:"
prtconf | grep Memory

echo "\n===== Kernel Memory ====="
echo ::memstat | mdb -k

echo "\n===== ZFS ARC Cache ====="
kstat zfs::arcstats:size

echo "\n===== Process Memory ====="
prstat -a -s rss 1 5

echo "\n===== Traditional Stats ====="
vmstat -p

The "missing" memory typically falls into these categories:

For deeper investigation, use these DTrace scripts to track memory allocation:

#!/usr/sbin/dtrace -s
#pragma D option quiet

proc:::exec-success
{
    printf("%s\n", execname);
}

syscall::mmap:entry
{
    @size[execname] = sum(arg2);
}

END
{
    printa("%-20s %@16d\n", @size);
}

This script helps identify which processes are requesting large memory allocations.

For more accurate reporting, consider these adjustments:

# Set minimum free memory threshold
echo "set zfs:zfs_arc_min = 0x10000000" >> /etc/system

# Adjust kernel memory reporting
echo "set kmem_flags=0x1" >> /etc/system

When working with Solaris 10 systems, many administrators encounter puzzling memory statistics where the sum of RSS (Resident Set Size) values doesn't match the total physical memory usage reported by tools like top and vmstat. Let's dissect this common scenario:

# Sample memory output from Solaris 10
$ top
Memory: 16G phys mem, 672M free mem, 2048M total swap, 2023M free swap

$ prstat -a -s size
USERNAME  SWAP   RSS MEMORY
orbixadm 1449M 1592M   9.7%
root     146M  160M   1.0%

Solaris 10 employs several memory allocation mechanisms that explain these discrepancies:

• Kernel Memory: The OS kernel maintains its own memory pools
• ZFS ARC Cache: Dynamic memory allocation for filesystem caching
• Memory Zones: Solaris virtualization features consume memory
• Dynamic Reconfiguration: Memory may be temporarily reserved

To get a complete picture, combine these commands:

# Check kernel memory usage
echo ::memstat | mdb -k

# View ZFS ARC cache size
kstat zfs::arcstats:size

# Comprehensive memory report
prstat -Z -a -s size

Here's how to properly calculate memory usage:

# Calculate total memory usage including kernel
total_mem=16384 # 16GB in MB
free_mem=$(vmstat 1 2 | tail -1 | awk '{print $5/1024}')
arc_size=$(kstat -p zfs::arcstats:size | awk '{print $2/1024/1024}')
kernel_mem=$(echo ::memstat | mdb -k | grep Kernel | awk '{print $3}')

used_mem=$(echo "$total_mem - $free_mem - $arc_size - $kernel_mem" | bc)
echo "Unaccounted memory: ${used_mem}MB"

These factors often lead to confusion:

• Shared Libraries: Counted multiple times in RSS but stored once
• tmpfs Usage: Not always visible in standard reports
• Memory Reservations: Some processes may reserve but not actively use memory

For deeper analysis, consider these approaches:

# Check memory reservations
echo "::memseg -a" | mdb -k

# View large memory consumers
echo "::memstat -g" | mdb -k

# Analyze process memory mapping
pmap -x $(pgrep process_name)

Remember that Solaris 10's memory management is fundamentally different from Linux/BSD systems. The ZFS filesystem in particular dynamically adjusts its ARC cache size based on available memory, which explains why you might see significant "missing" memory.