When monitoring our Java application under moderate load, we observed significant memory consumption that couldn't be explained by our heap configuration:

PID   USER    PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12663 test    20   0 8378m 6.0g 4492 S   43  8.4 162:29.95 java

Despite conservative JVM settings (-Xmx2048m, -Xms2048m), the process shows 6GB resident memory. Heap analysis revealed no issues:

Heap Configuration:
MaxHeapSize      = 2147483648 (2048.0MB)
NewSize          = 536870912 (512.0MB)
concurrent mark-sweep generation: 63.7% used
Perm Generation: 82.5% used

Using pmap revealed numerous 64MB anonymous mappings:

00007f32b8000000       0   65508   65508 rwx--    [ anon ]
00007f32ac000000       0   65512   65512 rwx--    [ anon ]
00007f3268000000       0   65516   65516 rwx--    [ anon ]

Several tools can help identify these off-heap allocations:

# NMT (Native Memory Tracking)
jcmd <pid> VM.native_memory detail

# jemalloc profiling
MALLOC_CONF=prof:true,lg_prof_sample:17 java -Xmx2048m ...

# gdb examination
gdb -p <pid>
(gdb) dump memory /tmp/mem_region.bin 0x00007f32b8000000 0x00007f32b8000000+65508

Possible sources of these 64MB chunks:

• Memory-mapped files (MappedByteBuffer)
• JNI allocations
• Direct ByteBuffers (even beyond MaxDirectMemorySize)
• Uncommon garbage collector structures (G1 regions, etc.)
• Third-party native libraries

To check direct buffer usage:

import java.lang.reflect.Field;
import java.nio.Buffer;
import sun.misc.Unsafe;

public class DirectMemoryCheck {
    public static void main(String[] args) throws Exception {
        Field f = Unsafe.class.getDeclaredField("theUnsafe");
        f.setAccessible(true);
        Unsafe unsafe = (Unsafe) f.get(null);
        
        Field maxMemoryField = Class.forName("java.nio.Bits")
            .getDeclaredField("RESERVED_MEMORY");
        maxMemoryField.setAccessible(true);
        System.out.println("Direct memory reserved: " + 
            ((sun.misc.Counter)maxMemoryField.get(null)).value / (1024*1024) + "MB");
    }
}

For detailed native memory profiling:

# Build with debug symbols
./configure --prefix=$HOME/jvm-debug --with-debug-level=slowdebug
make all

# Then profile with:
perf record -g -p <pid>
perf report

If the 64MB chunks are from native libraries:

1. Update to newer library versions
2. Configure alternative memory allocators (jemalloc, tcmalloc)
3. Set environment variables to limit arena counts:

   export MALLOC_ARENA_MAX=2
   

---

When examining Java process memory usage, we observe a significant gap between configured heap settings (-Xmx2048m) and actual resident memory (6GB). This suggests substantial native memory allocation outside the JVM heap. The pmap output reveals numerous 64MB anonymous memory mappings that aren't accounted for in standard heap analysis.

Several components can contribute to native memory usage in Java processes:

• Direct Byte Buffers: Controlled by -XX:MaxDirectMemorySize (256MB in this case)
• Thread Stacks: Default varies by OS (typically 1MB per thread in Linux)
• JNI Code: Native libraries called through JNI
• Memory Mapped Files: Particularly relevant for database applications
• JVM Internal Structures: Metaspace, JIT code cache, GC structures

To identify the source of anonymous 64MB chunks:

# Use NMT (Native Memory Tracking)
java -XX:NativeMemoryTracking=summary -XX:+UnlockDiagnosticVMOptions -Xmx2048m -Xms2048m ...

# After running, check current allocations
jcmd <pid> VM.native_memory summary

# For detailed tracking
jcmd <pid> VM.native_memory detail

For the 64MB chunks specifically:

# Use jemalloc profiling (Linux)
MALLOC_CONF=prof:true,lg_prof_sample:20 java ...

# Or examine with gdb (advanced)
gdb -p <pid>
(gdb) dump memory /tmp/mem_region.bin 0x7f32b8000000 0x7f32b8000000+65536*1024

The 64MB chunks often originate from:

• G1 GC regions: If using G1 collector (-XX:+UseG1GC)
• Compressed Oops: Large native memory allocations for pointer compression
• Third-party libraries: Netty, Lucene, or native image processing libraries

To verify G1 GC involvement:

# Check GC logs or add these JVM flags:
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/path/to/gc.log

1. First identify the largest native allocations:

pmap -x <pid> | awk '/rwx/ {print $1, $3}' | sort -k2 -n | tail -10

2. Cross-reference with NMT output:

jcmd <pid> VM.native_memory | grep -A10 "Total:"

3. For suspected Netty allocations (common in network apps):

-Dio.netty.allocator.type=pooled -Dio.netty.allocator.numHeapArenas=2 -Dio.netty.allocator.numDirectArenas=2

For persistent issues, consider:

# Linux perf tool for memory profiling
perf record -e mem-loads,mem-stores -p <pid> -g -- sleep 30

# Or build with debug symbols and use jemalloc profiling
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 MALLOC_CONF=prof:true,prof_prefix:/tmp/jeprof java ...

Remember that resident memory (RES) includes shared libraries and memory-mapped files, while private memory is what's truly unique to your process. The difference between VIRT and RES often comes from:

• Memory-mapped JAR files
• Shared libraries loaded by the JVM
• Memory allocated but not yet touched (demand paging)