Sep 9, 2022

JVM (Java Virtual Machine) Architecture

INDEX

  • The Java Virtual Machine (JVM) is the foundation of the Java ecosystem and one of the most battle-tested runtime platforms in enterprise computing. 
  • JVM is not just responsible for executing Java bytecode — it is a sophisticated runtime system that handles memory management, adaptive optimization, thread scheduling, garbage collection, security isolation, and platform portability.

In modern cloud-native architectures, understanding JVM internals is no longer optional for senior engineers. JVM behavior directly impacts:

  • API latency
  • Throughput
  • Kubernetes stability
  • Cloud infrastructure cost
  • Scalability
  • GC pause times
  • Startup performance
  • Distributed system resiliency

This article explains JVM architecture from both:

  • conceptual perspective,
  • and production engineering perspective.

What is JVM?

The Java Virtual Machine (JVM) is an abstract runtime engine that executes Java bytecode generated by the Java compiler (javac).

JVM provides:

  • Platform independence (“Write Once, Run Anywhere”)
  • Automatic memory management
  • Runtime optimization
  • Security sandboxing
  • Thread management
  • Dynamic class loading

The same .class bytecode can execute on:

  • Linux
  • Windows
  • macOS
  • Containers
  • Cloud environments
without recompilation because JVM abstracts the underlying operating system and hardware.


JVM Architecture Overview

The JVM architecture mainly consists of:

  1. ClassLoader Subsystem
  2. Runtime Data Areas
  3. Execution Engine
  4. Native Interface (JNI)
  5. Native Method Libraries

Each component plays a critical role in performance, scalability, and runtime behavior.

1. ClassLoader Subsystem

The ClassLoader subsystem is responsible for:

  • loading .class files into memory,
  • verifying bytecode,
  • linking dependencies,
  • and initializing classes at runtime.

Unlike traditional compiled languages, Java supports dynamic class loading during runtime.

This enables:

  • plugin architectures,
  • application servers,
  • Spring Boot dependency injection,
  • reflection,
  • dynamic proxies,
  • hot deployments.

2.1.1 Class Loading Lifecycle

The JVM class loading lifecycle consists of:

  1. Loading
  2. Linking
  3. Initialization

2.1.1.1 Loading Phase

During the loading phase:

  • JVM reads .class bytecode,
  • creates corresponding Class objects,
  • stores metadata into Metaspace.

The ClassLoader does not execute code during loading.

Modern JVMs use:

  • Bootstrap ClassLoader
  • Platform ClassLoader
  • Application ClassLoader

In enterprise systems, frameworks like Spring Boot, Hibernate, Kafka, and Netty heavily depend on dynamic class loading.

2.1.1.2 Linking Phase

Linking consists of:

2.1.1.2.1 Verification

    • Ensures bytecode is valid and secure.
    • Checks include:

      • invalid bytecode,
      • stack overflow rules,
      • illegal memory access,
      • type safety validation.
    • This is one of the reasons JVM is highly secure compared to native runtimes.

2.1.1.2.2 Preparation

    • During preparation:
    • static variables are allocated memory,
    • default values are assigned.
    • Example:
      • static int count = 10;
    • During preparation:
    • During preparation:
    • count = 0;

2.1.1.2.3 Resolution

    • Converts symbolic references into direct memory references.
    • Example:

      • method references,
      • interface references,
      • field references.
    • This improves runtime execution efficiency.

2.1.1.3 Initialization Phase

During initialization:

  • static blocks execute,
  • actual variable assignments happen,
  • class becomes fully usable.

Example:

static int count = 10;

Now:

count = 10;

2.1.5 Important Production Insight

Improper ClassLoader management is one of the most common causes of:

  • Metaspace leaks,
  • memory retention,
  • application server instability.

This is especially common in:

  • plugin systems,
  • dynamic proxy generation,
  • hot redeployment environments,
  • OSGi-based systems.

In modern cloud-native deployments, container restarts often hide these issues temporarily, making them difficult to detect.

2. Runtime Data Areas

Runtime Data Areas are memory regions used during program execution.

These areas are divided into:

  • Thread Shared Memory
  • Thread Private Memory

2.2.1 Method Area (Metaspace)

The Method Area stores:

  • class metadata,
  • runtime constant pool,
  • method metadata,
  • static variables,
  • JIT compiled code metadata.

Before Java 8:

  • this was called PermGen.

Modern JVMs use:

  • Metaspace,
  • which allocates memory from native memory instead of heap.

2.2.2 Why Metaspace Matters in Production

Metaspace leaks are common in:

  • dynamically generated classes,
  • Spring proxies,
  • Hibernate enhancement,
  • classloader leaks,
  • reflection-heavy systems.

Symptoms:

  • increasing native memory usage,
  • container OOMKills,
  • OutOfMemoryError: Metaspace.

2.2.3 Heap Memory

Heap is the primary memory area for object allocation.

All objects and arrays are allocated here.

Heap is shared across all threads.

2.2.4 Heap Structure

Modern JVM heaps are divided into:

  • Young Generation
    • Eden Space
    • Survivor Spaces
  • Old Generation

2.2.5 Object Allocation Lifecycle

Most objects are first allocated in:

  • Eden Space.

Objects surviving multiple GC cycles get promoted to:

  • Old Generation.

This design is based on the observation:

Most objects die young.

This principle is called:

  • Generational Hypothesis.

2.2.6 Production Engineering Perspective

High allocation rate systems create:

  • excessive GC pressure,
  • CPU spikes,
  • latency instability.

Common examples:

  • excessive JSON serialization,
  • object-heavy streams,
  • large temporary collections,
  • reactive pipelines creating millions of short-lived objects.

2.2.7 Stack Memory

Each thread gets its own JVM stack.

Stack stores:

  • method frames,
  • local variables,
  • method arguments,
  • partial computation results.

Stack memory is thread-private.

2.2.8 Common Stack Errors

StackOverflowError

Occurs due to:

  • deep recursion,
  • infinite recursion,
  • very large call chains.

OutOfMemoryError (unable to create native thread)

Occurs when:

  • too many threads are created,
  • native memory is exhausted.

This is extremely common in improperly tuned microservices.

2.2.9 PC Register

Each thread has its own Program Counter (PC) Register.

It stores:

  • address of currently executing instruction.

This enables JVM thread context switching.

2.2.10 Native Method Stack

Used for executing:

  • native methods,
  • JNI calls,
  • OS-level integrations.

Used heavily in:

  • compression libraries,
  • networking libraries,
  • database drivers,
  • cryptographic operations. 

3. Execution Engine

The Execution Engine executes bytecode loaded into memory.

Major components include:

  1. Interpreter
  2. JIT Compiler
  3. Garbage Collector

Interpreter

The interpreter executes bytecode instruction-by-instruction.

Advantages:

  • fast startup,
  • lower compilation overhead.

Disadvantages:

  • slower execution for repeated code paths.

JIT (Just-In-Time) Compiler

The JIT compiler improves runtime performance by:

  • compiling hot bytecode paths into native machine code.

This enables JVM applications to achieve near-native performance.

Modern JVMs use:

  • Tiered Compilation
  • C1 Compiler
  • C2 Compiler

HotSpot Optimization Techniques

Modern JVMs perform advanced optimizations:

  • Method inlining
  • Escape analysis
  • Dead code elimination
  • Loop unrolling
  • Lock coarsening
  • Lock elimination
  • Branch prediction optimization

These optimizations happen dynamically during runtime.

Why JVM Warmup Matters

JVM performance changes over time.

During startup:

  • interpretation dominates.

Later:

  • optimized native code dominates.

This is why:

  • latency profiles improve after warmup,
  • benchmark results can be misleading.

Production systems often use:

  • prewarming,
  • CDS/AppCDS,
  • traffic ramp-up strategies. 

4. Garbage Collection (GC)

Garbage Collection automatically reclaims unused memory.

This is one of the JVM’s most powerful features.

However:

GC tuning is one of the most critical production engineering disciplines.

Types of Garbage Collectors

Serial GC

  • Single-threaded
  • Suitable for small applications

Parallel GC

  • High throughput
  • Longer pause times

G1GC

  • Default in modern JVMs
  • Balanced throughput and latency

ZGC

  • Ultra-low latency GC
  • Suitable for massive heaps

Shenandoah GC

  • Low pause-time collector
  • Concurrent compaction support

GC Tradeoffs

GC Best ForTradeoff
Parallel GC    Throughput        Long pauses
G1GC    Enterprise apps        Balanced
ZGC    Low latency            Higher CPU
Shenandoah        Large heaps        More overhead

Stop-The-World (STW) Events

Certain GC phases pause application threads.

These pauses are called:

  • Stop-The-World pauses.

Long STW pauses can cause:

  • API timeouts,
  • Kafka consumer lag,
  • distributed transaction failures,
  • cascading retries,
  • SLA violations.

Real Production Example

A payment platform experienced:

  • random 3-second latency spikes.

Root cause:

  • Full GC pauses caused by improper heap sizing and excessive object retention.

Impact:

  • retry storms,
  • thread pool exhaustion,
  • downstream timeout cascades.

GC tuning reduced:

  • p99 latency from 3.2s to 180ms. 

5. Java Native Interface (JNI)

JNI allows Java code to interact with:

  • native C/C++ libraries,
  • OS-level APIs,
  • hardware integrations.

JNI is commonly used in:

  • high-performance networking,
  • cryptography,
  • compression,
  • GPU integrations.

JNI Risks

JNI bypasses JVM safety guarantees.

Improper JNI usage can cause:

  • segmentation faults,
  • native memory leaks,
  • JVM crashes.

Unlike regular Java exceptions:

  • native crashes terminate the JVM process completely. 

6. Native Method Libraries

These are platform-specific libraries used by JNI.

Examples:

  • .dll files on Windows
  • .so files on Linux

Used for:

  • database drivers,
  • ML accelerators,
  • image processing,
  • operating system integrations. 

JVM in Cloud-Native & Kubernetes Environments

Modern JVM engineering must consider containers.

Key challenges:

  • container memory limits,
  • CPU throttling,
  • startup time,
  • autoscaling behavior.

Common Kubernetes JVM Problems

Pod OOMKilled Despite Free Heap

Reason:

  • JVM heap is not total JVM memory.

Additional memory includes:

  • Metaspace
  • Direct memory
  • Thread stacks
  • Native memory
  • GC structures

CPU Throttling

Container CPU limits can:

  • delay GC,
  • increase latency,
  • impact JIT compilation.

Startup Optimization

Modern JVM startup improvements include:

  • CDS/AppCDS
  • GraalVM Native Image
  • CRaC
  • Tiered compilation tuning

JVM Observability & Monitoring

Principal engineers must deeply understand JVM observability.

Critical tools include:

  • JFR (Java Flight Recorder)
  • JMC (Java Mission Control)
  • async-profiler
  • VisualVM
  • GC logs
  • Heap dumps
  • Thread dumps
  • Flame graphs

Essential JVM Metrics

Key production metrics:

  • Heap utilization
  • Allocation rate
  • GC pause time
  • Safepoint duration
  • Thread count
  • Native memory usage
  • CPU utilization
  • Class loading rate
 

JVM Failure Patterns Every 

Memory Leak

Symptoms:

  • increasing heap usage,
  • Full GC frequency rise,
  • eventual OOM.

Metaspace Leak

Usually caused by:

  • classloader retention,
  • dynamic proxy creation.

Thread Pool Exhaustion

Causes:

  • blocking calls,
  • slow downstream dependencies,
  • unbounded queues.

GC Thrashing

Occurs when:

  • JVM spends most time in GC,
  • application throughput collapses.

Why JVM Still Dominates Enterprise Systems

Despite newer runtimes:

  • Go,
  • Rust,
  • Node.js,

JVM remains dominant because of:

  • mature ecosystem,
  • runtime optimization,
  • observability tooling,
  • scalability,
  • decades of production hardening.

Modern JVMs can efficiently run:

  • high-throughput systems,
  • low-latency APIs,
  • AI workloads,
  • streaming platforms,
  • cloud-native microservices. 

JVM Memory Is More Than Heap

One of the biggest misconceptions in JVM engineering is assuming that every OutOfMemoryError (OOM) is caused by Java heap exhaustion.

In reality, many production JVM failures occur outside the Java heap.

This is especially important in:

  • Kubernetes,
  • microservices,
  • high-concurrency systems,
  • Netty/reactive applications,
  • streaming platforms,
  • Spring Boot services.

Many engineers only analyze:

  • heap dumps,
  • GC logs,
  • heap utilization.

But the JVM process consumes far more memory than just heap.

Complete JVM Process Memory Layout

A JVM process typically consumes memory from:

Memory AreaInside Heap?Common Failure
Heap Memory    Yes        Java heap OOM
MetaspaceNo    Metaspace OOM
Direct MemoryNo    Direct buffer OOM
Thread StacksNo        Unable to create native thread
JNI Native MemoryNo    Native crash/OOM
Code CacheNo    CodeCache full
GC StructuresNo    Native pressure
OS Memory Mapping    No        Container OOMKill

Common Non-Heap Memory Failures

1. Metaspace OOM

Error:

OutOfMemoryError: Metaspace

Usually caused by:

  • classloader leaks,
  • excessive proxy generation,
  • dynamic bytecode generation,
  • hot redeployments.

Common in:

  • Spring Boot
  • Hibernate
  • CGLIB
  • ByteBuddy
  • Application servers

2. Direct Buffer Memory OOM

Error:

OutOfMemoryError: Direct buffer memory

Very common in:

  • Netty
  • Kafka
  • WebFlux
  • Reactive systems
  • High-throughput networking applications

Direct memory is allocated outside heap using:

ByteBuffer.allocateDirect()

Heap may appear healthy while JVM crashes due to native memory exhaustion.

3. Unable to Create Native Thread

Error:

OutOfMemoryError: unable to create native thread

This does NOT mean heap issue.

Usually caused by:

  • too many threads,
  • excessive thread pools,
  • blocked threads,
  • low container memory limits,
  • OS thread limits.

Each thread requires:

  • native stack memory.

Thousands of threads can exhaust native memory even with low heap usage.

4. Container OOMKill (Most Misunderstood Issue)

In Kubernetes:

  • JVM may be killed even when heap looks healthy.

Reason:
Container memory includes:

  • Heap
  • Metaspace
  • Direct memory
  • Thread stacks
  • Native libraries
  • GC structures
  • JIT compiler memory

Example:

Heap Usage: 1.2 GB
Container Limit: 2 GB

But:
Metaspace: 300 MB
Direct Memory: 400 MB
Thread Stacks: 250 MB
Native: 200 MB

Total Process Memory = 2.35 GB

Result:

OOMKilled

This is one of the most common cloud-native JVM production incidents.

5. Code Cache Exhaustion

Error:

CodeCache is full

JIT compiler stores generated native code in Code Cache.

Large applications with:

  • many hot methods,
  • excessive dynamic code,
  • aggressive compilation

can exhaust code cache.

Symptoms:

  • sudden performance degradation,
  • compiler disabled,
  • CPU spikes.

Why Heap Dumps Alone Are Misleading

Heap dumps only show:

  • heap objects.

They do NOT show:

  • direct memory,
  • thread stacks,
  • JNI allocations,
  • metaspace internals,
  • native fragmentation.

This is why:

“Heap looks fine” does not mean JVM memory is healthy.

Essential JVM Memory Metrics

Principal engineers monitor:

MetricWhy It Matters
Heap UsageGC pressure
Allocation RateObject churn
Metaspace UsageClassloader leaks
Direct MemoryNetty/Kafka pressure
Thread CountNative stack growth
Native MemoryTotal process health
RSS MemoryActual container usage
GC Pause TimeLatency impact

Tools for Deep Memory Analysis

Heap Analysis

  • MAT
  • VisualVM
  • Heap dump analyzers

Native Memory Analysis

  • Native Memory Tracking (NMT)
  • jcmd VM.native_memory
  • pmap
  • top
  • smaps

Thread Analysis

  • jstack
  • Thread dumps

Container Analysis

  • cAdvisor
  • Prometheus
  • Grafana
  • Kubernetes metrics

Production Engineering Lesson

One of the biggest JVM mistakes is:

Setting only -Xmx and ignoring total process memory.

In containers:

  • JVM tuning must consider TOTAL memory footprint,
  • not just heap size.

A properly engineered JVM memory strategy includes:

  • heap sizing,
  • metaspace sizing,
  • direct memory limits,
  • thread pool sizing,
  • container memory headroom,
  • GC overhead planning.

Spring-Boot-performance tuning guide 

Labels: , ,

You may also like

Kubernetes Microservices
Python AI/ML
Spring Framework Spring Boot
Core Java Java Coding Question
Maven AWS