The Nutanix Bible

Copyright (c) 2021: The Nutanix Bible and, 2021. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Nutanix and with appropriate and specific direction to the original content.

Have feedback? Find a typo? Send feedback to biblefeedback at nutanix dot com

Localized versions available:

PDF Versions: Given the frequent updates to the Nutanix Bible, a dedicated PDF version is not viable. However, you can 'Print to PDF' in the event you want to print a hard copy.

A Brief Lesson in History

A brief look at the history of infrastructure and what has led us to where we are today.

The Evolution of the Datacenter

The datacenter has evolved significantly over the last several decades. The following sections will examine each era in detail.  

The Era of the Mainframe

The mainframe ruled for many years and laid the core foundation of where we are today. It allowed companies to leverage the following key characteristics:

  • Natively converged CPU, main memory, and storage
  • Engineered internal redundancy

But the mainframe also introduced the following issues:

  • The high costs of procuring infrastructure
  • Inherent complexity
  • A lack of flexibility and highly siloed environments

The Move to Stand-Alone Servers

With mainframes, it was very difficult for organizations within a business to leverage these capabilities which partly led to the entrance of pizza boxes or stand-alone servers. Key characteristics of stand-alone servers included:

  • CPU, main memory, and direct-attached storage (DAS)
  • Higher flexibility than the mainframe
  • Accessed over the network

These stand-alone servers introduced more issues:

  • Increased number of silos
  • Low or unequal resource utilization
  • The server became a single point of failure (SPOF) for both compute AND storage

Centralized Storage

Businesses always need to make money and data is a key piece of that puzzle. With direct-attached storage (DAS), organizations either needed more space than was locally available, or data high availability (HA) where a server failure wouldn’t cause data unavailability.

Centralized storage replaced both the mainframe and the stand-alone server with sharable, larger pools of storage that also provided data protection. Key characteristics of centralized storage included:

  • Pooled storage resources led to better storage utilization
  • Centralized data protection via RAID eliminated the chance that server loss caused data loss
  • Storage were performed over the network

Issues with centralized storage included:

  • They were potentially more expensive, however data is more valuable than the hardware
  • Increased complexity (SAN Fabric, WWPNs, RAID groups, volumes, spindle counts, etc.)
  • They required another management tool / team

The Introduction of Virtualization

At this point in time, compute utilization was low and resource efficiency was impacting the bottom line. Virtualization was then introduced and enabled multiple workloads and operating systems (OSs) to run as virtual machines (VMs) on a single piece of hardware. Virtualization enabled businesses to increase utilization of their pizza boxes, but also increased the number of silos and the impacts of an outage. Key characteristics of virtualization included:

  • Abstracting the OS from hardware (VM)
  • Very efficient compute utilization led to workload consolidation

Issues with virtualization included:

  • An increase in the number of silos and management complexity
  • A lack of VM high-availability, so if a compute node failed the impact was much larger
  • A lack of pooled resources
  • The need for another management tool / team

Virtualization Matures

The hypervisor became a very efficient and feature-filled solution. With the advent of tools, including VMware vMotion, HA, and DRS, users obtained the ability to provide VM high availability and migrate compute workloads dynamically. The only caveat was the reliance on centralized storage, causing the two paths to merge. The only down turn was the increased load on the storage array before and VM sprawl led to contention for storage I/O.

Key characteristics included:

  • Clustering led to pooled compute resources
  • The ability to dynamically migrate workloads between compute nodes (DRS / vMotion)
  • The introduction of VM high availability (HA) in the case of a compute node failure
  • A requirement for centralized storage

Issues included:

  • Higher demand on storage due to VM sprawl
  • Requirements to scale out more arrays creating more silos and more complexity
  • Higher $ / GB due to requirement of an array
  • The possibility of resource contention on array
  • It made storage configuration much more complex due to the necessity to ensure:
    • VM to datastore / LUN ratios
    • Spindle count to facilitate I/O requirements

Solid State Disks (SSDs)

SSDs helped alleviate this I/O bottleneck by providing much higher I/O performance without the need for tons of disk enclosures.  However, given the extreme advances in performance, the controllers and network had not yet evolved to handle the vast I/O available. Key characteristics of SSDs included:

  • Much higher I/O characteristics than traditional HDD
  • Essentially eliminated seek times

SSD issues included:

  • The bottleneck shifted from storage I/O on disk to the controller / network
  • Silos still remained
  • Array configuration complexity still remained

In Comes Cloud

The term cloud can be very ambiguous by definition. Simply put it's the ability to consume and leverage a service hosted somewhere provided by someone else.

With the introduction of cloud, the perspectives IT, the business and end-users have shifted.

Business groups and IT consumers require IT provide the same capabilities of cloud, its agility and time to value. If not, they will go directly to cloud which causes another issue for IT: data security.

Core pillars of any cloud service:

  • Self-service / On-demand
    • Rapid time to value (TTV) / little barrier to entry
  • Service and SLA focus
    • Contractual guarantees around uptime / availability / performance
  • Fractional consumption model
    • Pay for what you use (some services are free)
Cloud Classifications

Most general classifications of cloud fall into three main buckets (starting at the highest level and moving downward):

  • Software as a Service (SaaS)
    • Any software / service consumed via a simple url
    • Examples: Workday,, Google search, etc.
  • Platform as a Service (PaaS)
    • Development and deployment platform
    • Examples: Amazon Elastic Beanstalk / Relational Database Services (RDS), Google App Engine, etc.
  • Infrastructure as a Service (IaaS)
    • VMs/Containers/NFV as a service
    • Examples: Amazon EC2/ECS, Microsoft Azure, Google Compute Engine (GCE), etc.
Shift in IT focus

Cloud poses an interesting dilemma for IT. They can embrace it, or they can try to provide an alternative. They want to keep the data internal, but need to allow for the self-service, rapid nature of cloud.

This shift forces IT to act more as a legitimate service provider to their end-users (company employees).

The Importance of Latency

The figure below characterizes the various latencies for specific types of I/O:

Item Latency Comments
L1 cache reference 0.5 ns
L2 cache reference 7 ns 14x L1 cache
DRAM access 100 ns 20x L2 cache, 200x L1 cache
3D XPoint based NVMe SSD read 10,000 of ns (expected) 10 us or 0.01 ms
NAND NVMe SSD R/W 20,000 ns 20 us or 0.02 ms
NAND SATA SSD R/W 50,000-60,000 ns 50-60 us or 0.05-0.06 ms
Read 4K randomly from SSD 150,000 ns 150 us or 0.15 ms
P2P TCP/IP latency (phy to phy) 150,000 ns 150 us or 0.15 ms
P2P TCP/IP latency (vm to vm) 250,000 ns 250 us or 0.25 ms
Read 1MB sequentially from memory 250,000 ns 250 us or 0.25 ms
Round trip within datacenter 500,000 ns 500 us or 0.5 ms
Read 1MB sequentially from SSD 1,000,000 ns 1 ms, 4x memory
Disk seek 10,000,000 ns or 10,000 us 10 ms, 20x datacenter round trip
Read 1MB sequentially from disk 20,000,000 ns or 20,000 us 20 ms, 80x memory, 20x SSD
Send packet CA -> Netherlands -> CA 150,000,000 ns 150 ms

(credit: Jeff Dean,

The table above shows that the CPU can access its caches at anywhere from ~0.5-7ns (L1 vs. L2). For main memory, these accesses occur at ~100ns, whereas a local 4K SSD read is ~150,000ns or 0.15ms.

If we take a typical enterprise-class SSD (in this case the Intel S3700 - SPEC), this device is capable of the following:

  • Random I/O performance:
    • Random 4K Reads: Up to 75,000 IOPS
    • Random 4K Writes: Up to 36,000 IOPS
  • Sequential bandwidth:
    • Sustained Sequential Read: Up to 500MB/s
    • Sustained Sequential Write: Up to 460MB/s
  • Latency:
    • Read: 50us
    • Write: 65us

Looking at the Bandwidth

For traditional storage, there are a few main types of media for I/O:

  • Fiber Channel (FC)
    • 4-, 8-, 16- and 32-Gb
  • Ethernet (including FCoE)
    • 1-, 10-Gb, (40-Gb IB), etc.

For the calculation below, we are using the 500MB/s Read and 460MB/s Write BW available from the Intel S3700.

The calculation is done as follows:

numSSD = ROUNDUP((numConnections * connBW (in GB/s))/ ssdBW (R or W))

NOTE: Numbers were rounded up as a partial SSD isn’t possible. This also does not account for the necessary CPU required to handle all of the I/O and assumes unlimited controller CPU power.

Network BW SSDs required to saturate network BW
Controller Connectivity Available Network BW Read I/O Write I/O
Dual 4Gb FC 8Gb == 1GB 2 3
Dual 8Gb FC 16Gb == 2GB 4 5
Dual 16Gb FC 32Gb == 4GB 8 9
Dual 32Gb FC 64Gb == 8GB 16 19
Dual 1Gb ETH 2Gb == 0.25GB 1 1
Dual 10Gb ETH 20Gb == 2.5GB 5 6

As the table shows, if you wanted to leverage the theoretical maximum performance an SSD could offer, the network can become a bottleneck with anywhere from 1 to 9 SSDs depending on the type of networking leveraged

The Impact to Memory Latency

Typical main memory latency is ~100ns (will vary), we can perform the following calculations:

  • Local memory read latency = 100ns + [OS / hypervisor overhead]
  • Network memory read latency = 100ns + NW RTT latency + [2 x OS / hypervisor overhead]

If we assume a typical network RTT is ~0.5ms (will vary by switch vendor) which is ~500,000ns that would come down to:

  • Network memory read latency = 100ns + 500,000ns + [2 x OS / hypervisor overhead]

If we theoretically assume a very fast network with a 10,000ns RTT:

  • Network memory read latency = 100ns + 10,000ns + [2 x OS / hypervisor overhead]

What that means is even with a theoretically fast network, there is a 10,000% overhead when compared to a non-network memory access. With a slow network this can be upwards of a 500,000% latency overhead.

In order to alleviate this overhead, server side caching technologies are introduced.

User vs. Kernel Space

One frequently debated topic is the argument between doing things in kernel vs. in user-space. Here I'll explain what each is and their respective pros/cons.

Any operating system (OS) has two core areas of execution:

  • Kernel space
    • The most priviliged part of the OS
    • Handles scheduling, memory management, etc.
    • Contains the physical device drivers and handles hardware interaction
  • User space
    • "Everything else"
    • This is where most applications and processes live
    • Protected memory and execution

These two spaces work in conjunction for the OS to operate. Now before moving on let's define a few key items:

  • System call
    • A.k.a. kernel call, a request made via interrupt (more here later) from an active process that something be done by the kernel
  • Context switch
    • Shifting the execution from the process to the kernel and vice-versa

For example, take the following use-case of a simple app writing some data to disk. In this the following would take place:

  1. App wants to write data to disk
  2. Invokes a system call
  3. Context switch to kernel
  4. Kernel copies data
  5. Executes write to disk via driver

The following shows a sample of these interactions:

User and Kernel Space Interaction
User and Kernel Space Interaction

Is one better than the other? In reality there are pros and cons for each:

  • User space
    • Very flexible
    • Isolated failure domains (process)
    • Can be inefficient
      • Context switches cost time(~1,000ns)
  • Kernel space
    • Very rigid
    • Large failure domain
    • Can be efficient
      • Reduces context switches

Polling vs. Interrupts

Another core component is how the interaction between the two is handled. There are two key types of interaction:

  • Polling
    • Constantly "poll" e.g. consistently ask for something
    • Examples: Mouse, monitor refresh rate, etc.
    • Requires constant CPU, but much lower latency
    • Eliminates expense of kernel interrupt handler
      • Removes context switch
  • Interrupt
    • "Excuse me, I need foo"
    • Example: Raising hand to ask for something
    • Can be more "CPU efficient", but not necessarily
    • Typically much higher latency

The Move to User Space / Polling

As devices have become far faster (e.g. NVMe, Intel Optane, pMEM), the kernel and device interaction has become a bottleneck. To eliminate these bottlenecks, a lot of vendors are moving things out of the kernel to user space with polling and seeing much better results.

A great example of this are the Intel Storage Performance Development Kit (SPDK) and Data Plane Development Kit (DPDK). These projects are geared at maximizing the performance and reducing latency as much as possible, and have shown great success.

This shift is composed of two core changes:

  1. Moving device drivers to user space (instead of kernel)
  2. Using polling (instead of interrupts)

This enables far superior performance when compared to the kernel based predecessors, as it eliminates:

  • Expensive system calls and the interrupt handler
  • Data copies
  • Context switches

The following shows the device interaction using user space drivers:

User Space and Polling Interaction
User Space and Polling Interaction

In fact, a piece of software Nutanix had developed for their AHV product (vhost-user-scsi), is actually being used by Intel for their SPDK project.

Book of Web-Scale

web·scale - /web ' skãl/ - noun - computing architecture
a new architectural approach to infrastructure and computing.

This section will present some of the core concepts behind “Web-scale” infrastructure and why we leverage them. Before I get started, I just wanted to clearly state the Web-scale doesn’t mean you need to be “web-scale” (e.g. Google, Facebook, or Microsoft).  These constructs are applicable and beneficial at any scale (3-nodes or thousands of nodes).

Historical challenges included:

  • Complexity, complexity, complexity
  • Desire for incremental based growth
  • The need to be agile

There are a few key constructs used when talking about “Web-scale” infrastructure:

  • Hyper-convergence
  • Software defined intelligence
  • Distributed autonomous systems
  • Incremental and linear scale out

Other related items:

  • API-based automation and rich analytics
  • Security as a core tenant
  • Self-healing

The following sections will provide a technical perspective on what they actually mean.


There are differing opinions on what hyper-convergence actually is.  It also varies based on the scope of components (e.g. virtualization, networking, etc.). However, the core concept comes down to the following: natively combining two or more components into a single unit. ‘Natively’ is the key word here. In order to be the most effective, the components must be natively integrated and not just bundled together. In the case of Nutanix, we natively converge compute + storage to form a single node used in our appliance.  For others, this might be converging storage with the network, etc.

What it really means:

  • Natively integrating two or more components into a single unit which can be easily scaled

Benefits include:

  • Single unit to scale
  • Localized I/O
  • Eliminates traditional compute / storage silos by converging them

Software-Defined Intelligence

Software-defined intelligence is taking the core logic from normally proprietary or specialized hardware (e.g. ASIC / FPGA) and doing it in software on commodity hardware. For Nutanix, we take the traditional storage logic (e.g. RAID, deduplication, compression, etc.) and put that into software that runs in each of the Nutanix Controller VMs (CVM) on standard hardware.

Supported Architectures

Nutanix currently supports both x86 and IBM POWER architectures.

What it really means:

  • Pulling key logic from hardware and doing it in software on commodity hardware

Benefits include:

  • Rapid release cycles
  • Elimination of proprietary hardware reliance
  • Utilization of commodity hardware for better economics
  • Lifespan investment protection

To elaborate on the last point: old hardware can run the latest and greatest software. This means that a piece of hardware years into its depreciation cycle can run the latest shipping software and be feature parity with new deployments shipping from the factory.

Distributed Autonomous Systems

Distributed autonomous systems involve moving away from the traditional concept of having a single unit responsible for doing something and distributing that role among all nodes within the cluster.  You can think of this as creating a purely distributed system. Traditionally, vendors have assumed that hardware will be reliable, which, in most cases can be true.  However, core to distributed systems is the idea that hardware will eventually fail and handling that fault in an elegant and non-disruptive way is key.

These distributed systems are designed to accommodate and remediate failure, to form something that is self-healing and autonomous.  In the event of a component failure, the system will transparently handle and remediate the failure, continuing to operate as expected. Alerting will make the user aware, but rather than being a critical time-sensitive item, any remediation (e.g. replace a failed node) can be done on the admin’s schedule.  Another way to put it is fail in-place (rebuild without replace) For items where a “leader” is needed, an election process is utilized. In the event this leader fails a new leader is elected.  To distribute the processing of tasks MapReduce concepts are leveraged.

What it really means:

  • Distributing roles and responsibilities to all nodes within the system
  • Utilizing concepts like MapReduce to perform distributed processing of tasks
  • Using an election process in the case where a “leader” is needed

Benefits include:

  • Eliminates any single points of failure (SPOF)
  • Distributes workload to eliminate any bottlenecks

Incremental and linear scale out

Incremental and linear scale out relates to the ability to start with a certain set of resources and as needed scale them out while linearly increasing the performance of the system.  All of the constructs mentioned above are critical enablers in making this a reality. For example, traditionally you’d have 3-layers of components for running virtual workloads: servers, storage, and network – all of which are scaled independently.  As an example, when you scale out the number of servers you’re not scaling out your storage performance. With a hyper-converged platform like Nutanix, when you scale out with new node(s) you’re scaling out:

  • The number of hypervisor / compute nodes
  • The number of storage controllers
  • The compute and storage performance / capacity
  • The number of nodes participating in cluster wide operations

What it really means:

  • The ability to incrementally scale storage / compute with linear increases to performance / ability

Benefits include:

  • The ability to start small and scale
  • Uniform and consistent performance at any scale

Making Sense of It All

In summary:

  1. Inefficient compute utilization led to the move to virtualization
  2. Features including vMotion, HA, and DRS led to the requirement of centralized storage
  3. VM sprawl led to increased load and contention on storage
  4. SSDs came in to alleviate the issues but changed the bottleneck to the network / controllers
  5. Cache / memory accesses over the network face large overheads, minimizing their benefits
  6. Array configuration complexity still remains the same
  7. Server side caches were introduced to alleviate the load on the array / impact of the network, however introduces another component to the solution
  8. Locality helps alleviate the bottlenecks / overheads traditionally faced when going over the network
  9. Shifts the focus from infrastructure to ease of management and simplifying the stack
  10. The birth of the Web-Scale world!

Part 1: Core

Book of Basics

As described in the generalized 'Book of Webscale', Nutanix leverages these principles throughout the stack. This section will cover these basics as well as the core architectural concepts.

Strategy and Vision

When Nutanix was conceived it was focused on one goal:

Make infrastructure computing invisible, anywhere.

This simplicity was to be achieved by focus in three core areas:

  1. Enable choice and portability (HCI/Cloud/Hypervisor)
  2. Simplify the "stack" through convergence, abstraction and intelligent software (AOS)
  3. Provide an intuitive user interface (UI) through focus on user experience (UX) and design (Prism)
HCI/Cloud/Hypervisor: "The Choice"

Though we started with a single hardware platform (NX) supporting a single hypervisor (ESXi), we've always known we were more than a single hypervisor/platform/cloud company. This was one of the reasons we chose to build our own UI from scratch vs. run as a plug-in in vCenter, run as a VM vs. natively in the kernel (a lot more reasons there), etc. Why you may ask? Choice.

Not one hypervisor, platform, or cloud will fit all customer's needs. By supporting multiple under the same platform we give the customer choice and leverage. By giving them the ability to move between them, we give them flexibility. All delivered with the same experience since it's all part of the Nutanix platform.

We now have support for over 12 different hardware platforms (direct/OEM/third-party), multiple hypervisors (AHV, ESXi, Hyper-V, etc.), and expanding integrations with all of the major cloud vendors (AWS, Azure, GCP). This allows the customer to choose what is best for them, as well as use this for negotiations purposes between vendors.

NOTE: Platform is one key word that is used throughout the section and in general. We're not trying to build one-off products, we're building a platform.

The following shows a high-level architecture of the Nutanix platform:

Nutanix Platform - Architecture
Nutanix Platform - Architecture
AOS + AHV/Hypervisor: "The Runtime"

We started this journey by simplifying storage with a feature called the Distributed Storage Fabric (DSF then known as the Nutanix Distributed Filesystem aka NDFS), which combined local storage resources with intelligent software to provide "centralized storage" like capabilities.

Over the years, we've added a great deal of features and capabilities. To simplify things I've broken these down into two core areas:

  1. Core Services
    • Foundational services
  2. Platform Services
    • Services building upon core services providing additional capabilities/services

The core provides the foundational services and components that facilitate the running of workloads (VMs/Containers) and other higher-level Nutanix services. In the beginning this was just the DSF product, however we continue to expand the platform's capabilities to help simplify and abstract the stack.

The following shows a high-level view of the AOS core platform:

Nutanix Platform - AOS Core
Nutanix Platform - AOS Core

Over the years this has expanded into things like abstracting virtualization (we believe this should be something transparent and part of the system) by introducing our own hypervisor (AHV), simplifying upgrades, and providing other essential services like security and encryption.

With these capabilities we solved for a lot of the infrastructure level issues, but we didn't stop there. People still needed additional services like file shares, object storage, or containers.

Rather than requiring customers to use other vendors and products for some services we figured which ones we should partner on and which ones we should build ourselves. For backup we partnered with vendors like Veeam and Hycu, for others like file and object services we built them as services into the platform.

The following shows a high-level view of the Nutanix platform services:

Nutanix Platform - Services
Nutanix Platform - Services
Prism: "The Interface"
Nutanix Platform - Prism
Nutanix Platform - Prism

Simply put, apply some of the design principles fostered by companies like Apple focused on simplicity, consistency and intuitiveness. Since the beginning we've invested significant time and effort on the Nutanix product's "front-end". Rather than being an afterthought, the UI/UX and design teams have always been pushing the boundaries. Case in point, we were one of the first enterprise software companies (besides the SaaS players), to have the management UI be written in HTML5.

Another core item here is the focus on providing a single interface for the platform and keeping the experience consistent throughout that. Our goal is to converge UIs like we've converged infrastructure. We want to Prism to be a single interface allowing you to manage and consume the Nutanix platform, whether that is managing virtualization in your datacenter, Desktops-as-a-Service in the cloud, or providing spend visibility.

This is important as we continue to expand the platform through feature / service creation and acquisition. Rather than bolting the new capabilities on, we'd rather spend the time to natively integrate them into the platform. It is a slower process, but in the long run it keeps the experience consistent and reduces risk.

Nutanix: The Platform

To summarize, our vision is simple: "one platform, any app, any location". NOTE: I did steal this from the marketing folks, but it fits perfectly and states our purpose succinctly.

Nutanix Platform - Architecture
Nutanix Platform - Architecture

This has been our goal from close to the beginning. Testament to this, here's an image I created in 2014 time frame to talk about the Nutanix platform architecture. As you can see not much has changed, we just continue expanding and working towards this goal.

Nutanix Platform - Circa 2014
Nutanix Platform - Circa 2014

Products & Platforms

Over the years the Nutanix Platform capability set and services has grown substantially. Over the years this has evolved to simplify and abstract virtualization, automate upgrades and operations and much more. This section will cover the current portfolio and partnerships. NOTE: refer to the Nutanix website for the latest portfolio and offerings.

Over the years as the product portfolio has grown, rather than talking about products, I'd rather focus on the results and the journey towards achieving those. The following steps cover the customer "journey" and results Nutanix can help them achieve.

Step 1: Datacenter Modernization (Core)

Core includes the foundational Nutanix products facilitating the migration from complex 3-tier infrastructure to a simple HCI platform. AOS provides all of the core services (storage, upgrades, replication, etc.), Prism provides the control plane and management console and AHV provides a free virtualization platform (NOTE: you can also use ESXi and Hyper-V).

Core capabilities include:

  • Core Platforms (HCI)
  • Storage Services
  • Virtualization
  • Centralized Management & Operations
  • Upgrades
  • Replication / DR
Products Ecosystem - Core
Products Ecosystem - Core
Step 2: Enabling Private Cloud (Essentials)

Essentials is focused on providing the capabilities to enable Core infrastructure to be consumed like a private cloud. Flow provides network segmentation and security, Files provides file services and Calm provides self-service, quotas and orchestration capabilities.

Essentials capabilities include:

  • Advanced Analytics & Anomaly Detection
  • Automation & Orchestration
  • Self-service Portal (SSP) and Quotas
  • Microsegmentation
  • File Services
Products Ecosystem - Private Cloud
Products Ecosystem - Private Cloud
Step 3: Enabling Hybrid Cloud (Enterprise)

Enterprise is focused on providing the ability to migrate workloads between clouds and cloud services. This includes features like Beam which is focused on cost governance and compliance across cloud and on-premise deployments as well as other cloud services like Frame (DaaS) and Xi Leap (DRaaS).

Enterprise capabilities include:

  • Policy Driven DR / Run-book Automation
  • DRaaS
  • Hybrid Cloud Cost Governance & Compliance
  • Desktops As-A-Service (DaaS)
  • Database As-A-Service (RDS)
  • Kubernetes / Docker Services
  • Object Storage
  • Block Services
Products Ecosystem - Hybrid Cloud
Products Ecosystem - Hybrid Cloud

Nutanix currently supports the following platforms:

  • Nutanix Appliances
    • NX (Supermicro)
  • OEM Appliances
    • Nutanix on HPE ProLiant DX
    • Nutanix on Lenovo HX
    • Nutanix on Fujitsu XF
    • Nutanix on Dell XC
    • Nutanix on Inspur InMerge
  • Third-Party Server Support
    • Nutanix on HPE Apollo
    • Nutanix on Cisco UCS
    • Nutanix on Intel Data Center Blocks
    • Nutanix Tactical and Ruggedized platforms on Klas

Hyperconverged Platform

For a video explanation you can watch the following video: LINK

There are a few core structs for hyperconverged systems:

  • Must converge and collapse the computing stack (e.g. compute + storage)
  • Must shard (distribute) data and services across nodes in the system
  • Must appear and provide the same capabilities as centralized storage (e.g. HA, live-migration, etc.)
  • Must keep data as close to the execution (compute) as possible (Importance of Latency)
  • Should be hypervisor agnostic
  • Should be hardware agnostic

The following figure shows an example of a typical 3-tier stack vs. hyperconverged:

3-Tier vs. HCI
3-Tier vs. HCI

As you can see, the hyperconverged system does the following:

  • Virtualizes and moves the controllers to the host
  • Provides core services and logic through software
  • Distributes (shards) data across all nodes in the system
  • Moves the storage local to the compute

The Nutanix solution is a converged storage + compute solution which leverages local components and creates a distributed platform for running workloads.

Each node runs an industry-standard hypervisor (ESXi, AHV, and Hyper-V) and the Nutanix Controller VM (CVM).  The Nutanix CVM is what runs the Nutanix software and serves all of the I/O operations for the hypervisor and all VMs running on that host.

The following figure provides an example of what a typical node logically looks like:

Converged Platform
Converged Platform

The Nutanix CVM is responsible for the core Nutanix platform logic and handles services like:

  • Storage I/O & transforms (Deduplication, Compression, EC)
  • UI / API
  • Upgrades
  • DR / Replication
  • Etc.

NOTE: Some services / features will spawn additional helper VMs or use the Microservices Platform (MSP). For example, Nutanix Files will deploy additional VMs, whereas Nutanix Objects will deploy VMs for MSP and leverage those.

For the Nutanix units running VMware vSphere, the SCSI controller, which manages the SSD and HDD devices, is directly passed to the CVM leveraging VM-Direct Path (Intel VT-d).  In the case of Hyper-V, the storage devices are passed through to the CVM.

Virtualizing the Controller

The key reasons for running the Nutanix controllers as VMs in user-space really come down to two core areas:

  1. Mobility
  2. Resiliency
  3. Maintenance / Upgrades
  4. Performance, yes really

Since the beginning we knew we were more than a single platform company. In that sense, choice has always been a big thing for us, whether it is with hardware, cloud or hypervisor vendors.

By running as a VM in user-space it decouples the Nutanix software from the underlying hypervisor and hardware platforms. This enabled us to rapidly add support for other hypervisor while keeping the core code base the same across all operating environments (on-premise & cloud). Additionally, it gave us flexibility to not be bound to vendor specific release cycles

Due to the nature of running as a VM in user-space, we can elegantly handle things like upgrades or CVM "failures" as they are outside of the hypervisor. For example, if there is some catastrophic issue where a CVM goes down, the whole node still continues to operate with storage I/Os and services coming from other CVMs in the cluster. During a AOS (Nutanix Core Software) upgrade, we can reboot the CVM without any impact to the workloads running on that host.

But isn't being in the kernel is so much faster? Simple answer, NO.

A common discussion topic is the debate around being in the kernel vs. in user-space. As a matter of background, I recommend reading the 'User vs. Kernel Space' section which covers what both actually are and the pros and cons of each.

To summarize, there are two areas of execution in an operating system (OS): the kernel (privileged core of the OS where drivers may sit) and user space (where applications/processes sit). Traditionally moving between user-space and the kernel (aka context switch) can be expensive in terms of CPU and time (~1,000ns / context switch).

The debate is that being in the kernel is always better / faster. Which is false. No matter what there will always be context switches in the guest VM's OS.

Distributed System

There are three core principles for distributed systems:

  1. Must have no single points of failure (SPOF)
  2. Must not have any bottlenecks at any scale (must be linearly scalable)
  3. Must leverage concurrency (MapReduce)

Together, a group of Nutanix nodes forms a distributed system (Nutanix cluster) responsible for providing the Prism and AOS capabilities. All services and components are distributed across all CVMs in a cluster to provide for high-availability and linear performance at scale.

The following figure shows an example of how these Nutanix nodes form a Nutanix cluster:

Distributed Storage Fabric Overview
Nutanix Cluster - Distributed System

These techniques are also applied to metadata and data alike. By ensuring metadata and data is distributed across all nodes and all disk devices we can ensure the highest possible performance during normal data ingest and re-protection.

This enables our MapReduce Framework (Curator) to leverage the full power of the cluster to perform activities concurrently. Sample activities include that of data re-protection, compression, erasure coding, deduplication, etc.

The following figure shows how the % of work handled by each node drastically decreases as the cluster scales:

Work Distribution - Cluster Scale
Work Distribution - Cluster Scale

Key point: As the number of nodes in a cluster increases (cluster scaling), certain activities actually become more efficient as each node is handling only a fraction of the work.


There are four core principles for software definition systems:

  • Must provide platform mobility (hardware, hypervisor)
  • Must not be reliant on any custom hardware
  • Must enable rapid speed of development (features, bug fixes, security patches)
  • Must take advantage of Moore's Law

As mentioned above (likely numerous times), the Nutanix platform is a software-based solution which ships as a bundled software + hardware appliance.  The controller VM is where the vast majority of the Nutanix software and logic sits and was designed from the beginning to be an extensible and pluggable architecture. A key benefit to being software-defined and not relying upon any hardware offloads or constructs is around extensibility.  As with any product life cycle, advancements and new features will always be introduced. 

By not relying on any custom ASIC/FPGA or hardware capabilities, Nutanix can develop and deploy these new features through a simple software update.  This means that the deployment of a new feature (e.g., deduplication) can be deployed by upgrading the current version of the Nutanix software.  This also allows newer generation features to be deployed on legacy hardware models. For example, say you’re running a workload running an older version of Nutanix software on a prior generation hardware platform (e.g., 2400).  The running software version doesn’t provide deduplication capabilities which your workload could benefit greatly from.  To get these features, you perform a rolling upgrade of the Nutanix software version while the workload is running, and you now have deduplication.  It’s really that easy.

Similar to features, the ability to create new “adapters” or interfaces into DSF is another key capability.  When the product first shipped, it solely supported iSCSI for I/O from the hypervisor, this has now grown to include NFS and SMB.  In the future, there is the ability to create new adapters for various workloads and hypervisors (HDFS, etc.).  And again, all of this can be deployed via a software update. This is contrary to most legacy infrastructures, where a hardware upgrade or software purchase is normally required to get the “latest and greatest” features.  With Nutanix, it’s different. Since all features are deployed in software, they can run on any hardware platform, any hypervisor, and be deployed through simple software upgrades.

The following figure shows a logical representation of what this software-defined controller framework looks like:

Software-Defined Controller Framework
Software-Defined Controller Framework

Cluster Components

The user-facing Nutanix product is extremely simple to deploy and use. This is primarily possible through abstraction and a lot of automation / integration in the software.

The following is a detailed view of the main Nutanix Cluster components (don't worry, no need to memorize or know what everything does):

Nutanix Cluster Components
Nutanix Cluster Components
  • Key Role: Distributed metadata store
  • Description: Cassandra stores and manages all of the cluster metadata in a distributed ring-like manner based upon a heavily modified Apache Cassandra.  The Paxos algorithm is utilized to enforce strict consistency.  This service runs on every node in the cluster.  The Cassandra is accessed via an interface called Medusa.
  • Key Role: Cluster configuration manager
  • Description: Zookeeper stores all of the cluster configuration including hosts, IPs, state, etc. and is based upon Apache Zookeeper.  This service runs on three nodes in the cluster, one of which is elected as a leader.  The leader receives all requests and forwards them to its peers.  If the leader fails to respond, a new leader is automatically elected.   Zookeeper is accessed via an interface called Zeus.
  • Key Role: Data I/O manager
  • Description: Stargate is responsible for all data management and I/O operations and is the main interface from the hypervisor (via NFS, iSCSI, or SMB).  This service runs on every node in the cluster in order to serve localized I/O.
  • Key Role: MapReduce cluster management and cleanup
  • Description: Curator is responsible for managing and distributing tasks throughout the cluster, including disk balancing, proactive scrubbing, and many more items.  Curator runs on every node and is controlled by an elected Curator Leader who is responsible for the task and job delegation.  There are two scan types for Curator, a full scan which occurs around every 6 hours and a partial scan which occurs every hour.
  • Key Role: UI and API
  • Description: Prism is the management gateway for component and administrators to configure and monitor the Nutanix cluster.  This includes Ncli, the HTML5 UI, and REST API.  Prism runs on every node in the cluster and uses an elected leader like all components in the cluster.
  • Key Role: Cluster component & service manager
  • Description:  Genesis is a process which runs on each node and is responsible for any services interactions (start/stop/etc.) as well as for the initial configuration.  Genesis is a process which runs independently of the cluster and does not require the cluster to be configured/running.  The only requirement for Genesis to be running is that Zookeeper is up and running.  The cluster_init and cluster_status pages are displayed by the Genesis process.
  • Key Role: Job and task scheduler
  • Description: Chronos is responsible for taking the jobs and tasks resulting from a Curator scan and scheduling/throttling tasks among nodes.  Chronos runs on every node and is controlled by an elected Chronos Leader that is responsible for the task and job delegation and runs on the same node as the Curator Leader.
  • Key Role: Replication/DR manager
  • Description: Cerebro is responsible for the replication and DR capabilities of DSF.  This includes the scheduling of snapshots, the replication to remote sites, and the site migration/failover.  Cerebro runs on every node in the Nutanix cluster and all nodes participate in replication to remote clusters/sites.
  • Key Role: vDisk configuration manager
  • Description: Pithos is responsible for vDisk (DSF file) configuration data.  Pithos runs on every node and is built on top of Cassandra.

Non-Disruptive Upgrades

In the 'Nutanix Software Upgrade' and 'Hypervisor Upgrade' sections in the Book of Prism, we highlighted the steps used to perform an upgrade of AOS and hypervisor versions. This section will cover the techniques allowing us to perform different types of upgrades in a non-disruptive manner.

AOS Upgrades

For an AOS upgrade there are a few core steps that are performed:

1 - Pre-upgrade Checks

During the pre-upgrade checks, the following items are verified. NOTE: This must complete successfully before an upgrade can continue.

  • Check version compatibility between AOS, hypervisor versions
  • Check cluster health (cluster status, free space, and component checks (e.g. Medusa, Stargate, Zookeeper, etc.)
  • Check network connectivity between all CVM and Hypervisors
2 - Upload upgrade software to 2 nodes

Once the pre-upgrade checks have been completed, the system will upload the upgrade software binaries to two nodes in the cluster. This is done for fault-tolerance and to ensure if one CVM is rebooting the other is available for others to pull the software from.

3 - Stage Upgrade Software

Once the software has been uploaded to two CVMs, all CVMs will stage the upgrade in parallel.

The CVMs have two partitions for AOS versions:

  • Active partition (the currently running version)
  • Passive partition (where upgrades are staged)

When an AOS upgrade occurs, we perform the upgrade on the non-active partition. When the upgrade token is received it will mark the upgraded partition as the active partition and reboot the CVM into the upgraded version. This is similar to a bootbank / altbootbank.

NOTE: the upgrade token is passed between nodes iteratively. This ensures only one CVM reboots at a time. Once the CVM reboots and is stable (check service status and communication) the token can be passed to the next CVM until all CVMs have been upgraded.

Upgrade Error Handling

A common question is what happens if the upgrade is unsuccessful or has an issue partially through the process?

In the event some upgrade issue occurs we will stall the upgrade and not progress. NOTE: this is a very infrequent occurrence as pre-upgrade checks will find most issues before the upgrade actually begins. However, in the event the pre-upgrade checks succeed and some issue occurs during the actual upgrade, there will be no impact to workloads and user I/O running on the cluster.

The Nutanix software is designed to work indefinitely in a mixed mode between supported upgrade versions. For example, if the cluster is running and is upgrading to the system can run indefinitely with CVMs on both versions. This is actually what occurs during the upgrade process.

For example, if you have a 4 node cluster on and start the upgrade to, when the first node upgrades it will be running while the others are on This process will continue and CVMs will reboot into as they receive the upgrade token.

Foundation (Imaging)


Foundation is a Nutanix provided tool leveraged for bootstrapping, imaging and deployment of Nutanix clusters. The imaging process will install the desired version of the AOS software as well as the hypervisor of choice.

By default Nutanix nodes ship with AHV pre-installed, to leverage a different hypervisor type you must use foundation to re-image the nodes with the desired hypervisor. NOTE: Some OEMs will ship directly from the factory with the desired hypervisor.

The figure shows a high level view of the Foundation architecture:

Foundation - Architecture
Foundation - Architecture

As of 4.5, Foundation is built in to the CVMs to simplify configuration. The installer store is a directory for storing uploaded images, these can be used for the initial imaging as well as cluster expansion when imaging is required.

The Foundation Discovery Applet (which can be found HERE) is responsible for discovering nodes and allowing the user to select a node to connect to. Once the user has selected a node to connect to, the applet will proxy localhost:9442 IPv4 to the CVM's IPv6 link-local address on port 8000.

The figure shows a high level view of the applet architecture:

Foundation - Applet Architecture
Foundation - Applet Architecture

NOTE: the discovery applet is merely a means of discovery and proxy to the Foundation service which runs on the nodes. All of the imaging and configuration is handled by the Foundation service, not the applet.

Pro tip

If you're on a different network (L2) than your target Nutanix nodes (e.g. over the WAN) you can connect directly to the Foundation service on the CVM if it has an IPv4 address assigned (instead of using the discovery applet).

To directly connect browse to <CVM_IP>:8000/gui/index.html


The Foundation tool has the following configuration inputs (below). A typical deployment requires 3 IP addresses per node (hypervisor, CVM, remote management (e.g. IPMI, iDRAC, etc.)). In addition to the per node addresses, it is recommended to set a Cluster and Data Services IP addresses.

  • Cluster
    • Name
    • IP*
    • NTP*
    • DNS*
  • CVM
    • IP per CVM
    • Netmask
    • Gateway
    • Memory
  • Hypervisor
    • IP per hypervisor host
    • Netmask
    • Gateway
    • DNS*
    • Hostname prefix
  • IPMI*
    • IP per node
    • Netmask
    • Gateway

NOTE: Items marked with '*' are optional but highly advisable

System Imaging and Deployment

The first step is to connect to the Foundation UI which can be done via the discovery applet (if on same L2, node IPs unecessary):

Foundation - Discovery Applet
Foundation - Discovery Applet

If you can't find the desired node, make sure you're on the same L2 network.

After connecting into the selected node's Foundation instance the main Foundation UI will appear:

Foundation - Discovery Page
Foundation - Discovery Page

This will show all of the discovered nodes and their chassis. Select the desired nodes to form the cluster and click 'Next'

Foundation - Node Selection
Foundation - Node Selection

The next page prompts for the cluster and network inputs:

Foundation - Cluster Information
Foundation - Cluster Information
Foundation - Network Applet
Foundation - Network Information

Once the details have been input, click 'Next'

Next we'll input the node details and IP addresses:

Foundation - Node Setup
Foundation - Node Setup

You can manually override the hostname and IP addresses if necessary:

Foundation - Hostname and IP
Foundation - Hostname and IP

Click 'Validate Network' to validate network configuration and proceed. This will check for IP address conflicts and ensure connectivity.

Foundation - Network Validation
Foundation - Network Validation

Once network validation has completed successfully we'll now proceed to selecting the desired images.

To upgrade AOS to a newer version than currently on the CVM, download it from the portal and upload the Tarball. Once we have the desired AOS image, we'll select the hypervisor.

For AHV, the image is built-in to the AOS image. For others you must upload the desired hypervisor image. NOTE: make sure the AOS and hypervisor versions are on the compatibility matrix (LINK).

Once we have the desired images, click 'Create':

Foundation - Select Images
Foundation - Select Images

If imaging is not necessary you can also click 'Skip' to skip the imaging process. This will not re-image the hypervisor or Nutanix cluster, but just configure the cluster (e.g. IP addresses, etc.).

Foundation will then proceed with the imaging (if necessary) and cluster creation process.

Foundation - Cluster Creation Process
Foundation - Cluster Creation Process

Once the creation is successful you'll get a completion screen:

Foundation - Cluster Creation Complete
Foundation - Cluster Creation Complete

At this point you can now log into any CVM or the Cluster IP and start using the Nutanix platform!

Drive Breakdown

In this section, I’ll cover how the various storage devices (Performance (NVMe/SSD) / Capacity (SSD/HDD) are broken down, partitioned, and utilized by the Nutanix platform. NOTE: All of the capacities used are in Base2 Gibibyte (GiB) instead of the Base10 Gigabyte (GB).  Formatting of the drives with a filesystem and associated overheads has also been taken into account.

Performance Devices

Performance devices are the highest performance device in a node. These can be NVMe or a mix of NVMe and SSD devices. They store a few key items which are explained in greater detail above:

  • Nutanix Home (CVM core)
  • Metadata (Cassandra / AES storage)
  • OpLog (persistent write buffer)
  • Extent Store (persistent storage)

The following figure shows an example of the storage breakdown for a Nutanix node’s performance device:

Performance Drive Breakdown
Performance Drive Breakdown

Graphics and proportions aren’t drawn to scale.  When evaluating the Remaining GiB capacities, do so from the top down.  For example, the Remaining GiB to be used for the OpLog calculation would be after Nutanix Home and Cassandra have been subtracted from the formatted SSD capacity.

Nutanix Home is mirrored across the first two SSDs to ensure availability and has a 60GiB reservation for two devices.

As of 5.0 Cassandra is sharded across multiple SSDs in the node (currently up to 4) with an initial reservation of 15GiB per SSD (can leverage some Stargate SSD if metadata usage increases). In dual SSD systems, metadata will be mirrored between the SSDs. The metadata reservation per SSD is 15 GiB (30GiB for dual SSD, 60GiB for 4+ SSD).

Prior to 5.0, Cassandra was on the first SSD by default, if that SSD fails the CVM will be restarted and Cassandra storage will then be on the 2nd. In this case the metadata reservation per SSD is 30 GiB for the first two devices.

The OpLog is distributed among all SSD devices up to a max of 12 per node (Gflag: max_ssds_for_oplog). If NVMe devices are available, OpLog will be placed on those devices instead of SATA SSD.

The OpLog reservation per disk can be calculated using the following formula: MIN(((Max cluster RF/2)*400 GiB)/ numDevForOplog), ((Max cluster RF/2)*25%) x Remaining GiB). NOTE: The sizing for OpLog is done dynamically as of release 4.0.1 which will allow the extent store portion to grow dynamically.  The values used are assuming a completely utilized OpLog.

For example, in a RF2 (FT1) cluster with 8 SSD devices that are 1TB the result would be:

  • MIN(((2/2)*400 GiB)/ 8), ((2/2)*25%) x ~900GiB) == MIN(50, 225) == 50 GiB reserved for Oplog per device.

For a RF3 (FT2) cluster this would be:

  • MIN(((3/2)*400 GiB)/ 8), ((3/2)*25%) x ~900GiB) == MIN(75, 337) == 75 GiB reserved for Oplog per device.

For a RF2 (FT1) cluster with 4 NVMe and 8 SSD devices that are 1TB the result would be:

  • MIN(((2/2)*400 GiB)/ 4), ((2/2)*25%) x ~900GiB) == MIN(100, 225) == 100 GiB reserved for Oplog per device.

The Extent Store capacity would be the remaining capacity after all other reservations are accounted for.

HDD Devices

Since HDD devices are primarily used for bulk storage, their breakdown is much simpler:

  • Curator Reservation (Curator storage)
  • Extent Store (persistent storage)
HDD Drive Breakdown
HDD Drive Breakdown

Book of Prism

prism - /'prizɘm/ - noun - control plane
one-click management and interface for datacenter operations.

Design Methodology and Iterations

Building a beautiful, empathetic and intuitive product is core to the Nutanix platform and something we take very seriously. This section will cover our design methodologies and how we iterate on design. More coming here soon!

In the meantime feel free to check out this great post on our design methodology and iterations by our Product Design Lead, Jeremy Sallee (who also designed this) -

You can download the Nutanix Visio stencils here:


Prism is a distributed resource management platform which allows users to manage and monitor objects and services across their Nutanix environment, whether hosted locally or in the cloud.

These capabilities are broken down into two key categories:

  • Interfaces
    • HTML5 UI, REST API, CLI, PowerShell CMDlets, etc.
  • Management Capabilities
    • Platform management, VM / Container CRUD, policy definition and compliance, service design and status, analytics and monitoring

The following figure illustrates the conceptual nature of Prism as part of the Nutanix platform:

High-Level Prism Architecture
High-Level Prism Architecture

Prism is broken down into two main components:

  • Prism Central (PC)
    • Multi-cluster manager responsible for managing multiple Nutanix Clusters to provide a single, centralized management interface.  Prism Central is an optional software appliance (VM) which can be deployed in addition to the AOS Cluster (can run on it).
    • 1-to-many cluster manager
  • Prism Element (PE)
    • Localized cluster manager responsible for local cluster management and operations.  Every Nutanix Cluster has Prism Element built-in.
    • 1-to-1 cluster manager

The figure shows an image illustrating the conceptual relationship between Prism Central and Prism Element:

Prism Architecture
Prism Architecture
Pro tip

For larger or distributed deployments (e.g. more than one cluster or multiple sites) it is recommended to use Prism Central to simplify operations and provide a single management UI for all clusters / sites.

Prism Services

A Prism service runs on every CVM with an elected Prism Leader which is responsible for handling HTTP requests.  Similar to other components which have a Leader, if the Prism Leader fails, a new one will be elected.  When a CVM which is not the Prism Leader gets a HTTP request it will permanently redirect the request to the current Prism Leader using HTTP response status code 301.

Here we show a conceptual view of the Prism services and how HTTP request(s) are handled:

Prism Services - Request Handling
Prism Services - Request Handling
Prism ports

Prism listens on ports 80 and 9440, if HTTP traffic comes in on port 80 it is redirected to HTTPS on port 9440.

When using the cluster external IP (recommended), it will always be hosted by the current Prism Leader.  In the event of a Prism Leader failure the cluster IP will be assumed by the newly elected Prism Leader and a gratuitous ARP (gARP) will be used to clean any stale ARP cache entries.  In this scenario any time the cluster IP is used to access Prism, no redirection is necessary as that will already be the Prism Leader.

Pro tip

You can determine the current Prism leader by running 'curl localhost:2019/prism/leader' on any CVM.

Authentication and Access Control (RBAC)


Prism currently supports integrations with the following authentication providers:

  • Prism Element (PE)
    • Local
    • Active Directory
    • LDAP
  • Prism Central (PC)
    • Local
    • Active Directory
    • LDAP
    • SAML Authn (IDP)

SAML Authn allows Prism to integrate with external identity providers (IDP) that are SAML compliant (e.g. Okta, ADFS, etc.).

This also allows you to leverage the multi-factor authentication (MFA) / two-factor authentication (2FA) capabilities these providers support for users logging into Prism.

Access Control

Coming soon!


Prism is fairly straight forward and simple to use, however we'll cover some of the main pages and basic usage.

Prism Central (if deployed) can be accessed using the IP address specified during configuration or corresponding DNS entry.  Prism Element can be accessed via Prism Central (by clicking on a specific cluster) or by navigating to any Nutanix CVM or cluster IP (preferred).

Once the page has been loaded you will be greeted with the Login page where you will use your Prism or Active Directory credentials to login.

Prism Login Page

Upon successful login you will be sent to the dashboard page which will provide overview information for managed cluster(s) in Prism Central or the local cluster in Prism Element.

Prism Central and Prism Element will be covered in more detail in the following sections.

Prism Central

The figure shows a sample Prism Central dashboard where multiple clusters can be monitored / managed:

Prism Central - Dashboard
Prism Central - Dashboard

From here you can monitor the overall status of your environment, and dive deeper if there are any alerts or items of interest.

Prism Central contains the following main pages (NOTE: Search is the preferred / recommended method to navigation):

  • Home Page
    • Environment wide monitoring dashboard including detailed information on service status, capacity planning, performance, tasks, etc.  To get further information on any of them you can click on the item of interest.
  • Virtual Infrastructure
    • Virtual entities (e.g. VMs, containers, Images, categories, etc.)
  • Policies
    • Policy management and creation (e.g. security (FLOW), Protection (Backup/Replication), Recovery (DR), NGT)
  • Hardware
    • Physical devices management (e.g. clusters, hosts, disks, GPU)
  • Activity
    • Environment wide alerts, events and tasks
  • Operations
    • Operations dashboards, reporting and actions (X-Play)
  • Administration
    • Environment construct management (e.g. users, groups, roles, availability zones)
  • Services
    • Add-on service management (e.g. Calm, Karbon)
  • Settings
    • Prism Central configuration

To access the menu click on the hamburger icon::

Prism Central - Hamburger Menu
Prism Central - Hamburger

The menu expands to display the available options:

Prism Central - Menu Bar
Prism Central - Menu Bar


Search is now the primary mechanism for Navigating the Prism Central UI (menus are still available).

To use the search bar to navigate you can use the search bar in the top left corner next to the menu icon.

Prism Central - Search
Prism Central - Search
Search Semantics

PC Search allows for a great deal to semantics to be leveraged, some examples include:

Rule Example
Entity type vms
Entity type + metric perspective (io, cpu, memory) vms io
Entity type + alerts vm alerts
Entity type + alerts + alert filters vm alerts severity=critical
Entity type + events vm events
Entity type + events + event filters vm events classification=anomaly
Entity type + filters (both metric and attribute) vm “power state”=on
Entity type + filters + metric perspective (io, cpu, memory) vm “power state”=on io
Entity type + filters + alerts vm “power state”=on alerts
Entity type + filters + alerts + (alert filters) vm “power state”=on alerts severity=critical
Entity type + filters + events vm “power state”=on events
Entity type + filters + events + event filters vm “power state”=on events classification=anomaly
Entity instance (name, ip address, disk serial etc) vm1,, BHTXSPWRM
Entity instance + Metric perspective (io, cpu, memory) vm1 io
Entity instance + alerts vm1 alerts
Entity instance + alerts + alert filters vm1 alerts severity=critical
Entity instance + events vm1 events
Entity instance + events + event filters vm1 events classification=anomaly
Entity instance + pages vm1 nics, c1 capacity
Parent instance + entity type c1 vms
Alert title search Disk bad alerts
Page name search Analysis, tasks

The prior is just a small subset of the semantics, the best way to get familiar with them is to give it a shot!

Prism Element

Prism Element contains the following main pages:

  • Home Page
    • Local cluster monitoring dashboard including detailed information on alerts, capacity, performance, health, tasks, etc.  To get further information on any of them you can click on the item of interest.
  • Health Page
    • Environment, hardware and managed object health and state information.  Includes NCC health check status as well.
  • VM Page
    • Full VM management, monitoring and CRUD (AOS)
  • Storage Page
    • Container management, monitoring and CRUD
  • Hardware
    • Server, disk and network management, monitoring and health.  Includes cluster expansion as well as node and disk removal.
  • Data Protection
    • DR, Cloud Connect and Metro Availability configuration.  Management of PD objects, snapshots, replication and restore.
  • Analysis
    • Detailed performance analysis for cluster and managed objects with event correlation
  • Alerts
    • Local cluster and environment alerts

The home page will provide detailed information on alerts, service status, capacity, performance, tasks, and much more.  To get further information on any of them you can click on the item of interest.

The figure shows a sample Prism Element dashboard where local cluster details are displayed:

Prism Element - Dashboard
Prism Element - Dashboard

Keyboard Shortcuts

Accessibility and ease of use is a very critical construct in Prism.  To simplify things for the end-user a set of shortcuts have been added to allow users to do everything from their keyboard.

The following characterizes some of the key shortcuts:

Change view (page context aware):

  • O - Overview View
  • D - Diagram View
  • T - Table View

Activities and Events:

  • A - Alerts
  • P - Tasks

Drop down and Menus (Navigate selection using arrow keys):

  • M - Menu drop-down
  • S - Settings (gear icon)
  • F - Search bar
  • U - User drop down
  • H - Help

Features and Usage

In the following sections we'll cover some of the typical Prism uses as well as some common troubleshooting scenarios.

Anomaly Detection

In the world of IT operations there is a lot of noise. Traditionally systems would generate a great deal of alerts, events and notifications, often leading to the operator either a) not seeing critical alerts since they are lost in the noise or b) disregarding the alerts/events.

With Nutanix Anomaly Detection the system will monitor seasonal trends for time-series data (e.g. CPU usage, memory usage, latency, etc.) and establish a "band" of expected values. Only values that hit outside the "band" will trigger an event / alert. You can see the anomaly events / alerts from any entity or events page.

The following chart shows a lot of I/O and disk usage anomalies as we were performing some large batch loads on these systems:

Prism - Anomaly Chart
Prism - Anomaly Chart

The following image shows the time-series values for a sample metric and the established "band":

Prism - Anomaly Band
Prism - Anomaly Band

This reduces unnecessary alerts as we don't want alerts for a "normal" state. For example, a database system will normally run at >95% memory utilization due to caching, etc. In the event this drops to say 10% that would be an anomaly as something may be wrong (e.g. database service down).

Another example would be how some batched workloads run on the weekend. For example, I/O bandwidth may be low during the work week, however on the weekends when some batch processes run (e.g. backups, reports, etc.) there may be a large spike in I/O. The system would detect the seasonality of this and bump up the band during the weekend.

Here you can see an anomaly event has occured as the values are outside the expected band:

Prism - Anomaly Event
Prism - Anomaly Event

Another topic of interest for anomalies is seasonality. For example, during the holiday period retailers will see higher demand than other times of the year, or during the end of month close.

Anomaly detection accounts for this seasonality and leverages the following periods to compare between micro (daily) and macro (quarterly) trends:

  • Daily
  • Weekly
  • Monthly

You can also set your own custom alerts or static thresholds:

Prism - Anomaly Custom Event
Prism - Anomaly Custom Event
Anomaly Detection Algorithm

Nutanix leverages a method for determining the bands called 'Generalized Extreme Studentized Deviate Test'. A simple way to think about this is similar to a confidence interval where the values are between the lower and upper limits established by the algorithm.

The algorithm requires 3 x the granularity (e.g. daily, weekly, monthly, etc.) to calculate the seasonality and expected bands. For example, the following amounts of data would be required to adapt to each seasonality:

  • Daily: 3 days
  • Weekly: 3 weeks (21 days)
  • Monthly: 3 months (90 days)

Twitter has a good resource on how they leverage this which goes into more detail on the logic: LINK

Nutanix Software Upgrade

Performing a Nutanix software upgrade is a very simple and non-disruptive process.

To begin, start by logging into Prism and clicking on the gear icon on the top right (settings) or by pressing 'S' and selecting 'Upgrade Software':

Prism - Settings - Upgrade Software
Prism - Settings - Upgrade Software

This will launch the 'Upgrade Software' dialog box and will show your current software version and if there are any upgrade versions available.  It is also possible to manually upload a NOS binary file.

You can then download the upgrade version from the cloud or upload the version manually:

Upgrade Software - Main
Upgrade Software - Main
Upload software from the CVM

In certain cases you may want to download the software and upload from the CVM itself. I use this in my environment when I want to download builds locally to the CVM.

First SSH into a CVM and find the Prism leader:

curl localhost:2019/prism/leader && echo

SSH to the Prism leader and download the software bundle and metadata JSON

Run the following command to "upload" the software to Prism:

ncli software upload file-path=<PATH_TO_SOFTWARE> meta-file-path=<PATH_TO_METADATA_JSON> software-type=<SOFTWARE_TYPE>

The following shows an example for Prism Central:

ncli software upload file-path=/home/nutanix/tmp/leader-prism_central.tar meta-file-path=/home/nutanix/tmp/leader-prism_central-metadata.json software-type=prism_central_deploy

It will then upload the upgrade software onto the Nutanix CVMs:

Upgrade Software - Upload
Upgrade Software - Upload

After the software is loaded click on 'Upgrade' to start the upgrade process:

Upgrade Software - Upgrade Validation
Upgrade Software - Upgrade Validation

You'll then be prompted with a confirmation box:

Upgrade Software - Confirm Upgrade
Upgrade Software - Confirm Upgrade

The upgrade will start with pre-upgrade checks then start upgrading the software in a rolling manner:

Upgrade Software - Execution
Upgrade Software - Execution

Once the upgrade is complete you'll see an updated status and have access to all of the new features:

Upgrade Software - Complete
Upgrade Software - Complete

Your Prism session will briefly disconnect during the upgrade when the current Prism Leader is upgraded.  All VMs and services running remain unaffected.

Hypervisor Upgrade

Similar to Nutanix software upgrades, hypervisor upgrades can be fully automated in a rolling manner via Prism.

To begin follow the similar steps above to launch the 'Upgrade Software' dialogue box and select 'Hypervisor'.

You can then download the hypervisor upgrade version from the cloud or upload the version manually:

Upgrade Hypervisor - Main
Upgrade Hypervisor - Main

It will then load the upgrade software onto the Hypervisors.  After the software is loaded click on 'Upgrade' to start the upgrade process:

Upgrade Hypervisor - Upgrade Validation
Upgrade Hypervisor - Upgrade Validation

You'll then be prompted with a confirmation box:

Upgrade Hypervisor - Confirm Upgrade
Upgrade Hypervisor - Confirm Upgrade

The system will then go through host pre-upgrade checks and upload the hypervisor upgrade to the cluster:

Upgrade Hypervisor - Pre-upgrade Checks
Upgrade Hypervisor - Pre-upgrade Checks

Once the pre-upgrade checks are complete the rolling hypervisor upgrade will then proceed:

Upgrade Hypervisor - Execution
Upgrade Hypervisor - Execution

Similar to the rolling nature of the Nutanix software upgrades, each host will be upgraded in a rolling manner with zero impact to running VMs.  VMs will be live-migrated off the current host, the host will be upgraded, and then rebooted.  This process will iterate through each host until all hosts in the cluster are upgraded.

Pro tip

You can also get cluster wide upgrade status from any Nutanix CVM by running 'host_upgrade --status'.  The detailed per host status is logged to ~/data/logs/host_upgrade.out on each CVM.

Once the upgrade is complete you'll see an updated status and have access to all of the new features:

Upgrade Hypervisor - Complete
Upgrade Hypervisor - Complete

Cluster Expansion (add node)

Cluster Expansion
Cluster Expansion

The ability to dynamically scale the Nutanix cluster is core to its functionality. To scale an Nutanix cluster, rack / stack / cable the nodes and power them on. Once the nodes are powered up they will be discoverable by the current cluster using mDNS.

The figure shows an example 7 node cluster with 1 node which has been discovered:

Add Node - Discovery
Add Node - Discovery

Multiple nodes can be discovered and added to the cluster concurrently.

Once the nodes have been discovered you can begin the expansion by clicking 'Expand Cluster' on the upper right hand corner of the 'Hardware' page:

Hardware Page - Expand Cluster
Hardware Page - Expand Cluster

You can also begin the cluster expansion process from any page by clicking on the gear icon:

Gear Menu - Expand Cluster
Gear Menu - Expand Cluster

This launches the expand cluster menu where you can select the node(s) to add and specify IP addresses for the components:

Expand Cluster - Host Selection
Expand Cluster - Host Selection

After the hosts have been selected you'll be prompted to upload a hypervisor image which will be used to image the nodes being added. For AHV or cases where the image already exists in the Foundation installer store, no upload is necessary.

Expand Cluster - Host Configuration
Expand Cluster - Host Configuration

After the upload is completed you can click on 'Expand Cluster' to begin the imaging and expansion process:

Expand Cluster - Execution
Expand Cluster - Execution

The job will then be submitted and the corresponding task item will appear:

Expand Cluster - Execution
Expand Cluster - Execution

Detailed tasks status can be viewed by expanding the task(s):

Expand Cluster - Execution
Expand Cluster - Execution

After the imaging and add node process has been completed you'll see the updated cluster size and resources:

Expand Cluster - Execution
Expand Cluster - Execution

I/O Metrics

Identification of bottlenecks is a critical piece of the performance troubleshooting process. In order to aid in this process, Nutanix has introduced a new 'I/O Metrics' section to the VM page.

Latency is dependent on multitude of variables (queue depth, I/O size, system conditions, network speed, etc.). This page aims to offer insight on the I/O size, latency, source, and patterns.

To use the new section, go to the 'VM' page and select a desired VM from the table. Here we can see high level usage metrics:

VM Page - Details
VM Page - Details

The 'I/O Metrics' tab can be found in the section below the table:

VM Page - I/O Metrics Tab
VM Page - I/O Metrics Tab

Upon selecting the 'I/O Metrics' tab a detailed view will be shown. We will break this page down and how to use it in this section.

The first view is the 'Avg I/O Latency' section that shows average R/W latency for the past three hours. By default the latest reported values are shown with the corresponding detailed metrics below for that point in time.

You can also mouse over the plot to see the historical latency values and click on a time of the plot to view the detailed metrics below.

I/O Metrics - Latency Plot
I/O Metrics - Latency Plot

This can be useful when a sudden spike is seen. If you see a spike and want to investigate further, click on the spike and evaluate the details below.

I/O Metrics - Latency Plot
I/O Metrics - Latency Plot

If latency is all good, no need to dig any further.

The next section shows a histogram of I/O sizes for read and write I/Os:

I/O Metrics - I/O Size histogram
I/O Metrics - I/O Size histogram

Here we can see our read I/Os range from 4K to 32K in size:

I/O Metrics - Read I/O Size histogram
I/O Metrics - Read I/O Size histogram

Here we can see our write I/Os range from 16K to 64K with some up to 512K in size:

I/O Metrics - Write I/O Size histogram
I/O Metrics - Write I/O Size histogram
Pro tip

If you see a spike in latency the first thing to check is the I/O size. Larger I/Os (64K up to 1MB) will typically see higher latencies than smaller I/Os (4K to 32K).

The next section shows a histogram of I/O latencies for read and write I/Os:

I/O Metrics - Latency histogram
I/O Metrics - Latency histogram

Looking at the read latency histogram we can see the majority of read I/Os are sub-ms (<1ms) with some up to 2-5ms.

I/O Metrics - Read Latency histogram
I/O Metrics - Read Latency histogram

Taking a look below at the 'Read Source' we can see most I/Os are being served from the SSD tier:

I/O Metrics - Read Source SSD
I/O Metrics - Read Source SSD

As data is read it will be pulled in to the Unified Cache realtime (Check the 'I/O Path and Cache' section to learn more). Here we can see the data has been pulled into the cache and is now being served from DRAM:

I/O Metrics - Read Source DRAM
I/O Metrics - Read Source DRAM

We can now see basically all of our read I/Os are seeing sub-ms (<1ms) latency:

I/O Metrics - Read Latency histogram
I/O Metrics - Read Latency histogram

Here we can see the majority of our write I/O are seeing <1-2ms latency:

I/O Metrics - Write Latency histogram
I/O Metrics - Write Latency histogram
Pro tip

If you see a spike in read latency and the I/O sizes aren't large, check where the read I/Os are being served from. Any initial read from HDD will see higher latency than the DRAM cache; however, once it is in the cache all subsequent reads will hit DRAM and see an improvement in latency.

The last section shows the I/O patterns and how much is random vs. sequential:

I/O Metrics - RW Random vs. Sequential
I/O Metrics - RW Random vs. Sequential

Typically I/O patterns will vary by application or workload (e.g. VDI is mainly random, whereas Hadoop would primarily be sequential). Other workloads will be a mix of both. For example, a database might be random on inserts or some queries, however sequential during ETL.

Capacity Planning

To get detailed capacity planning details you can click on a specific cluster under the 'cluster runway' section in Prism Central to get more details:

Prism Central - Capacity Planning
Prism Central - Capacity Planning

This view provides detailed information on cluster runway and identifies the most constrained resource (limiting resource).  You can also get detailed information on what the top consumers are as well as some potential options to clean up additional capacity or ideal node types for cluster expansion.

Prism Central - Capacity Planning - Recommendations
Prism Central - Capacity Planning - Recommendations


When we think about our daily activities the more we can automate the better. We are constantly doing this in our daily lives with our routines and technology enables us to do the same in other areas. Prism Pro X-Play allows us to automate a common set of activities via Prism. However, before diving into the product, let's first cover what we're trying to do.

Event driven automation works in the following manner:

event(s) → logic → action(s)

In this scenario there's some sort of event (or cascading events) that occur which triggers a series or set of actions. A great example of this is IFTTT which takes an event, applies some logic (hench the 'if this then that' acronym), then performs some action.

For example, take turning off the lights at home when we leave. If we can program the event (e.g. leaving home / device not present) to trigger the system to turn off all the lights automatically, that makes our lives much simpler. I personally use this all over my home and it makes life much easier and allows me to focus on other higher impact activities.

If we compare this to our IT operations activities we see a similar pattern. An event occurs (e.g. a VM needs more disk space) and then we perform a series of actions (e.g. create a ticket, add storage, close ticket etc.). These repetetive activies are a perfect example of where automation can add value and enable us to focus on more beneficial activities.

With X-Play we can take a series of events / alerts and allow the system to intercept those and perform a series of actions.

To get started navigate to the 'Plays' section under 'Operations' in Prism Central:

X-Play - Navigation
X-Play - Navigation

This will launch the main X-Play page:

X-Play - Playbooks Overview
X-Play - Playbooks Overview

Click on 'Get Started' to view the current plays and/or create a new one:

X-Play - Playbooks
X-Play - Playbooks

From here you can create a new playbook by first defining the trigger:

X-Play - Trigger
X-Play - Trigger

The following shows an example trigger which is based upon a custom alert:

X-Play - Trigger - Custom Alert
X-Play - Trigger - Custom Alert

Once the trigger has been defined, you now specify a series of actions. The following shows some sample actions:

X-Play - Actions
X-Play - Actions

You then input the details for the action, this shows a sample REST API call:

X-Play - Sample REST Action
X-Play - Sample REST Action
REST API Actions and External Systems

X-Play provides a multitude of default actions like sending email, sending a slack message, as well as others like performing a REST API call.

This is critical when we think about interfacing with external systems like a CMDB or other ticketing / automation tools. By using a REST API action we can interface with those to create / resolve tickets, kick off other workflows, etc. This is an extremely powerful option as it enables all systems to be in sync.

For entity / event specific details you can use the 'parameters' variables which will give you details about the event, entity and others:

X-Play - Action Parameters
X-Play - Action Parameters

Once complete you can save you play and it will start to execute as defined.

The following shows a sample play whith multiple actions performed:

X-Play - Sample Playbook
X-Play - Sample Playbook

The plays tab will show execution time of the play and status:

X-Play - Plays Executed
X-Play - Plays Executed

Remember, automate all the things!

APIs and Interfaces

The HTML5 UI is a key part to Prism to provide a simple, easy to use management interface.  However, another core ability are the APIs which are available for automation.  All functionality exposed through the Prism UI is also exposed through a full set of REST APIs to allow for the ability to programmatically interface with the Nutanix platform.  This allows customers and partners to enable automation, 3rd-party tools, or even create their own UI.  

Core to any dynamic or “software-defined” environment, Nutanix provides a vast array of interfaces allowing for simple programmability and interfacing. Here are the main interfaces:

  • Scripting interfaces

To learn more about the APIs and review sample code, be sure to check out!

Core to this is the REST API which exposes every capability and data point of the Prism UI and allows for orchestration or automation tools to easily drive Nutanix action.  This enables tools like Saltstack, Puppet, vRealize Operations, System Center Orchestrator, Ansible, etc. to easily create custom workflows for Nutanix. Also, this means that any third-party developer could create their own custom UI and pull in Nutanix data via REST.

The following figure shows a small snippet of the Nutanix REST API explorer which allows developers to interact with the API and see expected data formats:

Prism REST API Explorer
Prism REST API Explorer

Operations can be expanded to display details and examples of the REST call:

Prism REST API Sample Call
Prism REST API Sample Call
API Authentication Scheme(s)

As of 4.5.x basic authentication over HTTPS is leveraged for client and HTTP call authentication.


The AOS CLI (ACLI) is the CLI for managing the AOS portion of the Nutanix product.  These capabilities were enabled in releases after 4.1.2.

NOTE: All of these actions can be performed via the HTML5 GUI and REST API.  I just use these commands as part of my scripting to automate tasks.

Enter ACLI shell

Description: Enter ACLI shell (run from any CVM)



Description: Execute ACLI command via Linux shell

ACLI <Command>

Output ACLI response in json format

Description: Lists AOS nodes in the cluster.

Acli –o json

List hosts

Description: Lists AOS nodes in the cluster.


Create network

Description: Create network based on VLAN

net.create <TYPE>.<ID>[.<VSWITCH>] ip_config=<A.B.C.D>/<NN>

Example: net.create vlan.133 ip_config=

List network(s)

Description: List networks


Create DHCP scope

Description: Create dhcp scope

net.add_dhcp_pool <NET NAME> start=<START IP A.B.C.D> end=<END IP W.X.Y.Z>

Note: .254 is reserved and used by the AOS DHCP server if an address for the AOS DHCP server wasn’t set during network creation

Example: net.add_dhcp_pool vlan.100 start= end=

Get an existing network's details

Description: Get a network's VMs and details including VM name / UUID, MAC address and IP

net.list_vms <NET NAME>

Example: net.list_vms vlan.133

Configure DHCP DNS servers for network

Description: Set DHCP DNS

net.update_dhcp_dns <NET NAME> servers=<COMMA SEPARATED DNS IPs> domains=<COMMA SEPARATED DOMAINS>

Example: net.set_dhcp_dns vlan.100 servers=,

Create Virtual Machine

Description: Create VM

vm.create <COMMA SEPARATED VM NAMES> memory=<NUM MEM MB> num_vcpus=<NUM VCPU> num_cores_per_vcpu=<NUM CORES> ha_priority=<PRIORITY INT>

Example: vm.create testVM memory=2G num_vcpus=2

Bulk Create Virtual Machine

Description: Create bulk VM

vm.create  <CLONE PREFIX>[<STARTING INT>..<END INT>] memory=<NUM MEM MB> num_vcpus=<NUM VCPU> num_cores_per_vcpu=<NUM CORES> ha_priority=<PRIORITY INT>

Example: vm.create testVM[000..999] memory=2G num_vcpus=2

Clone VM from existing

Description: Create clone of existing VM

vm.clone <CLONE NAME(S)> clone_from_vm=<SOURCE VM NAME>

Example: vm.clone testClone clone_from_vm=MYBASEVM

Bulk Clone VM from existing

Description: Create bulk clones of existing VM

vm.clone <CLONE PREFIX>[<STARTING INT>..<END INT>] clone_from_vm=<SOURCE VM NAME>

Example: vm.clone testClone[001..999] clone_from_vm=MYBASEVM

Create disk and add to VM

# Description: Create disk for OS

vm.disk_create <VM NAME> create_size=<Size and qualifier, e.g. 500G> container=<CONTAINER NAME>

Example: vm.disk_create testVM create_size=500G container=default

Add NIC to VM

Description: Create and add NIC

vm.nic_create <VM NAME> network=<NETWORK NAME> model=<MODEL>

Example: vm.nic_create testVM network=vlan.100

Set VM’s boot device to disk

Description: Set a VM boot device

Set to boot from specific disk id

vm.update_boot_device <VM NAME> disk_addr=<DISK BUS>

Example: vm.update_boot_device testVM disk_addr=scsi.0

Set VM’s boot device to CD-ROM

Set to boot from CD-ROM

vm.update_boot_device <VM NAME> disk_addr=<CD-ROM BUS>

Example: vm.update_boot_device testVM disk_addr=ide.0

Mount ISO to CD-ROM

Description: Mount ISO to VM CD-ROM


1. Upload ISOs to container

2. Enable whitelist for client IPs

3. Upload ISOs to share

Create CD-ROM with ISO

vm.disk_create <VM NAME> clone_nfs_file=<PATH TO ISO> CD-ROM=true

Example: vm.disk_create testVM clone_nfs_file=/default/ISOs/myfile.iso CD-ROM=true

If a CD-ROM is already created just mount it

vm.disk_update <VM NAME> <CD-ROM BUS> clone_nfs_file<PATH TO ISO>

Example: vm.disk_update atestVM1 ide.0 clone_nfs_file=/default/ISOs/myfile.iso

Detach ISO from CD-ROM

Description: Remove ISO from CD-ROM

vm.disk_update <VM NAME> <CD-ROM BUS> empty=true

Power on VM(s)

Description: Power on VM(s)

vm.on <VM NAME(S)>

Example: vm.on testVM

Power on all VMs

Example: vm.on *

Power on all VMs matching a prefix

Example: vm.on testVM*

Power on range of VMs

Example: vm.on testVM[0-9][0-9]


NOTE: All of these actions can be performed via the HTML5 GUI and REST API.  I just use these commands as part of my scripting to automate tasks.

Add subnet to NFS whitelist

Description: Adds a particular subnet to the NFS whitelist

ncli cluster add-to-nfs-whitelist ip-subnet-masks=

Display Nutanix Version

Description: Displays the current version of the Nutanix software

ncli cluster version

Display hidden NCLI options

Description: Displays the hidden ncli commands/options

ncli helpsys listall hidden=true [detailed=false|true]

List Storage Pools

Description: Displays the existing storage pools

ncli sp ls

List containers

Description: Displays the existing containers

ncli ctr ls

Create container

Description: Creates a new container

ncli ctr create name=<NAME> sp-name=<SP NAME>

List VMs

Description: Displays the existing VMs

ncli vm ls

List public keys

Description: Displays the existing public keys

ncli cluster list-public-keys

Add public key

Description: Adds a public key for cluster access

SCP public key to CVM

Add public key to cluster

ncli cluster add-public-key name=myPK file-path=~/

Remove public key

Description: Removes a public key for cluster access

ncli cluster remove-public-keys name=myPK

Create protection domain

Description: Creates a protection domain

ncli pd create name=<NAME>

Create remote site

Description: Create a remote site for replication

ncli remote-site create name=<NAME> address-list=<Remote Cluster IP>

Create protection domain for all VMs in container

Description: Protect all VMs in the specified container

ncli pd protect name=<PD NAME> ctr-id=<Container ID> cg-name=<NAME>

Create protection domain with specified VMs

Description: Protect the VMs specified

ncli pd protect name=<PD NAME> vm-names=<VM Name(s)> cg-name=<NAME>

Create protection domain for DSF files (aka vDisk)

Description: Protect the DSF Files specified

ncli pd protect name=<PD NAME> files=<File Name(s)> cg-name=<NAME>

Create snapshot of protection domain

Description: Create a one-time snapshot of the protection domain

ncli pd add-one-time-snapshot name=<PD NAME> retention-time=<seconds>

Create snapshot and replication schedule to remote site

Description: Create a recurring snapshot schedule and replication to n remote sites

ncli pd set-schedule name=<PD NAME> interval=<seconds> retention-policy=<POLICY> remote-sites=<REMOTE SITE NAME>

List replication status

Description: Monitor replication status

ncli pd list-replication-status

Migrate protection domain to remote site

Description: Fail-over a protection domain to a remote site

ncli pd migrate name=<PD NAME> remote-site=<REMOTE SITE NAME>

Activate protection domain

Description: Activate a protection domain at a remote site

ncli pd activate name=<PD NAME>

Enable DSF shadow clones

Description: Enables the DSF Shadow Clone feature

ncli cluster edit-params enable-shadow-clones=true

Enable dedup for vDisk

Description: Enables fingerprinting and/or on disk dedup for a specific vDisk

ncli vdisk edit name=<VDISK NAME> fingerprint-on-write=<true/false> on-disk-dedup=<true/false>

Check cluster resiliency status

# Node status
ncli cluster get-domain-fault-tolerance-status type=node

# Block status
ncli cluster get-domain-fault-tolerance-status type=rackable_unit

PowerShell CMDlets

The below will cover the Nutanix PowerShell CMDlets, how to use them and some general background on Windows PowerShell.


Windows PowerShell is a powerful shell (hence the name ;P) and scripting language built on the .NET framework.  It is a very simple to use language and is built to be intuitive and interactive.  Within PowerShell there are a few key constructs/Items:


CMDlets are commands or .NET classes which perform a particular operation.  They are usually conformed to the Getter/Setter methodology and typically use a <Verb>-<Noun> based structure.  For example: Get-Process, Set-Partition, etc.

Piping or Pipelining

Piping is an important construct in PowerShell (similar to its use in Linux) and can greatly simplify things when used correctly.  With piping you’re essentially taking the output of one section of the pipeline and using that as input to the next section of the pipeline.  The pipeline can be as long as required (assuming there remains output which is being fed to the next section of the pipe). A very simple example could be getting the current processes, finding those that match a particular trait or filter and then sorting them:

Get-Service | where {$_.Status -eq "Running"} | Sort-Object Name

Piping can also be used in place of for-each, for example:

# For each item in my array
$myArray | %{
  # Do something

Key Object Types

Below are a few of the key object types in PowerShell.  You can easily get the object type by using the .getType() method, for example: $someVariable.getType() will return the objects type.


$myVariable = "foo"

Note: You can also set a variable to the output of a series or pipeline of commands:

$myVar2 = (Get-Process | where {$_.Status -eq "Running})

In this example the commands inside the parentheses will be evaluated first then variable will be the outcome of that.


$myArray = @("Value","Value")

Note: You can also have an array of arrays, hash tables or custom objects

Hash Table

$myHash = @{"Key" = "Value";"Key" = "Value"}

Useful commands

Get the help content for a particular CMDlet (similar to a man page in Linux)

Get-Help <CMDlet Name>

Example: Get-Help Get-Process

List properties and methods of a command or object

<Some expression or object> | Get-Member

Example: $someObject | Get-Member

Core Nutanix CMDlets and Usage

The Nutanix CMDlets can be downloaded directly from the Prism UI (post 4.0.1) and can be found on the drop down in the upper right hand corner:

Prism CMDlets Installer Link
Prism CMDlets Installer Link
Load Nutanix Snapin

Check if snapin is loaded and if not, load

if ( (Get-PSSnapin -Name NutanixCmdletsPSSnapin -ErrorAction SilentlyContinue) -eq $null )
    Add-PsSnapin NutanixCmdletsPSSnapin

List Nutanix CMDlets

Get-Command | Where-Object{$_.PSSnapin.Name -eq "NutanixCmdletsPSSnapin"}

Connect to a Nutanix cluster

Connect-NutanixCluster -Server $server -UserName "myuser" -Password (Read-Host "Password: " -AsSecureString) -AcceptInvalidSSLCerts

Get Nutanix VMs matching a certain search string

Set to variable

$searchString = "myVM"
$vms = Get-NTNXVM | where {$_.vmName -match $searchString}


Get-NTNXVM | where {$_.vmName -match "myString"}

Interactive and formatted

Get-NTNXVM | where {$_.vmName -match "myString"} | ft

Get Nutanix vDisks

Set to variable

$vdisks = Get-NTNXVDisk



Interactive and formatted

Get-NTNXVDisk | ft

Get Nutanix Containers

Set to variable

$containers = Get-NTNXContainer



Interactive and formatted

Get-NTNXContainer | ft

Get Nutanix Protection Domains

Set to variable

$pds = Get-NTNXProtectionDomain



Interactive and formatted

Get-NTNXProtectionDomain | ft

Get Nutanix Consistency Groups

Set to variable

$cgs = Get-NTNXProtectionDomainConsistencyGroup



Interactive and formatted

Get-NTNXProtectionDomainConsistencyGroup | ft

Resources and Scripts:

NOTE: some scripts above are not maintained and should be used for reference only.

You can find more scripts on the Nutanix Github located at

Book of AOS

a·crop·o·lis - /ɘ ' kräpɘlis/ - noun - data plane
storage, compute and virtualization platform.


The Acropolis Operating System (AOS) provides the core functionality leveraged by workloads and services running on the platform. This includes, but isn't limited to, things like storage services, upgrades, etc.

The figure highlights an image illustrating the conceptual nature of AOS at various layers:

High-level AOS Architecture
High-level AOS Architecture

Building upon the distributed nature of everything Nutanix does, we’re expanding this into the virtualization and resource management space.  AOS is a back-end service that allows for workload and resource management, provisioning, and operations.  Its goal is to abstract the facilitating resource (e.g., hypervisor, on-premise, cloud, etc.) from the workloads running, while providing a single “platform” to operate. 

This gives workloads the ability to seamlessly move between hypervisors, cloud providers, and platforms.

Supported Hypervisors for VM Management

As of 4.7, AHV and ESXi are the supported hypervisors for VM management, however this may expand in the future.  The Volumes API and read-only operations are still supported on all.

Acropolis Services

An Acropolis Worker runs on every CVM with an elected Acropolis Leader which is responsible for task scheduling, execution, IPAM, etc.  Similar to other components which have a Leader, if the Acropolis Leader fails, a new one will be elected.

The role breakdown for each can be seen below:

  • Acropolis Leader
    • Task scheduling & execution
    • Stat collection / publishing
    • Network Controller (for hypervisor)
    • VNC proxy (for hypervisor)
    • HA (for hypervisor)
  •  Acropolis Worker
    • Stat collection / publishing
    • VNC proxy (for hypervisor)

Here we show a conceptual view of the Acropolis Leader / Worker relationship:

Acropolis Services
Acropolis Services

Dynamic Scheduler

Efficient scheduling of resources is critical to ensure resources are effectively consumed. The AOS Dynamic Scheduler extends the traditional means of scheduling that relies upon compute utilization (CPU/MEM) to make placement decisions. It leverages compute, as well as storage and others to drive VM and volume (ABS) placement decisions. This ensures that resources are effectively consumed and end-user performance is optimal.

Resource scheduling can be broken down into two key areas:

  • Initial placement
    • Where an item is scheduled at power-on
  • Runtime Optimization
    • Movement of workloads based upon runtime metrics

The original AOS Scheduler had taken care of the initial placement decisions since its release. With its release in AOS 5.0, the AOS Dynamic Scheduler expands upon this to provide runtime resources optimization.

The figure shows a high-level view of the scheduler architecture:

AOS Dynamic Scheduler
AOS Dynamic Scheduler

The dynamic scheduler runs consistently throughout the day to optimize placement (currently every 15 minutes | Gflag: lazan_anomaly_detection_period_secs). Estimated demand is calculated using historical utilization values and fed into a smoothing algorithm. This estimated demand is what is used to determine movement, which ensures a sudden spike will not skew decisions.

A different approach towards resource optimization

When you look at existing scheduling / optimization platforms (VMware DRS, Microsoft PRO) they are all focused on balancing workloads / VMs evenly across cluster resources. NOTE: how aggressively it tries to eliminate skew is determined by the balancing configuration (e.g. manual -> none, conservative -> some, aggressive -> more).

For example, say we had 3 hosts in a cluster, each of which is utilized 50%, 5%, 5% respectively. Typical solutions would try to re-balance workloads to get each hosts utilization ~20%. But why?

What we're really trying to do is eliminate / negate any contention for resources, not eliminate skew. Unless there is contention for resources there is no positive gain from "balancing" workloads. In fact by forcing unnecessary movement we cause additional requisite work (e.g. memory transfer, cache re-localization, etc.), all of which consumes resources.

The AOS Dynamic Scheduler does just this, it will only invoke workload movement if there is expected contention for resources, not because of skew. NOTE: DSF works in a different way and works to ensure uniform distribution of data throughout the cluster to eliminate hot spots and speed up rebuilds. To learn more of DSF, check out the 'disk balancing' section.

At power-on ADS will balance VM initial placement throughout the cluster.

Placement Decisions

Placement decisions are based upon the following items:

  • Compute utilization
    • We monitor each individual node's compute utilization. In the event where a node's expected CPU allocation breaches its threshold (currently 85% of host CPU | Gflag: lazan_host_cpu_usage_threshold_fraction) we will migrate VMs off those host(s) to re-balance the workload. A key thing to mention here is a migration will only be performed when there is contention. If there is skew in utilization between nodes (e.g. 3 nodes at 10% and 1 at 50%) we will not perform a migration as there is no benefit from doing so until there is contention for resources.
  • Storage performance
    • Being a hyperconverged platform we manage both compute and storage resources. The scheduler will monitor each node's Stargate process utilization. In the event where certain Stargate(s) breach their allocation threshold (currently 85% of CPU allocated to Stargate | Gflag: lazan_stargate_cpu_usage_threshold_pct) , we will migrate resources across hosts to eliminate any hot spots. Both VMs and ABS Volumes can be migrated to eliminate any hot Stargates.
  • [Anti-]Affinity rules
    • Affinity or Anti-affinity constraints determine where certain resources are scheduled based upon other resources in the environment. In certain cases you want VMs to run on the same node for licensing reasons. In this case the VMs would be affined to the same host. In other cases you might want to ensure VMs run on different nodes for availability purposes. In this case the VMs would be anti-affined.

The scheduler will make its best effort to optimize workload placement based upon the prior items. The system places a penalty on movement to ensure not too many migrations are taking place. This is a key item as we want to make sure the movement doesn't have any negative impacts on the workload.

After a migration the system will judge its "effectiveness" and see what the actual benefit is. This learning model can self-optimize to ensure there is a valid basis for any migration decision.

Security and Encryption

Security is a core part of the Nutanix platform and was kept in mind from day one. The Nutanix Security Development Lifecycle (SecDL) incorporates security into every step of the development process. The Nutanix controlled parts of the platform is secure out of the box, rather than being an afterthought requiring end-users to "harden" the platform.

When we think about security we're really trying to achieve 3 core things (aptly named the CIA triad):

  1. Confidentially
    • Protect and secure data by preventing unauthorized access
  2. Integrity
    • Ensure the consistency and accuracy of data by preventing unauthorized alteration
  3. Availability
    • Ensure authorized users get access to data through resiliency and redundancy

This can be simplified down to a simple statement: enable users to do their jobs while keeping the bad people out. When we're designing for security we need to look at a few core areas of interest which is highlighted in the following diagram:

Security Layers
Security Layers

We will break down each section in the prior graphic in the following sections.

Systems & Configuration
At a glance
  • Patch and remove known vulnerabilities
  • Enforce strong passwords and remove default accounts
  • Configure permissions and user privileges
  • Close unused ports / protocols
  • Use automation to ensure baselines

Traditionally people refer to system (OS + App) security using a method called "hardening". This is the process to which you would secure the system by configuring things to a certain standard called a baseline.

The DoD's IT org (DISA) has a sample hardening guide which they call the STIG (more details in the SCMA section following). This includes things like directory permissions, user account management, password complexity, firewalls and a slew of other configuration settings.

Once a system is configured to that standard it is considered "secure" however that is just the beginning of the process. System security is something that must be maintained throughout its lifespan. For example, to ensure that standard hardening baseline is met, configuration automation tools should be employed. This ensures the system is always meeting your baseline "desired state".

Nutanix ensures this for its CVM and AHV hypervisor using a tool we've developed called SCMA which is covered later in this section.

At a glance:
  • Secure access control to data
  • Always take backups
  • Encrypt data and secure keys

Data is at the core of any business and is arguably the company's most valuable asset. When thinking of security we need to focus on ensuring data accessibility, quality, and theft avoidance.

On the concept of accessibility, we constantly need access to systems and data to make decisions. One recent method of attack called 'Ransomware' threatens the ability to access data by encrypting the data and then ransoming the user to get access back. This can be avoided in a variety of methods, but also highlights to importance of backups.

Data quality is also a critical item since a lot of decisions or actions are depending on this. For example, an attacker could get access to a system and place malicious orders or update shipping addresses diverting goods to his location. This is where logging and checksumming can be very critical to ensure data remains clean.

Last but not least is how do we secure or harden the data. This is typically done using encryption which renders the data useless if they don't have the keys to decrypt the data. In this case if someone were to steal an encrypted file or disk device, they would be unable to get access to the underlying data.

At a glance:
  • Segment trusted/untrusted networks
  • Firewall at the perimeter and between segments
  • Leverage an IDPS to detect anomalies

The network is the typically communication vector attackers use to gain access to systems. This includes things like perimeter security (e.g. external firewalls) and internal intrusion prevention / detection.

Like any good design there should always be layers of security; the same holds true with the network. We need to segment our high-security networks from our trusted networks and secure those from our untrusted networks (e.g. business / wifi networks). It is never safe to assume your local network in the office is secure.

By having multiple layers of the network we can ensure someone who gains access our most untrusted network has a more difficult time working towards our secure networks. During this process a good IDPS system can detect access anomalies or scanning tools like nmap.

Authentication and Authorization
At a glance:
  • Use MFA/2FA where possible
  • Use granular permissions

Authentication is all about authenticating a users identity against a trusted source of truth like Active Directory or any other IDP (Identity provider). Tools like MFA (multi-factor authentication) or 2FA add additional assurance the user is who they're trying to authenticate as.

Once the identity has been verified the next piece is to determine what they are authorized to do or what they can access; this is the authorization piece. User foo is authorized to perform x,y on bar and y,z on bas.

Compliance & Monitoring
At a glance:
  • Compliance is a continuous activity
  • Monitor and look for anomalies

Compliance is typically something people refer to when looking at certain accreditations like PCI, HIPAA, etc. However this extends further into ensure compliance with any hardening guide or standards that have been set. For example, the STIG is a sample hardening baseline, however each company may have additional policies / rules in place. In order to ensure a secure system, we must make sure our systems meet these policies and are in a compliant state.

Traditionally compliance is checked retroactively and is a fairly manual process. I believe this is absolutely the wrong approach. Compliance is something we must constantly ensure as that's the only way we can make sure we limit any potential threat vectors, or close any that may have been opened.

Tools that handle configuration management automation (aka desired state configuration - DSC) are a critical piece here. These will ensure our configuration / settings is always set to our baseline or desired state.

Monitoring and penetration testing are critical to validate and ensure this compliance. Tools like Nessus, Nmap or metasploit can be used to to test the security of a system. During these tests monitoring and detection systems should detect these and alert.

At a glance:
  • Educate, educate, educate
  • Enforce strong practices and habits (e.g. locking computer)

In any system, the people are traditionally the weakest link. In order to ensure users aren't prone to phishing attacks or social manipulation, training and education is critical. We must ensure that users know what to look for, and to escalate to a known resource if they are unsure.

One method of education is actually simulating phishing attacks so they can start to question things and learn what to look for. We must also enforce other policies like not leaving their computer unlocked or writing down their passwords.

Certifications & Accreditations

Nutanix has the following security certifications / qualifications across portions of the stack (on and off premise):

  • Common Criteria*
    • Common Criteria was produced predominantly so that companies selling computer products for the government market (mainly for Defense or Intelligence use) would only need to have them evaluated against one set of standards. The CC was developed by the governments of Canada, France, Germany, the Netherlands, the UK, and the U.S.
    • *This is currently under re-certification as of March 2020
  • Security Technical Implementation Guides (STIGs)
    • Configuration standards for DOD IA and IA-enabled devices/systems. Since 1998, DISA Field Security Operations (FSO) has played a critical role enhancing the security posture of DoD's (Dept of Defense) security systems by providing the Security Technical Implementation Guides. The STIGs contain technical guidance to "lock down" information systems/software that might otherwise be vulnerable to a malicious computer attack.
  • FIPS 140-2
    • FIPS 140-2 standard is an information technology security accreditation program for cryptographic modules produced by private sector vendors who seek to have their products certified for use in government departments and regulated industries (such as financial and health-care institutions) that collect, store, transfer, share and disseminate sensitive but unclassified (SBU) information.
  • NIST 800-53
  • NIST 800-131a
  • ISO 27001
  • ISO 27017
  • ISO 27018
Security Configuration Management Automation (SCMA)

Nutanix Security engineering now provides customers with the ability to evolve from point-in-time security baseline checking to a continuous monitoring/self-remediating baseline to ensure all CVM/AHV hosts in a cluster remain baseline compliant throughout the deployment lifecycle. This new innovation checks all components of the documented security baselines (STIGs) , and if found to be non-compliant, sets it back to the supported security settings without customer intervention. SCMA is enabled by default so no action is necessary to enable.

Ad-hoc SCMA execution

The SCMA will run on the configured schedule (Default: HOURLY), however it is also possible to run this on-demand. To run the SCMA tool you can execute the following command from the CVM(s):

# Run on a single CVM
sudo salt-call state.highstate

# Run on all CVMs
allssh "sudo salt-call state.highstate"

The Nutanix Command Line Interface (NCLI) allows customers to control various configuration settings to enable more stringent security requirements.

CVM Security Settings

The following commands have been added to NCLI to support cluster-wide configuration of the SCMA policy. The list below gives all commands and functions:

Get CVM security settings

ncli cluster get-cvm-security-config

This command outputs the current cluster configuration. The default output will display:

Enable Aide : false
Enable Core : false
Enable High Strength P... : false
Enable Banner : false
Enable SNMPv3 Only : false
Schedule : DAILY

I've defined what each of these means below:

  • Aide
    • Enables the 'Advanced Intrusion Detection Environment' to periodically run.
  • Core
    • Generates stack traces when there's an issue or SCMA is unable to remediate.
  • High Strength Passwords
    • Enforces high strength passwords (minlen=15,difok=8,remember=24)
    • NOTE: I personally disable interactive login and enforce key based access using lockdown mode.
  • Banner
    • Enables a custom login banner
  • SNMPv3 Only
    • Forces SNMPv3 instead of v2

Set CVM login banner

This command enables or disables the Department of Defense (DoD) knowledge of consent login banner when logging in to any Nutanix CVM.

ncli cluster edit-cvm-security-params enable-banner=[yes|no] #Default:no

Custom login banner

By default the DoD knowledge of consent login banner is used. To utilize a custom banner follow the following steps (run as the Nutanix user on any CVM):

  1. Create backup of existing banner
    • sudo cp -a /srv/salt/security/KVM/sshd/DODbanner /srv/salt/security/KVM/sshd/DODbannerbak
  2. Use vi to modify existing banner
    • sudo vi /srv/salt/security/KVM/sshd/DODbanner
  3. Repeat steps on every CVM or SCP modified banner to all other CVMs
  4. Enable banner using command above

Set CVM password strength

This command enables or disables high-strength password policies (minlen=15,difok=8,remember=24).

ncli cluster edit-cvm-security-params enable-high-strength-password=[yes|no] #Default:no

Set Advanced Intrusion Detection Environment (AIDE)

This command enables or disables the AIDE service to run weekly.

ncli cluster edit-cvm-security-params enable-aide=true=[yes|no] #Default:no

Set SNMPv3 only

This command enables or disables SNMPv3 only traps.

ncli cluster edit-cvm-security-params enable-snmpv3-only=[true|false] #Default:false

Set SCMA schedule

This command sets the frequency at which SCMA runs.

ncli cluster edit-cvm-security-params schedule=[HOURLY|DAILY|WEEKLY|MONTHLY] #Default:HOURLY

Hypervisor Security Settings

The following commands have been added to NCLI to support cluster-wide configuration of the SCMA policy. The list below gives all commands and functions:

Get hypervisor security settings

ncli cluster get-hypervisor-security-config

This command outputs the current cluster configuration. The default output will display:

Enable Aide : false
Enable Core : false
Enable High Strength P... : false
Enable Banner : false
Schedule : DAILY

Set hypervisor login banner

This command enables or disables the Department of Defense (DoD) knowledge of consent login banner when loging in to any Nutanix hypervisor.

ncli cluster edit-hypervisor-security-params enable-banner=[yes|no] #Default:no

Set hypervisor password strength

This command enables or disables high-strength password policies (minlen=15,difok=8,remember=24).

ncli cluster edit-hypervisor-security-params enable-high-strength-password=[yes|no] #Default:no

Set Advanced Intrusion Detection Environment (AIDE)

This command enables or disables the AIDE service to run weekly.

ncli cluster edit-hypervisor-security-params enable-aide=true=[yes|no] #Default:no

Set SCMA schedule

This command sets the frequency at which SCMA runs.

ncli cluster edit-hypervisor-security-params schedule=[HOURLY|DAILY|WEEKLY|MONTHLY] #Default:HOURLY

Cluster Lockdown

Cluster lockdown is the ability to disable password based CVM access and/or only allow key based access.

The cluster lockdown configuration can be found in Prism under the gear menu:

Cluster Lockdown Menu
Cluster Lockdown Menu

This will show the current configuration and allow you to add/remove SSH keys for access:

Cluster Lockdown Page
Cluster Lockdown Page

To add a new key click on the 'New Public Key' button and enter the public key details:

Cluster Lockdown - Add Key
Cluster Lockdown - Add Key
Working with SSH keys

To generate a SSH key, run the following command:

ssh-keygen -t rsa -b 2048

This will generate the key pair which creates two files:

  • id_rsa (private key)
  • (public key - this one is used when adding a key to the cluster)

Once you've added some key(s) and have validated access with them, you can disable password based login, by un-checking 'Enable Remote Login with Password.' A popup will appear to confirm the action, click 'Ok' to proceed with lockdown.

Data Encryption and Key Management

Data encryption is a method that allows parties to encode data in a manner that only those who are authorized can make sense of the data, making it unintelligible for anyone who is unauthorized.

For example, if I have a message I want to send to someone and ensure only they can read it, I can encrypt the message (plaintext) with a cipher (key) and send them the encrypted message (ciphertext). If this message is stolen or intercepted the attacker can only see the ciphertext which is mostly useless without having the cipher to decipher the message. Once the desired party has received the message they can decrypt the message using the key we have given them.

There are a few main methods of encrypting data:

  • Symmetric Encryption (private key encryption):
    • The same key is used to both encrypt and decrypt data
    • Examples: AES, PGP*, Blowfish, Twofish, etc.
  • Asymmetric Encryption (public key encryption):
    • One key is used for encryption (public key), another is used for decryption (private key)
    • Examples: RSA, PGP*, etc.

NOTE: PGP (or GPG) uses both a symmetric and asymmetric key.

When data encryption is talked about it is normally done so in two main contexts:

  • In-transit: data that is in transit between two parties (e.g. sending data over the network)
  • At-rest: static data (e.g. data that is stored on a device)

With Native software-based encryption (with or without SEDs) Nutanix solves for both in-transit* and at-rest encryption. With SED only based encryption Nutanix solves for at-rest data encryption. *NOTE: In-transit encryption is currently applicable within a Nutanix cluster for data RF.

The following sections will describe how Nutanix manages data encryption and its key management options.

Data Encryption

Nutanix provides data encryption via three main options:

  • Native software-based encryption (FIPS-140-2 Level-1) *released in 5.5
  • Using self-encrypting drives (SED) (FIPS-140-2 Level-2)
  • Software + hardware encryption

This encryption is configured at either the cluster or container level, and is dependent on the hypervisor type:

  • Cluster level encryption:
    • AHV, ESXi, Hyper-V
  • Container level encryption:
    • ESXi, Hyper-V

NOTE: for deployments using SED based encryption, this will be cluster level as the physical devices are encrypted themselves.

You can view the encryption state of the cluster by navigating to 'Data-at-Rest Encryption' in the settings menu (gear icon). This will provide the current status and allow you to configure encryption (if not currently enabled).

In this example we can see that encryption is enabled at the cluster level:

Data Encryption - Enabled (cluster level)
Data Encryption - Enabled (cluster level)

In this example encryption is enabled for particular containers which are listed:

Data Encryption - Enabled (container level)
Data Encryption - Enabled (container level)

You can enable / modify the configuration by clicking the 'edit configuration' button. This will bring up the menu to configure the KMS used for encryption or the type of KMS currently being leveraged:

Data Encryption - Configure
Data Encryption - Configure

For external KMS the menus will guide your through the CSR request process which you can then give to your CA for signing.

Native Software-based Encryption

Nutanix software encryption provides native AES-256 data-at-rest encryption. This can either interact with any KMIP or TCG compliant external KMS server (Vormetric, SafeNet, etc.) or the Nutanix native KMS introduced in 5.8 (more on this below). For encryption / decryption the system leverages the Intel AES-NI acceleration to minimize any potential performance impact of doing this in software.

As data is written (OpLog and Extent Store) the data is encrypted before it is written to disk at the checksum boundary. This also means that data is encrypted locally and then the encrypted data is replicated to the remote CVM(s) for RF.

Encryption is the last transform applied to data before it is written to disk:

Data Encryption - Transform Application
Data Encryption - Transform Application
Encryption and Data Efficiency

Since we encrypt the data after we've applied any deduplication or compression, we ensure that all space savings from those methods are maintained. Put simply, deduplication and compression ratios will be the exact same for encrypted or non-encrypted data.

When data is read we will read the encrypted data from disk at the checksum boundary, decrypt and return the data to the guest. By doing [de/en]cryption at the checksum boundary we ensure no read amplification occurs. Given we are leveraging the Intel AES NI offload, we've seen very little impact to performance / latency.

SED Based Encryption

The figure shows a high-level overview of the architecture:

Data Encryption - SED
Data Encryption - SED

SED encryption works by splitting the storage device into "data bands" which can be in an secured or un-secured state. In the case of Nutanix, the boot and Nutanix Home partitions are trivially encrypted. All data devices and bands are heavily encrypted with big keys to level-2 standards.

When the cluster starts it will call out to the KMS server to get the keys to unlock the drives. In order to ensure security no keys are cached on the cluster. In the event of a cold boot and IPMI reset, the node will need to make a call back to the KMS server to unlock the drives. Soft reboots of the CVM will not force this to occur.

Key Management (KMS)

Nutanix provides native key management (local key manager - LKM) and storage capabilities (introduced in 5.8) as an alternative to other dedicated KMS solutions. This was introduced to negate the need for a dedicated KMS solution and simplify the environment, however external KMS are still supported.

As mentioned in the prior section, key management is a very crucial piece of any data encryption solution. Multiple keys are used throughout the stack to provide a very secure key management solution.

There are three types of keys used in the solution:

  • Data Encryption Key (DEK)
    • Key used to encrypt the data
  • Key Encryption Key (KEK)
    • Encryption key used to encrypt the DEK
  • Master Encryption Key (MEK)
    • Encryption key used to encrypt the KEK
    • Only applicable when using the Local Key Manager

The following figure shows the relationships between the various keys and KMS options:

Data Encryption - Key Management
Data Encryption - Key Management

The local key manager (LKM) service is distributed among every Nutanix node and runs natively on each CVM. The service uses a FIPS 140-2 Crypto module (under certification), and key management is transparent to the end-user besides doing any key management activities (e.g. re-key, backup keys, etc.).

When configuring data encryption, the native KMS can be leveraged by selecting 'Cluster's local KMS':

Data Encryption - Configure
Data Encryption - Configure

The master key (MEK) is split and stored across all nodes in the cluster leveraging Shamir's Secret Sharing algorithm to allow for resiliency and security. A minimum of ROUNDUP(N/2) nodes must be available to re-construct the keys, where N = number of nodes in the cluster.

Key Backups and Key Rotation

Once encryption has been enabled, it is recommended to take a backup of the data encryption key(s) (DEK). If a backup is taken, it must be secured with a strong password and stored in a secure location.

The system provides the ability to rotate (re-key) both the KEK and MEK. It automatically rotates the master key (MEK) every year, however, this operation can also be done on demand. In the event of a node add/remove, we also rotate the master key.

Distributed Storage Fabric

The Distributed Storage Fabric (DSF) appears to the hypervisor like any centralized storage array, however all of the I/Os are handled locally to provide the highest performance.  More detail on how these nodes form a distributed system can be found in the next section.

Data Structure

The Nutanix Distributed Storage Fabric (DSF) is composed of the following high-level struct:

Storage Pool
  • Key Role: Group of physical devices
  • Description: A storage pool is a group of physical storage devices including PCIe SSD, SSD, and HDD devices for the cluster.  The storage pool can span multiple Nutanix nodes and is expanded as the cluster scales.  In most configurations, only a single storage pool is leveraged.
  • Key Role: Group of VMs/files
  • Description: A container is a logical segmentation of the Storage Pool and contains a group of VM or files (vDisks).  Some configuration options (e.g., RF) are configured at the container level, however are applied at the individual VM/file level.  Containers typically have a 1 to 1 mapping with a datastore (in the case of NFS/SMB).
  • Key Role: vDisk
  • Description: A vDisk is any file over 512KB on DSF including .vmdks and VM hard disks.  vDisks are logically composed of vBlocks which make up the 'block map.'
Maximum DSF vDisk Size

No artificial limits are imposed on the vdisk size on the DSF/stargate side. As of 4.6, the vdisk size is stored as a 64 bit signed integer that stores the size in bytes. This means the theoretical maximum vDisk size can be 2^63-1 or 9E18 (9 Exabytes). Any limits below this value would be due to limitations on the client side, such as the maximum vmdk size on ESXi.

The following figure shows how these map between DSF and the hypervisor:

High-level Filesystem Breakdown
High-level Filesystem Breakdown
  • Key Role: 1MB chunk of vDisk address space
  • Description: A vBlock is a 1MB chunk of virtual address space composing a vDisk. For example, a vDisk of 100MB will have 100 x 1MB vBlocks, vBlock 0 would be for 0-1MB, vBlock 1 would be from 1-2MB, and so forth. These vBlocks map to extents which are stored as files on disk as extent groups.
  • Key Role: Logically contiguous data
  • Description: An extent is a 1MB piece of logically contiguous data which consists of n number of contiguous blocks (varies depending on guest OS block size).  Extents are written/read/modified on a sub-extent basis (aka slice) for granularity and efficiency.  An extent’s slice may be trimmed when moving into the cache depending on the amount of data being read/cached.
Extent Group
  • Key Role: Physically contiguous stored data
  • Description: An extent group is a 1MB or 4MB piece of physically contiguous stored data.  This data is stored as a file on the storage device owned by the CVM.  Extents are dynamically distributed among extent groups to provide data striping across nodes/disks to improve performance.  NOTE: as of 4.0, extent groups can now be either 1MB or 4MB depending on deduplication.

The following figure shows how these structs relate between the various file systems: 

Low-level Filesystem Breakdown
Low-level Filesystem Breakdown

Here is another graphical representation of how these units are related:

Graphical Filesystem Breakdown
Graphical Filesystem Breakdown

I/O Path and Cache

For a visual explanation, you can watch the following video: LINK

The typical hyperconverged storage I/O path can be characterized into the following core layers:

  1. Guest OS (UVM) to virtual disk(s)
    • This remains unchanged with Nutanix. Depending on the hypervisor the guest OS will use a device driver to talk to a virtual disk device. Depending on the hypervisor this could be virtio-scsi (AHV), pv-scsi (ESXi), etc. The virtual disks will also vary based upon the hypervisor (e.g. vmdk, vhd, etc.)
  2. Hypervisor to DSF (via CVM)
    • Communication between the hypervisor and Nutanix occurs via standard storage protocols (e.g. iSCSI, NFS, SMBv3) over the local interface of the CVM and hypervisor. At this point all communication has been local to the host (there are scenarios where I/O will be remote (e.g. local CVM down, etc.).
  3. Nutanix I/O path
    • This is all transparent to the hypervisor and UVMs and it native to the Nutanix platform.

The following image shows a high-level overview of these layers:

High-level I/O Path
High-level I/O Path - Traditional
Communication and I/O

Within the CVM the Stargate process is responsible for handling all storage I/O requests and interaction with other CVMs / physical devices. Storage device controllers are passed through directly to the CVM so all storage I/O bypasses the hypervisor.

The following image shows a high-level overview of the traditional I/O path:

High-level I/O Path
High-level I/O Path

Nutanix BlockStore is an AOS capability (currently in development) which creates an extensible filesystem and block management layer all handled in user space.This eliminates the filesystem from the devices and removes the invoking of any filesystem kernel driver. The introduction of newer storage media (e.g. NVMe), devices now come with user space libraries to handle device I/O directly (e.g. SPDK) eliminating the need to make any system calls (context switches). With the combination of BlockStore + SPDK all Stargate device interaction has moved into user space eliminating any context switching or kernel driver invocation.

Stargate - Device I/O Path
Stargate - Device I/O Path

The following image shows a high-level overview of the updated I/O path with BlockStore + SPDK:

High-level I/O Path - BlockStore
High-level I/O Path - BlockStore

To perform data replication the CVMs communicate over the network. With the default stack this will invoke kernel level drivers to do so.

However, with RDMA these NICs are passed through to the CVM bypassing anything in the hypervisor. Also, within the CVM all network traffic using RDMA only uses a kernel level driver for the control path, then all actual data I/O is done in user-space without any context switches.

The following image shows a high-level overview of the I/O path with RDMA:

High-level I/O Path - RDMA
High-level I/O Path - RDMA

To summarize, the following enhancements optimize with the following:

  1. PCI passthrough bypasses the hypervisor for device I/O
  2. SPDK + Blockstore eliminates kernel storage driver interactions and moves them to user-space
  3. RDMA bypasses the hypervisor and all data transfer is done in CVM user-space
Stargate I/O Logic

Within the CVM the Stargate process is responsible for handling all I/O coming from user VMs (UVMs) and persistence (RF, etc.). When a write request comes to Stargate, there is a write characterizer which will determine if the write gets persisted to the OpLog, Extent Store, or Autonomous Extent Store. Similarly for reads the read characterizer is responsible to handling reads and managing caching / readahead.

The Nutanix I/O path is composed of the following high-level components:

DSF I/O Path
DSF I/O Path

*As of AOS 5.10, the Autonomous Extent Store (AES) can be used to handle sustained random workloads when requisite conditions are met.

^In all-flash node configurations the Extent Store will only consist of SSD devices and no tier ILM will occur as only a single flash tier exists. In cases where hybrid flash is used (e.g. NVMe, Intel Optane, etc. + SATA SSD) the highest performance media will be Tier 0 and the lower performance media will be Tier 1. For hybrid, non all flash scenarios, the flash would be Tier 0 with HDD being Tier 1.

  • Key Role: Persistent write buffer
  • Description: The OpLog is similar to a filesystem journal and is built as a staging area to handle bursts of random writes, coalesce them, and then sequentially drain the data to the extent store.  Upon a write, the OpLog is synchronously replicated to another n number of CVM’s OpLog before the write is acknowledged for data availability purposes.  All CVM OpLogs partake in the replication and are dynamically chosen based upon load.  The OpLog is stored on the SSD tier on the CVM to provide extremely fast write I/O performance, especially for random I/O workloads. All SSD devices participate and handle a portion of OpLog storage. For sequential workloads, the OpLog is bypassed and the writes go directly to the extent store.  If data is currently sitting in the OpLog and has not been drained, all read requests will be directly fulfilled from the OpLog until they have been drained, where they would then be served by the extent store/unified cache.  For containers where fingerprinting (aka Dedupe) has been enabled, all write I/Os will be fingerprinted using a hashing scheme allowing them to be deduplicated based upon fingerprint in the unified cache.
Per-vDisk OpLog Sizing

The OpLog is a shared resource, however allocation is done on a per-vDisk basis to ensure each vDisk has an equal opportunity to leverage. This is implemented through a per-vDisk OpLog limit (max amount of data per-vDisk in the OpLog). VMs with multiple vDisk(s) will be able to leverage the per-vDisk limit times the number of disk(s).

The per-vDisk OpLog limit is currently 6GB (as of 4.6), up from 2GB in prior versions.

This is controlled by the following Gflag: vdisk_distributed_oplog_max_dirty_MB.

Extent Store
  • Key Role: Persistent data storage
  • Description: The Extent Store is the persistent bulk storage of DSF and spans all device tiers (PCIe SSD, SATA SSD, HDD) and is extensible to facilitate additional devices/tiers.  Data entering the extent store is either being A) drained from the OpLog or B) is sequential/sustained in nature and has bypassed the OpLog directly.  Nutanix ILM will determine tier placement dynamically based upon I/O patterns and will move data between tiers.
Sequential Write Characterization

Write IO is deemed as sequential when there is more than 1.5MB of outstanding write IO to a vDisk (as of 4.6). IOs meeting this will bypass the OpLog and go directly to the Extent Store since they are already large chunks of aligned data and won't benefit from coalescing.

This is controlled by the following Gflag: vdisk_distributed_oplog_skip_min_outstanding_write_bytes.

All other IOs, including those which can be large (e.g. >64K) will still be handled by the OpLog.

Autonomous Extent Store (AES)
  • Key Role: Persistent data storage
  • Description: The Autonomous Extent Store (AES) is a new method for writing / storing data in the Extent Store introduced in AOS 5.10. It leverages a mix of primarily local + global metadata (more detail in the 'Scalable Metadata' section following) allowing for much more efficient sustained performance due to metadata locality. For sustained random write workloads, these will bypass the OpLog and be written directly to the Extent Store using AES. For bursty random workloads these will take the typical OpLog I/O path then drain to the Extent Store using AES where possible. NOTE: As of 5.11.1, for AES to be enabled, the node must have a minimum of 8 flash devices or any amount of flash devices if at least one device is NVMe.
Unified Cache
  • Key Role: Dynamic read cache
  • Description: The Unified Cache is a read cache which is used for data, metadata and deduplication and stored in the CVM's memory. Upon a read request of data not in the cache (or based upon a particular fingerprint), the data will be placed into the single-touch pool of the Unified Cache which completely sits in memory, where it will use LRU (least recently used) until it is evicted from the cache.  Any subsequent read request will “move” (no data is actually moved, just cache metadata) the data into the multi-touch pool. Any read request for data in the multi-touch pool will cause the data to go to the peak of the multi-touch pool where it will be given a new LRU counter. Cache size can be calculated using the following formula: ((CVM Memory - 12 GB) * 0.45). For example a 32GB CVM would have the following cache size: ((32 - 12)*0.45) == 9GB.

The following figure shows a high-level overview of the Unified Cache:

DSF Unified Cache
DSF Unified Cache
Cache Granularity and Logic

Data is brought into the cache at a 4K granularity and all caching is done real-time (e.g. no delay or batch process data to pull data into the cache).

Each CVM has its own local cache that it manages for the vDisk(s) it is hosting (e.g. VM(s) running on the same node). When a vDisk is cloned (e.g. new clones, snapshots, etc.) each new vDisk has its own block map and the original vDisk is marked as immutable. This allows us to ensure that each CVM can have it's own cached copy of the base vDisk with cache coherency.

In the event of an overwrite, that will be re-directed to a new extent in the VM's own block map. This ensures that there will not be any cache corruption.

Extent Cache
  • Key Role: In-memory read cache
  • Description: The Extent Cache is an in-memory read cache that is completely in the CVM’s memory.  This will store non-fingerprinted extents for containers where fingerprinting and deduplication are disabled.  As of version 3.5, this is separate from the Unified Cache, however these are merged in 4.5 with the unified cache.

Scalable Metadata

For a visual explanation, you can watch the following video: LINK

Metadata is at the core of any intelligent system and is even more critical for any filesystem or storage array. For those unsure about the term 'metadata'; essentially metadata is 'data about data'. In terms of DSF, there are a few key principles that are critical for its success:

  • Must be right 100% of the time (known as "strictly consistent")
  • Must be ACID compliant
  • Must have unlimited scalability
  • Must not have any bottlenecks at any scale (must be linearly scalable)

As of AOS 5.10 metadata is broken into two areas: global vs. local metadata (prior all metadata was global). The motivation for this is to optimize for "metadata locality" and limit the network traffic on the system for metadata lookups.

The basis for this change is that not all data needs to be global. For example, every CVM doesn't need to know which physical disk a particular extent sits on, they just need to know which node holds that data, and only that node needs to know which disk has the data.

By doing this we can limit the amount of metadata stored by the system (eliminate metadata RF for local only data), and optimize for "metadata locality."

The following image shows the differentiation between global vs. local metadata:

Global vs. Local Metadata
Local Metadata
  • Description:
    • Local metadata store per CVM containing information only needed by the local node. This is leveraged by the Autonomous Extent Store (AES) introduced in 5.10.
  • Storage Mechanism:
    • AES DB (based on Rocksdb)
  • Types of data stored:
    • Physical extent / extent group placement (e.g. egroup to disk mappings), etc.
Global Metadata
  • Description:
    • Metadata that is globally available to any CVM and sharded across CVMs in the cluster. All metadata prior to 5.10.
  • Storage Mechanism:
    • Medusa Store (based on Cassandra)
  • Types of data stored:
    • vDisk block maps, extent to node mappings, time series stats, configurations, etc.

The section below covers how global metadata is managed:

As mentioned in the architecture section above, DSF utilizes a “ring-like” structure as a key-value store which stores essential global metadata as well as other platform data (e.g., stats, etc.). In order to ensure global metadata availability and redundancy a replication factor (RF) is utilized among an odd amount of nodes (e.g., 3, 5, etc.). Upon a global metadata write or update, the row is written to a node in the ring that owns that key and then replicated to n number of peers (where n is dependent on cluster size).  A majority of nodes must agree before anything is committed, which is enforced using the Paxos algorithm.  This ensures strict consistency for all data and global metadata stored as part of the platform.

The following figure shows an example of a global metadata insert/update for a 4 node cluster:

Cassandra Ring Structure

Performance at scale is also another important struct for DSF global metadata.  Contrary to traditional dual-controller or “leader/worker” models, each Nutanix node is responsible for a subset of the overall platform’s metadata.  This eliminates the traditional bottlenecks by allowing global metadata to be served and manipulated by all nodes in the cluster.  A consistent hashing scheme is utilized for key partitioning to minimize the redistribution of keys during cluster size modifications (also known as “add/remove node”). When the cluster scales (e.g., from 4 to 8 nodes), the nodes are inserted throughout the ring between nodes for “block awareness” and reliability.

The following figure shows an example of the global metadata “ring” and how it scales:

Cassandra Scale Out

Data Protection

For a visual explanation, you can watch the following video: LINK

The Nutanix platform currently uses a resiliency factor, also known as a replication factor (RF), and checksum to ensure data redundancy and availability in the case of a node or disk failure or corruption.  As explained above, the OpLog acts as a staging area to absorb incoming writes onto a low-latency SSD tier.  Upon being written to the local OpLog, the data is synchronously replicated to another one or two Nutanix CVM’s OpLog (dependent on RF) before being acknowledged (Ack) as a successful write to the host.  This ensures that the data exists in at least two or three independent locations and is fault tolerant. NOTE: For RF3, a minimum of 5 nodes is required since metadata will be RF5. 

OpLog peers are chosen for every episode (1GB of vDisk data) and all nodes actively participate. Multiple factors play into which peers are chosen (e.g. response time, business, capacity utilization, etc). This eliminates any fragmentation and ensures every CVM/OpLog can be used concurrently.

Data RF is configured via Prism and is done at the container level. All nodes participate in OpLog replication to eliminate any “hot nodes”, ensuring linear performance at scale.  While the data is being written, a checksum is computed and stored as part of its metadata. Data is then asynchronously drained to the extent store where the RF is implicitly maintained.  In the case of a node or disk failure, the data is then re-replicated among all nodes in the cluster to maintain the RF.  Any time the data is read, the checksum is computed to ensure the data is valid.  In the event where the checksum and data don’t match, the replica of the data will be read and will replace the non-valid copy.

Data is also consistently monitored to ensure integrity even when active I/O isn't occurring. Stargate's scrubber operation will consistently scan through extent groups and perform checksum validation when disks aren't heavily utilized. This protects against things like bit rot or corrupted sectors.

The following figure shows an example of what this logically looks like: 

DSF Data Protection
DSF Data Protection

Availability Domains

For a visual explanation, you can watch the following video: LINK

Availability Domains (aka node/block/rack awareness) is a key struct for distributed systems to abide by for determining component and data placement. Nutanix refers to a “block” as the chassis which contains either one, two, or four server “nodes” and a "rack" as a physical unit containing one or more "block". NOTE: A minimum of 3 blocks must be utilized for block awareness to be activated, otherwise node awareness will be used.

Nutanix currently supports the following levels or awareness:

  • Disk (always)
  • Node (always)
  • Block (as of AOS 4.5)
  • Rack (as of AOS 5.9)

It is recommended to utilize uniformly populated blocks / racks to ensure the awareness is enabled and no imbalance is possible. Common scenarios and the awareness level utilized can be found at the bottom of this section. The 3-block requirement is due to ensure quorum. For example, a 3450 would be a block which holds 4 nodes. The reason for distributing roles or data across blocks is to ensure if a block fails or needs maintenance the system can continue to run without interruption. NOTE: Within a block, the redundant PSU and fans are the only shared components.

NOTE: Rack awareness requires the administrator to define "racks" in which the blocks are placed.

The following shows how this is configured in Prism:

Rack Configuration
Rack Configuration

Awareness can be broken into a few key focus areas:

  • Data (The VM data)
  • Metadata (Cassandra)
  • Configuration Data (Zookeeper)

With DSF, data replicas will be written to other [blocks/racks] in the cluster to ensure that in the case of a [block/rack] failure or planned downtime, the data remains available.  This is true for both RF2 and RF3 scenarios, as well as in the case of a [block/rack] failure. An easy comparison would be “node awareness”, where a replica would need to be replicated to another node which will provide protection in the case of a node failure.  Block and rack awareness further enhances this by providing data availability assurances in the case of [block/rack] outages.

The following figure shows how the replica placement would work in a 3-block deployment:

Block/Rack Aware Replica Placement
Block/Rack Aware Replica Placement

In the case of a [block/rack] failure, [block/rack] awareness will be maintained (if possible) and the data will be replicated to other [blocks/racks] within the cluster:

Block/Rack Failure Replica Placement
Block/Rack Failure Replica Placement
Rack/Block Awareness vs. Metro clustering

A common question is can you span a cluster across two locations (rooms, buildings, etc.) and use block / rack awareness to provide resiliency around a location failure.

While theoretically possible this is not the recommended approach. Let's first think about what we're trying to achieve with this:

  1. Low RPO
  2. Low RTO (HA event instead of a DR event)

If we take the first case where we're trying to achieve an RPO ~0, it is preferred to leverage synchronous or near-synchronous replication. This will provide the same RPOs with less risk.

To minimize the RTO one can leverage a metro-cluster on top of synchronous replication and handle any failures as HA events instead of doing DR recoveries.

In summary it is preferred to leverage synchronous replication / metro clustering for the following reasons:

  • The same end result can be achieved with sync rep / metro clustering, avoiding any risks and keeping isolated fault domains
  • If network connectivity goes down between the two locations in a non-supported "stretched" deployment, one side will go down as quorum must be maintained (e.g. majority side will stay up). In the metro cluster scenario, both sides can continue to operate independently.
  • Availability domain placement of data is best effort in skewed scenarios
  • Additional Latency / reduced network bandwidth between both sites can impact performance in the "stretched" deployment
Awareness Conditions and Tolerance

Below we breakdown some common scenarios and the level of tolerance:

Desired Awareness Type FT Level EC Enabled? Min. Units Simultaneous failure tolerance
Node 1 No 3 Nodes 1 Node
Node 1 Yes 4 Nodes 1 Node
Node 2 No 5 Nodes 2 Node
Node 2 Yes 6 Nodes 2 Nodes
Block 1 No 3 Blocks 1 Block
Block 1 Yes 4 Blocks 1 Block
Block 2 No 5 Blocks 2 Blocks
Block 2 Yes 6 Blocks 2 Blocks
Rack 1 No 3 Racks 1 Rack
Rack 1 Yes 4 Racks 1 Rack
Rack 2 No 5 Racks 2 Racks
Rack 2 Yes 6 Racks 2 Racks

As of AOS base software version 4.5 and later block awareness is best effort and doesn't have strict requirements for enabling. This was done to ensure clusters with skewed storage resources (e.g. storage heavy nodes) don't disable the feature. With that stated, it is however still a best practice to have uniform blocks to minimize any storage skew.

Prior to 4.5 the following conditions must be met for block awareness:

  • If SSD or HDD tier variance between blocks is > max variance: NODE awareness
  • If SSD and HDD tier variance between blocks is < max variance: BLOCK + NODE awareness

Max tier variance is calculated as: 100 / (RF+1)

  • E.g., 33% for RF2 or 25% for RF3

As mentioned in the Scalable Metadata section above, Nutanix leverages a heavily modified Cassandra platform to store metadata and other essential information. Cassandra leverages a ring-like structure and replicates to n number of peers within the ring to ensure data consistency and availability.

The following figure shows an example of the Cassandra's ring for a 12-node cluster:

12 Node Cassandra Ring
12 Node Cassandra Ring

Cassandra peer replication iterates through nodes in a clockwise manner throughout the ring. With [block/rack] awareness, the peers are distributed among the [blocks/racks] to ensure no two peers are on the same [block/rack].

The following figure shows an example node layout translating the ring above into the [block/rack] based layout:

Cassandra Node Block/Rack Aware Placement
Cassandra Node Block/Rack Aware Placement

With this [block/rack]-aware nature, in the event of a [block/rack] failure there will still be at least two copies of the data (with Metadata RF3 – In larger clusters RF5 can be leveraged).

The following figure shows an example of all of the nodes replication topology to form the ring (yes – it’s a little busy):

Full Cassandra Node Block/Rack Aware Placement
Full Cassandra Node Block/Rack Aware Placement
Metadata Awareness Conditions

Below we breakdown some common scenarios and what level of awareness will be utilized:

  • FT1 (Data RF2 / Metadata RF3) will be block aware if:
    • >= 3 blocks
    • Let X be the number of nodes in the block with max nodes. Then, the remaining blocks should have at least 2X nodes.
      • Example: 4 blocks with 2,3,4,2 nodes per block respectively.
        • The max node block has 4 nodes which means the other 3 blocks should have 2x4 (8) nodes. In this case it WOULD NOT be block aware as the remaining blocks only have 7 nodes.

      • Example: 4 blocks with 3,3,4,3 nodes per block respectively.
        • The max node block has 4 nodes which means the other 3 blocks should have 2x4==8 nodes. In this case it WOULD be block aware as the remaining blocks have 9 nodes which is above our minimum.
  • FT2 (Data RF3 / Metadata RF5) will be block aware if:
    • >= 5 blocks
    • Let X be the number of nodes in the block with max nodes. Then, the remaining blocks should have at least 4X nodes.
      • Example: 6 blocks with 2,3,4,2,3,3 nodes per block respectively.
        • The max node block has 4 nodes which means the other 3 blocks should have 4x4==16 nodes. In this case it WOULD NOT be block aware as the remaining blocks only have 13 nodes.

      • Example: 6 blocks with 2,4,4,4,4,4 nodes per block respectively.
        • The max node block has 4 nodes which means the other 3 blocks should have 4x4==16 nodes. In this case it WOULD be block aware as the remaining blocks have 18 nodes which is above our minimum.
Configuration Data

Nutanix leverages Zookeeper to store essential configuration data for the cluster.  This role is also distributed in a [block/rack]-aware manner to ensure availability in the case of a [block/rack] failure.

The following figure shows an example layout showing 3 Zookeeper nodes distributed in a [block/rack]-aware manner:

Zookeeper Block/Rack Aware Placement
Zookeeper Block/Rack Aware Placement

In the event of a [block/rack] outage, meaning one of the Zookeeper nodes will be gone, the Zookeeper role would be transferred to another node in the cluster as shown below:

Zookeeper Placement Block/Rack Failure
Zookeeper Placement Block/Rack Failure

When the [block/rack] comes back online, the Zookeeper role would be transferred back to maintain [block/rack] awareness.

NOTE: Prior to 4.5, this migration was not automatic and must be done manually.

Data Path Resiliency

Reliability and resiliency are key, if not the most important concepts within DSF or any primary storage platform. 

Contrary to traditional architectures which are built around the idea that hardware will be reliable, Nutanix takes a different approach: it expects hardware will eventually fail.  By doing so, the system is designed to handle these failures in an elegant and non-disruptive manner.

NOTE: That doesn’t mean the hardware quality isn’t there, just a concept shift.  The Nutanix hardware and QA teams undergo an exhaustive qualification and vetting process.

As mentioned in the prior sections metadata and data are protected using a RF which is based upon the cluster FT level. As of 5.0 supported FT levels are FT1 and FT2 which correspond to metadata RF3 and data RF2, or metadata RF5 and data RF3 respectively.

To learn more about how metadata is sharded refer to the prior 'Scalable Metadata' section. To learn more about how data is protected refer to the prior 'Data protection' section.

In a normal state, cluster data layout will look similar to the following:

Data Path Resiliency - Normal State
Data Path Resiliency - Normal State

As you can see the VM/vDisk data has 2 or 3 copies on disk which are distributed among the nodes and associated storage devices.

Importance of Data Distribution

By ensuring metadata and data is distributed across all nodes and all disk devices we can ensure the highest possible performance during normal data ingest and re-protection.

As data is ingested into the system its primary and replica copies will be distributed across the local and all other remote nodes. By doing so we can eliminate any potential hot spots (e.g. a node or disk performing slowly) and ensure a consistent write performance.

In the event of a disk or node failure where data must be re-protected, the full power of the cluster can be used for the rebuild. In this event the scan of metadata (to find out the data on the failed device(s) and where the replicas exist) will be distributed evenly across all CVMs. Once the data replicas have been found all healthy CVMs, disk devices (SSD+HDD), and host network uplinks can be used concurrently to rebuild the data.

For example, in a 4 node cluster where a disk fails each CVM will handle 25% of the metadata scan and data rebuild. In a 10 node cluster, each CVM will handle 10% of the metadata scan and data rebuild. In a 50 node cluster, each CVM will handle 2% of the metadata scan and data rebuild.

Key point: With Nutanix and by ensuring uniform distribution of data we can ensure consistent write performance and far superior re-protection times. This also applies to any cluster wide activity (e.g. erasure coding, compression, deduplication, etc.)

Comparing this to other solutions where HA pairs are used or a single disk holds a full copy of the data, they will face frontend performance issues if the mirrored node/disk is under duress (facing heavy IO or resource constraints).

Also, in the event of a failure where data must be re-protected, they will be limited by a single controller, a single node's disk resources and a single node's network uplinks. When terabytes of data must be re-replicated this will be severely constrained by the local node's disk and network bandwidth, increasing the time the system is in a potential data loss state if another failure occurs.

Potential levels of failure

Being a distributed system, DSF is built to handle component, service, and CVM failures, which can be characterized on a few levels:

  • Disk Failure
  • CVM “Failure”
  • Node Failure
When does a rebuild begin?

When there is an unplanned failure (in some cases we will proactively take things offline if they aren't working correctly) we begin the rebuild process immediately.

Unlike some other vendors which wait 60 minutes to start rebuilding and only maintain a single copy during that period (very risky and can lead to data loss if there's any sort of failure), we are not willing to take that risk at the sacrifice of potentially higher storage utilization.

We can do this because of a) the granularity of our metadata b) choose peers for write RF dynamically (while there is a failure, all new data (e.g. new writes / overwrites) maintain their configured redundancy) and c) we can handle things coming back online during a rebuild and re-admit the data once it has been validated. In this scenario data may be "over-replicated" in which a Curator scan will kick off and remove the over-replicated copies.

Disk Failure

A disk failure can be characterized as just that, a disk which has either been removed, encounters a failure, or one that is not responding or has I/O errors. When Stargate sees I/O errors or the device fails to respond within a certain threshold it will mark the disk offline. Once that has occurred Hades will run S.M.A.R.T. and check the status of the device. If the tests pass the disk will be marked online, if they fail it will remain offline. If Stargate marks a disk offline multiple times (currently 3 times in an hour), Hades will stop marking the disk online even if S.M.A.R.T. tests pass.

VM impact:

  • HA event: No
  • Failed I/Os: No
  • Latency: No impact

In the event of a disk failure, a Curator scan (MapReduce Framework) will occur immediately.  It will scan the metadata (Cassandra) to find the data previously hosted on the failed disk and the nodes / disks hosting the replicas.

Once it has found that data that needs to be “re-replicated”, it will distribute the replication tasks to the nodes throughout the cluster. 

During this process a Drive Self Test (DST) is started for the bad disk and SMART logs are monitored for errors.

The following figure shows an example disk failure and re-protection:

Data Path Resiliency - Disk Failure
Data Path Resiliency - Disk Failure

An important thing to highlight here is given how Nutanix distributes data and replicas across all nodes / CVMs / disks; all nodes / CVMs / disks will participate in the re-replication. 

This substantially reduces the time required for re-protection, as the power of the full cluster can be utilized; the larger the cluster, the faster the re-protection.

Node Failure

VM Impact:

  • HA event: Yes
  • Failed I/Os: No
  • Latency: No impact

In the event of a node failure, a VM HA event will occur restarting the VMs on other nodes throughout the virtualization cluster.  Once restarted, the VMs will continue to perform I/Os as usual which will be handled by their local CVMs.

Similar to the case of a disk failure above, a Curator scan will find the data previously hosted on the node and its respective replicas. Once the replicas are found all nodes will participate in the reprotection.

Data Path Resiliency - Node Failure
Data Path Resiliency - Node Failure

In the event where the node remains down for a prolonged period of time (30 minutes as of 4.6), the down CVM will be removed from the metadata ring.  It will be joined back into the ring after it has been up and stable for a duration of time.

Pro tip

Data resiliency state will be shown in Prism on the dashboard page.

You can also check data resiliency state via the cli:

# Node status
ncli cluster get-domain-fault-tolerance-status type=node

# Block status
ncli cluster get-domain-fault-tolerance-status type=rackable_unit

These should always be up to date, however to refresh the data you can kick off a Curator partial scan.

CVM “Failure”

A CVM "failure” can be characterized as a CVM power action causing the CVM to be temporarily unavailable.  The system is designed to transparently handle these gracefully.  In the event of a failure, I/Os will be re-directed to other CVMs within the cluster.  The mechanism for this will vary by hypervisor. 

The rolling upgrade process actually leverages this capability as it will upgrade one CVM at a time, iterating through the cluster.

VM impact:

  • HA event: No
  • Failed I/Os: No
  • Latency: Potentially higher given I/Os over the network

In the event of a CVM "failure” the I/O which was previously being served from the down CVM, will be forwarded to other CVMs throughout the cluster.  ESXi and Hyper-V handle this via a process called CVM Autopathing, which leverages (like “happy”), where it will modify the routes to forward traffic going to the internal address ( to the external IP of other CVMs throughout the cluster.  This enables the datastore to remain intact, just the CVM responsible for serving the I/Os is remote.

Once the local CVM comes back up and is stable, the route would be removed and the local CVM would take over all new I/Os.

In the case of AHV, iSCSI multi-pathing is leveraged where the primary path is the local CVM and the two other paths would be remote.  In the event where the primary path fails, one of the other paths will become active.

Similar to Autopathing with ESXi and Hyper-V, when the local CVM comes back online, it’ll take over as the primary path.

Capacity Optimization

The Nutanix platform incorporates a wide range of storage optimization technologies that work in concert to make efficient use of available capacity for any workload. These technologies are intelligent and adaptive to workload characteristics, eliminating the need for manual configuration and fine-tuning.

The following optimizations are leveraged:

  • Erasure Coding (EC-X)
  • Compression
  • Deduplication

More detail on how each of these features can be found in the following sections.

The table describes which optimizations are applicable to workloads a high-level:

Data Transform Best suited Application(s) Comments
Erasure Coding (EC-X) Most, Ideal for Nutanix Files/Objects Provides higher availability with reduced overheads than traditional RF. No impact to normal write or read I/O performance. Does have some read overhead in the case of a disk / node / block failure where data must be decoded.
All No impact to random I/O, helps increase storage tier utilization. Benefits large or sequential I/O performance by reducing data to replicate and read from disk.
None Given inline compression will compress only large or sequential writes inline and do random or small I/Os post-process, that should be used instead.
Perf Tier
P2V/V2V,Hyper-V (ODX),Cross-container clones Greater cache efficiency for data which wasn't cloned or created using efficient AOS clones
Capacity Tier
Same as perf tier dedup Benefits of above with reduced overhead on disk

Erasure Coding

The Nutanix platform leverages a replication factor (RF) for data protection and availability.  This method provides the highest degree of availability because it does not require reading from more than one storage location or data re-computation on failure.  However, this does come at the cost of storage resources as full copies are required. 

To provide a balance between availability while reducing the amount of storage required, DSF provides the ability to encode data using erasure codes (EC).

Similar to the concept of RAID (levels 4, 5, 6, etc.) where parity is calculated, EC encodes a strip of data blocks on different nodes and calculates parity.  In the event of a host and/or disk failure, the parity can be leveraged to calculate any missing data blocks (decoding).  In the case of DSF, the data block is an extent group. Based upon the read nature of the data (read cold vs. read hot), the system will determine placement of the blocks in the strip.

For data that is read cold, we will prefer to distribute the data blocks from the same vDisk across nodes to form the strip (same-vDisk strip). This simplifies garbage collection (GC) as the full strip can be removed in the event the vDisk is deleted. For read hot data we will prefer to keep the vDisk data blocks local to the node and compose the strip with data from different vDisks (cross-vDisk strip). This minimizes remote reads as the local vDisk's data blocks can be local and other VMs/vDisks can compose the other data blocks in the strip. In the event a read cold strip becomes hot, the system will try to recompute the strip and localize the data blocks.

The number of data and parity blocks in a strip is configurable based upon the desired failures to tolerate.  The configuration is commonly referred to as the number of <data blocks>/<number of parity blocks>.

For example, “RF2 like” availability (e.g., N+1) could consist of 3 or 4 data blocks and 1 parity block in a strip (e.g., 3/1 or 4/1).  “RF3 like” availability (e.g. N+2) could consist of 3 or 4 data blocks and 2 parity blocks in a strip (e.g. 3/2 or 4/2).

EC + Block Awareness

As of 5.8, EC can place data and parity blocks in a block aware manner (prior to 5.8 this was done at a node level).

Pre-existing EC containers will not immediately change to block aware placement after being upgraded to 5.8. If there are enough blocks (strip size (k+n) + 1) available in the cluster these previously node aware strips will move to block aware. New EC containers will build block aware EC strips.

The expected overhead can be calculated as <# parity blocks> / <# data blocks>.  For example, a 4/1 strip has a 25% overhead or 1.25X compared to the 2X of RF2.  A 4/2 strip has a 50% overhead or 1.5X compared to the 3X of RF3.

The following table characterizes the encoded strip sizes and example overheads:

FT1 (RF2 equiv.)
FT2 (RF3 equiv.)
Cluster Size
EC Strip Size
(data/parity blocks)
EC Overhead
(vs. 2X of RF2)
EC Strip Size
EC Overhead
(vs. 3X of RF3)
4 2/1 1.5X N/A N/A
5 3/1 1.33X N/A N/A
6 4/1 1.25X 2/2 2X
7 4/1 1.25X 3/2 1.6X
8+ 4/1 1.25X 4/2 1.5X
Pro tip

It is always recommended to have a cluster size which has at least 1 more node (or block for block aware data / parity placement) than the combined strip size (data + parity) to allow for rebuilding of the strips in the event of a node or block failure. This eliminates any computation overhead on reads once the strips have been rebuilt (automated via Curator). For example, a 4/1 strip should have at least 6 nodes in the cluster for a node aware EC strip or 6 blocks for a block aware EC strip. The previous table follows this best practice.

The encoding is done post-process and leverages the Curator MapReduce framework for task distribution.  Since this is a post-process framework, the traditional write I/O path is unaffected.

A normal environment using RF would look like the following:

Typical DSF RF Data Layout
Typical DSF RF Data Layout

In this scenario, we have a mix of both RF2 and RF3 data whose primary copies are local and replicas are distributed to other nodes throughout the cluster.

When a Curator full scan runs, it will find eligible extent groups which are available to become encoded. Eligible extent groups must be "write-cold" meaning they haven't been written to for awhile. This is controlled with the following Curator Gflag: curator_erasure_code_threshold_seconds. After the eligible candidates are found, the encoding tasks will be distributed and throttled via Chronos.

The following figure shows an example 4/1 and 3/2 strip:

DSF Encoded Strip - Pre-savings
DSF Encoded Strip - Pre-savings

Once the data has been successfully encoded (strips and parity calculation), the replica extent groups are then removed.

The following figure shows the environment after EC has run with the storage savings:

DSF Encoded Strip - Post-savings
DSF Encoded Strip - Post-savings
Pro tip

Erasure Coding pairs perfectly with inline compression which will add to the storage savings. I leverage inline compression + EC in my environments.


For a visual explanation, you can watch the following video: LINK

The Nutanix Capacity Optimization Engine (COE) is responsible for performing data transformations to increase data efficiency on disk.  Currently compression is one of the key features of the COE to perform data optimization. DSF provides both inline and offline flavors of compression to best suit the customer’s needs and type of data. As of 5.1, offline compression is enabled by default.

Inline compression will compress sequential streams of data or large I/O sizes (>64K) when written to the Extent Store (SSD + HDD). This includes data draining from OpLog as well as sequential data skipping it.

OpLog Compression

As of 5.0, the OpLog will now compress all incoming writes >4K that show good compression (Gflag: vdisk_distributed_oplog_enable_compression). This will allow for a more efficient utilization of the OpLog capacity and help drive sustained performance.

When drained from OpLog to the Extent Store the data will be decompressed, aligned and then re-compressed at a 32K aligned unit size (as of 5.1).

This feature is on by default and no user configuration is necessary.

Offline compression will initially write the data as normal (in an un-compressed state) and then leverage the Curator framework to compress the data cluster wide. When inline compression is enabled but the I/Os are random in nature, the data will be written un-compressed in the OpLog, coalesced, and then compressed in memory before being written to the Extent Store.

Nutanix leverages LZ4 and LZ4HC for data compression with AOS 5.0 and beyond. Prior to AOS 5.0 the Google Snappy compression library is leveraged which provides good compression ratios with minimal computational overhead and extremely fast compression / decompression rates.

Normal data will be compressed using LZ4 which provides a very good blend between compression and performance. For cold data, LZ4HC will be leveraged to provide an improved compression ratio.

Cold data is characterized into two main categories:

  • Regular data: No R/W access for 3 days (Gflag: curator_medium_compress_mutable_data_delay_secs)
  • Immutable data (snapshots): No R/W access for 1 day (Gflag: curator_medium_compress_immutable_data_delay_secs)

The following figure shows an example of how inline compression interacts with the DSF write I/O path:

Inline Compression I/O Path
Inline Compression I/O Path
Pro tip

Almost always use inline compression (compression delay = 0) as it will only compress larger / sequential writes and not impact random write performance.

This will also increase the usable size of the SSD tier increasing effective performance and allowing more data to sit in the SSD tier. Also, for larger or sequential data that is written and compressed inline, the replication for RF will be shipping the compressed data, further increasing performance since it is sending less data across the wire.

Inline compression also pairs perfectly with erasure coding.

For offline compression, all new write I/O is written in an un-compressed state and follows the normal DSF I/O path.  After the compression delay (configurable) is met, the data is eligible to become compressed. Compression can occur anywhere in the Extent Store. Offline compression uses the Curator MapReduce framework and all nodes will perform compression tasks.  Compression tasks will be throttled by Chronos.

The following figure shows an example of how offline compression interacts with the DSF write I/O path:

Offline Compression I/O Path
Offline Compression I/O Path

For read I/O, the data is first decompressed in memory and then the I/O is served.

You can view the current compression rates via Prism on the Storage > Dashboard page.

Elastic Dedupe Engine

For a visual explanation, you can watch the following video: LINK

The Elastic Dedupe Engine is a software-based feature of DSF which allows for data deduplication in the capacity (Extent Store) and performance (Unified Cache) tiers.  Streams of data are fingerprinted during ingest using a SHA-1 hash at a 8K granularity (Controlled by: stargate_dedup_fingerprint).  This fingerprint is only done on data ingest and is then stored persistently as part of the written block’s metadata. Deduplicated data is pulled into the unified cache at a 4K granularity.

Contrary to traditional approaches which utilize background scans requiring the data to be re-read, Nutanix performs the fingerprint inline on ingest.  For duplicate data that can be deduplicated in the capacity tier, the data does not need to be scanned or re-read, essentially duplicate copies can be removed.

To make the metadata overhead more efficient, fingerprint refcounts are monitored to track dedupability. Fingerprints with low refcounts will be discarded to minimize the metadata overhead. To minimize fragmentation full extents will be preferred for capacity tier deduplication.

Pro tip

Use performance tier deduplication on your base images (you can manually fingerprint them using vdisk_manipulator) to take advantage of the unified cache.

Use capacity tier deduplication for P2V / V2V, when using Hyper-V since ODX does a full data copy, or when doing cross-container clones (not usually recommended as a single container is preferred).

In most other cases compression will yield the highest capacity savings and should be used instead.

The following figure shows an example of how the Elastic Dedupe Engine scales and handles local VM I/O requests:

Elastic Dedupe Engine - Scale
Elastic Dedupe Engine - Scale

Fingerprinting is done during data ingest of data with an I/O size of 64K or greater (initial I/O or when draining from OpLog).  Intel acceleration is leveraged for the SHA-1 computation which accounts for very minimal CPU overhead.  In cases where fingerprinting is not done during ingest (e.g., smaller I/O sizes), fingerprinting can be done as a background process. The Elastic Deduplication Engine spans both the capacity tier (Extent Store), but also the performance tier (Unified Cache).  As duplicate data is determined, based upon multiple copies of the same fingerprints, a background process will remove the duplicate data using the DSF MapReduce framework (Curator). For data that is being read, the data will be pulled into the DSF Unified Cache which is a multi-tier/pool cache.  Any subsequent requests for data having the same fingerprint will be pulled directly from the cache.  To learn more about the Unified Cache and pool structure, please refer to the 'Unified Cache' sub-section in the I/O path overview.

Fingerprinted vDisk Offsets

As of 4.6.1 there is no limit and the full vDisk can be fingerprinted / deduped.

Prior to 4.6.1 this was increased to 24GB due to higher metadata efficiencies. Prior to 4.5 only the first 12GB of a vDisk was eligible to be fingerprinted. This was done to maintain a smaller metadata footprint and since the OS is normally the most common data.

The following figure shows an example of how the Elastic Dedupe Engine interacts with the DSF I/O path:

EDE I/O Path
EDE I/O Path

You can view the current deduplication rates via Prism on the Storage > Dashboard page.

Dedup + Compression

As of 4.5 both deduplication and compression can be enabled on the same container. However, unless the data is dedupable (conditions explained earlier in section), stick with compression.

Storage Tiering and Prioritization

The Disk Balancing section above talked about how storage capacity was pooled among all nodes in a Nutanix cluster and that ILM would be used to keep hot data local.  A similar concept applies to disk tiering, in which the cluster’s SSD and HDD tiers are cluster-wide and DSF ILM is responsible for triggering data movement events. A local node’s SSD tier is always the highest priority tier for all I/O generated by VMs running on that node, however all of the cluster’s SSD resources are made available to all nodes within the cluster.  The SSD tier will always offer the highest performance and is a very important thing to manage for hybrid arrays.

The tier prioritization can be classified at a high-level by the following:

DSF Tier Prioritization
DSF Tier Prioritization

Specific types of resources (e.g. SSD, HDD, etc.) are pooled together and form a cluster wide storage tier.  This means that any node within the cluster can leverage the full tier capacity, regardless if it is local or not.

The following figure shows a high level example of what this pooled tiering looks like:

DSF Cluster-wide Tiering
DSF Cluster-wide Tiering

A common question is what happens when a local node’s SSD becomes full?  As mentioned in the Disk Balancing section, a key concept is trying to keep uniform utilization of devices within disk tiers.  In the case where a local node’s SSD utilization is high, disk balancing will kick in to move the coldest data on the local SSDs to the other SSDs throughout the cluster.  This will free up space on the local SSD to allow the local node to write to SSD locally instead of going over the network.  A key point to mention is that all CVMs and SSDs are used for this remote I/O to eliminate any potential bottlenecks and remediate some of the hit by performing I/O over the network.

DSF Cluster-wide Tier Balancing
DSF Cluster-wide Tier Balancing

The other case is when the overall tier utilization breaches a specific threshold [curator_tier_usage_ilm_threshold_percent (Default=75)] where DSF ILM will kick in and as part of a Curator job will down-migrate data from the SSD tier to the HDD tier.  This will bring utilization within the threshold mentioned above or free up space by the following amount [curator_tier_free_up_percent_by_ilm (Default=15)], whichever is greater. The data for down-migration is chosen using last access time. In the case where the SSD tier utilization is 95%, 20% of the data in the SSD tier will be moved to the HDD tier (95% –> 75%). 

However, if the utilization was 80%, only 15% of the data would be moved to the HDD tier using the minimum tier free up amount.


DSF ILM will constantly monitor the I/O patterns and (down/up) migrate data as necessary as well as bring the hottest data local regardless of tier. The logic for up-migration (or horizontal) follows the same as that defined for egroup locality: "3 touches for random or 10 touches for sequential within a 10 minute window where multiple reads every 10 second sampling count as a single touch".

Disk Balancing

For a visual explanation, you can watch the following video: LINK

DSF is designed to be a very dynamic platform which can react to various workloads as well as allow heterogeneous node types: compute heavy (3050, etc.) and storage heavy (60X0, etc.) to be mixed in a single cluster.  Ensuring uniform distribution of data is an important item when mixing nodes with larger storage capacities. DSF has a native feature, called disk balancing, which is used to ensure uniform distribution of data throughout the cluster.  Disk balancing works on a node’s utilization of its local storage capacity and is integrated with DSF ILM.  Its goal is to keep utilization uniform among nodes once the utilization has breached a certain threshold.

NOTE: Disk balancing jobs are handled by Curator which has different priority queues for primary I/O (UVM I/O) and background I/O (e.g. disk balancing). This is done to ensure disk balancing or any other background activity doesn't impact front-end latency / performance. In this cases the job's tasks will be given to Chronos who will throttle / control the execution of the tasks. Also, movement is done within the same tier for disk balancing. For example, if I have data which is skewed in the HDD tier, I will move is amongst nodes in the same tier.

The following figure shows an example of a mixed cluster (3050 + 6050) in an “unbalanced” state:

Disk Balancing - Unbalanced State
Disk Balancing - Unbalanced State

Disk balancing leverages the DSF Curator framework and is run as a scheduled process as well as when a threshold has been breached (e.g., local node capacity utilization > n %).  In the case where the data is not balanced, Curator will determine which data needs to be moved and will distribute the tasks to nodes in the cluster. In the case where the node types are homogeneous (e.g., 3050), utilization should be fairly uniform. However, if there are certain VMs running on a node which are writing much more data than others, this can result in a skew in the per node capacity utilization.  In this case, disk balancing would run and move the coldest data on that node to other nodes in the cluster. In the case where the node types are heterogeneous (e.g., 3050 + 6020/50/70), or where a node may be used in a “storage only” mode (not running any VMs), there will likely be a requirement to move data.

The following figure shows an example the mixed cluster after disk balancing has been run in a “balanced” state:

Disk Balancing - Balanced State
Disk Balancing - Balanced State

In some scenarios, customers might run some nodes in a “storage-only” state where only the CVM will run on the node whose primary purpose is bulk storage capacity.  In this case, the full node's memory can be added to the CVM to provide a much larger read cache.

The following figure shows an example of how a storage only node would look in a mixed cluster with disk balancing moving data to it from the active VM nodes:

Disk Balancing - Storage Only Node
Disk Balancing - Storage Only Node

Snapshots and Clones

For a visual explanation, you can watch the following video: LINK

DSF provides native support for offloaded snapshots and clones which can be leveraged via VAAI, ODX, ncli, REST, Prism, etc.  Both the snapshots and clones leverage the redirect-on-write algorithm which is the most effective and efficient. As explained in the Data Structure section above, a virtual machine consists of files (vmdk/vhdx) which are vDisks on the Nutanix platform. 

A vDisk is composed of extents which are logically contiguous chunks of data, which are stored within extent groups which are physically contiguous data stored as files on the storage devices. When a snapshot or clone is taken, the base vDisk is marked immutable and another vDisk is created as read/write.  At this point, both vDisks have the same block map, which is a metadata mapping of the vDisk to its corresponding extents. Contrary to traditional approaches which require traversal of the snapshot chain (which can add read latency), each vDisk has its own block map.  This eliminates any of the overhead normally seen by large snapshot chain depths and allows you to take continuous snapshots without any performance impact.

The following figure shows an example of how this works when a snapshot is taken (NOTE: I need to give some credit to NTAP as a base for these diagrams, as I thought their representation was the clearest):

Example Snapshot Block Map
Example Snapshot Block Map

The same method applies when a snapshot or clone of a previously snapped or cloned vDisk is performed:

Multi-snap Block Map and New Write
Multi-snap Block Map and New Write

The same methods are used for both snapshots and/or clones of a VM or vDisk(s).  When a VM or vDisk is cloned, the current block map is locked and the clones are created.  These updates are metadata only, so no I/O actually takes place.  The same method applies for clones of clones; essentially the previously cloned VM acts as the “Base vDisk” and upon cloning, that block map is locked and two “clones” are created: one for the VM being cloned and another for the new clone.  There is no imposed limit on the maximum number of clones.

They both inherit the prior block map and any new writes/updates would take place on their individual block maps.

Multi-Clone Block Maps
Multi-Clone Block Maps

As mentioned previously, each VM/vDisk has its own individual block map.  So in the above example, all of the clones from the base VM would now own their block map and any write/update would occur there.

In the event of an overwrite the data will go to a new extent / extent group. For example, if I had existing data at offset o1 in extent e1 that was being overwritten, Stargate would create a new extent e2 and track that the new data was written in extent e2 at offset o2. The Vblock map tracks this down to the byte level.

The following figure shows an example of what this looks like:

Clone Block Maps - New Write
Clone Block Maps - New Write

Any subsequent clones or snapshots of a VM/vDisk would cause the original block map to be locked and would create a new one for R/W access.

Networking and I/O

The Nutanix platform does not leverage any backplane for inter-node communication and only relies on a standard 10GbE network.  All storage I/O for VMs running on a Nutanix node is handled by the hypervisor on a dedicated private network.  The I/O request will be handled by the hypervisor, which will then forward the request to the private IP on the local CVM.  The CVM will then perform the remote replication with other Nutanix nodes using its external IP over the public 10GbE network. For all read requests, these will be served completely locally in most cases and never touch the 10GbE network. This means that the only traffic touching the public 10GbE network will be DSF remote replication traffic and VM network I/O.  There will, however, be cases where the CVM will forward requests to other CVMs in the cluster in the case of a CVM being down or data being remote.  Also, cluster-wide tasks, such as disk balancing, will temporarily generate I/O on the 10GbE network.

The following figure shows an example of how the VM’s I/O path interacts with the private and public 10GbE network:

DSF Networking
DSF Networking


Data Locality

Being a converged (compute+storage) platform, I/O and data locality are critical to cluster and VM performance with Nutanix.  As explained above in the I/O path, all read/write IOs are served by the local Controller VM (CVM) which is on each hypervisor adjacent to normal VMs.  A VM’s data is served locally from the CVM and sits on local disks under the CVM’s control.  When a VM is moved from one hypervisor node to another (or during a HA event), the newly migrated VM’s data will be served by the now local CVM. When reading old data (stored on the now remote node/CVM), the I/O will be forwarded by the local CVM to the remote CVM.  All write I/Os will occur locally right away.  DSF will detect the I/Os are occurring from a different node and will migrate the data locally in the background, allowing for all read I/Os to now be served locally.  The data will only be migrated on a read as to not flood the network.

Data locality occurs in two main flavors:

  • Cache Locality
    • Pulling remote data into the local Stargate's Unified Cache. This is done at a 4K granularity.
    • For instances where there are no local replicas, the requests will be forward to the Stargate(s) containing the replicas which will return the data and the local Stargate will store this locally then return the I/O. All subsequent requests for that data will be returned from the cache.
  • Extent Group (egroup) Locality
    • Migrating the vDisk extent group(s) (egroups) to be stored in the local Stargate's Extent Store.
    • If a replica egroup is already local, no movement is necessary.
    • In this scenario the actual replica egroup will be re-localized after certain I/O thresholds are met. We don't automatically re-localize / migrate egroups to ensure we're leveraging the network efficiently.
    • For AES enabled egroups the same horizontal migration occurs for cases where replicas aren't local and the patterns are met.

The following figure shows an example of how data will “follow” the VM as it moves between hypervisor nodes:

Data Locality
Data Locality
Thresholds for Data Migration

Cache locality occurs in real time and will be determined based upon vDisk ownership. When a vDisk / VM moves from one node to another the "ownership" of those vDisk(s) will transfer to the now local CVM. Once the ownership has transferred the data can be cached locally in the Unified Cache. In the interim the cache will be wherever the ownership is held (the now remote host). The previously hosting Stargate will relinquish the vDisk token when it sees remote I/Os for 300+ seconds at which it will then be taken by the local Stargate. Cache coherence is enforced as ownership is required to cache the vDisk data.

Egroup locality is a sampled operation and an extent group will be migrated when the following occurs: "3 touches for random or 10 touches for sequential within a 10 minute window where multiple reads every 10 second sampling count as a single touch".

Shadow Clones

The Distributed Storage Fabric has a feature called ‘Shadow Clones’, which allows for distributed caching of particular vDisks or VM data which is in a ‘multi-reader’ scenario.  A great example of this is during a VDI deployment many ‘linked clones’ will be forwarding read requests to a central master or ‘Base VM’.  In the case of VMware View, this is called the replica disk and is read by all linked clones, and in XenDesktop, this is called the MCS Master VM.  This will also work in any scenario which may be a multi-reader scenario (e.g., deployment servers, repositories, etc.). Data or I/O locality is critical for the highest possible VM performance and a key struct of DSF. 

With Shadow Clones, DSF will monitor vDisk access trends similar to what it does for data locality.  However, in the case there are requests occurring from more than two remote CVMs (as well as the local CVM), and all of the requests are read I/O, the vDisk will be marked as immutable.  Once the disk has been marked as immutable, the vDisk can then be cached locally by each CVM making read requests to it (aka Shadow Clones of the base vDisk). This will allow VMs on each node to read the Base VM’s vDisk locally. In the case of VDI, this means the replica disk can be cached by each node and all read requests for the base will be served locally.  NOTE:  The data will only be migrated on a read as to not flood the network and allow for efficient cache utilization.  In the case where the Base VM is modified, the Shadow Clones will be dropped and the process will start over.  Shadow clones are enabled by default (as of 4.0.2) and can be enabled/disabled using the following NCLI command: ncli cluster edit-params enable-shadow-clones=<true/false>.

The following figure shows an example of how Shadow Clones work and allow for distributed caching:

Shadow Clones
Shadow Clones

Storage Layers and Monitoring

The Nutanix platform monitors storage at multiple layers throughout the stack, ranging from the VM/Guest OS all the way down to the physical disk devices.  Knowing the various tiers and how these relate is important whenever monitoring the solution and allows you to get full visibility of how the ops relate. The following figure shows the various layers of where operations are monitored and the relative granularity which are explained below:

Storage Layers
Storage Layers


Virtual Machine Layer
  • Key Role: Metrics reported by the hypervisor for the VM
  • Description: Virtual Machine or guest level metrics are pulled directly from the hypervisor and represent the performance the VM is seeing and is indicative of the I/O performance the application is seeing.
  • When to use: When troubleshooting or looking for VM level detail
Hypervisor Layer
  • Key Role: Metrics reported by the Hypervisor(s)
  • Description: Hypervisor level metrics are pulled directly from the hypervisor and represent the most accurate metrics the hypervisor(s) are seeing.  This data can be viewed for one of more hypervisor node(s) or the aggregate cluster.  This layer will provide the most accurate data in terms of what performance the platform is seeing and should be leveraged in most cases.  In certain scenarios the hypervisor may combine or split operations coming from VMs which can show the difference in metrics reported by the VM and hypervisor.  These numbers will also include cache hits served by the Nutanix CVMs.
  • When to use: Most common cases as this will provide the most detailed and valuable metrics.
Controller Layer
  • Key Role: Metrics reported by the Nutanix Controller(s)
  • Description: Controller level metrics are pulled directly from the Nutanix Controller VMs (e.g., Stargate 2009 page) and represent what the Nutanix front-end is seeing from NFS/SMB/iSCSI or any back-end operations (e.g., ILM, disk balancing, etc.).  This data can be viewed for one of more Controller VM(s) or the aggregate cluster.  The metrics seen by the Controller Layer should normally match those seen by the hypervisor layer, however will include any backend operations (e.g., ILM, disk balancing). These numbers will also include cache hits served by memory.  In certain cases, metrics like (IOPS), might not match as the NFS / SMB / iSCSI client might split a large IO into multiple smaller IOPS.  However, metrics like bandwidth should match.
  • When to use: Similar to the hypervisor layer, can be used to show how much backend operation is taking place.
Disk Layer
  • Key Role: Metrics reported by the Disk Device(s)
  • Description: Disk level metrics are pulled directly from the physical disk devices (via the CVM) and represent what the back-end is seeing.  This includes data hitting the OpLog or Extent Store where an I/O is performed on the disk.  This data can be viewed for one of more disk(s), the disk(s) for a particular node, or the aggregate disks in the cluster.  In common cases, it is expected that the disk ops should match the number of incoming writes as well as reads not served from the memory portion of the cache.  Any reads being served by the memory portion of the cache will not be counted here as the op is not hitting the disk device.
  • When to use: When looking to see how many ops are served from cache or hitting the disks.
Metric and Stat Retention

Metrics and time series data is stored locally for 90 days in Prism Element. For Prism Central and Insights, data can be stored indefinitely (assuming capacity is available).


Nutanix Guest Tools (NGT)

Nutanix Guest Tools (NGT) is a software based in-guest agent framework which enables advanced VM management functionality through the Nutanix Platform.

The solution is composed of the NGT installer which is installed on the VMs and the Guest Tools Framework which is used for coordination between the agent and Nutanix platform.

The NGT installer contains the following components:

  • Guest Agent Service
  • Self-service Restore (SSR) aka File-level Restore (FLR) CLI
  • VM Mobility Drivers (VirtIO drivers for AHV)
  • VSS Agent and Hardware Provider for Windows VMs
  • App Consistent snapshot support for Linux VMs (via scripts to quiesce)

This framework is composed of a few high-level components:

  • Guest Tools Service
    • Gateway between the AOS and Nutanix services and the Guest Agent. Distributed across CVMs within the cluster with an elected NGT Leader which runs on the current Prism Leader (hosting cluster vIP)
  • Guest Agent
    • Agent and associated services deployed in the VM's OS as part of the NGT installation process. Handles any local functions (e.g. VSS, Self-service Restore (SSR), etc.) and interacts with the Guest Tools Service.

The figure shows the high-level mapping of the components:

Guest Tools Mapping
Guest Tools Mapping
Guest Tools Service

The Guest Tools Service is composed of two main roles:

  • NGT Leader
    • Handles requests coming from NGT Proxy and interfaces with AOS components. A single NGT Leader is dynamically elected per cluster; in the event the current leader fails a new one will be elected. The service listens internally on port 2073.
  • NGT Proxy
    • Runs on every CVM and will forward requests to the NGT Leader to perform the desired activity. The current VM acting as the Prism Leader (hosting the VIP) will be the active CVM handling communication from the Guest Agent. Listens externally on port 2074.
Current NGT Leader

You can find the IP of the CVM hosting the NGT Leader role with the following command (run on any CVM):

nutanix_guest_tools_cli get_leader_location

The figure shows the high-level mapping of the roles:

Guest Tools Service
Guest Tools Service
Guest Agent

The Guest Agent is composed of the following high-level components as mentioned prior:

Guest Agent
Guest Agent
Communication and Security

The Guest Agent Service communicates with Guest Tools Service via the Nutanix Cluster IP using SSL. For deployments where the Nutanix cluster components and UVMs are on a different network (hopefully all), ensure that the following are possible:

  • Ensure routed communication from UVM network(s) to Cluster IP OR
  • OR
  • Create a firewall rule (and associated NAT) from UVM network(s) allowing communication with the Cluster IP on port 2074 (preferred)

The Guest Tools Service acts as a Certificate Authority (CA) and is responsible for generating certificate pairs for each NGT enabled UVM. This certificate is embedded into the ISO which is configured for the UVM and used as part of the NGT deployment process. These certificates are installed inside the UVM as part of the installation process.

NGT Agent Installation

NGT Agent installation can be performed via Prism or CLI/Scripts (ncli/REST/PowerShell).

To install NGT via Prism, navigate to the 'VM' page, select a VM to install NGT on and click 'Enable NGT':

Enable NGT for VM
Enable NGT for VM

Click 'Yes' at the prompt to continue with NGT installation:

Enable NGT Prompt
Enable NGT Prompt

The VM must have a CD-ROM as the generated installer containing the software and unique certificate will be mounted there as shown:

Enable NGT - CD-ROM
Enable NGT - CD-ROM

The NGT installer CD-ROM will be visible in the OS:

Enable NGT - CD-ROM in OS
Enable NGT - CD-ROM in OS

Double click on the CD to begin the installation process.

Silent Installation

You can perform a silent installation of the Nutanix Guest Tools by running the following command (from CD-ROM location):

NutanixGuestTools.exe /quiet /l log.txt ACCEPTEULA=yes

Follow the prompts and accept the licenses to complete the installation:

Enable NGT - Installer
Enable NGT - Installer

As part of the installation process Python, PyWin and the Nutanix Mobility (cross-hypervisor compatibility) drivers will also be installed.

After the installation has been completed, a reboot will be required.

After successful installation and reboot, you will see the following items visible in 'Programs and Features':

Enable NGT - Installed Programs
Enable NGT - Installed Programs

Services for the NGT Agent and VSS Hardware Provider will also be running:

Enable NGT - Services
Enabled NGT - Services

NGT is now installed and can be leveraged.

Bulk NGT Deployment

Rather than installing NGT on each individual VM, it is possible to embed and deploy NGT in your base image.

Follow the following process to leverage NGT inside a base image:

  1. Install NGT on leader VM and ensure communication
  2. Clone VMs from leader VM
  3. Mount NGT ISO on each clone (required to get new certificate pair)
    • Example: ncli ngt mount vm-id=<CLONE_ID> OR via Prism
    • Automated way coming soon :)
  4. Power on clones

When the cloned VM is booted it will detect the new NGT ISO and copy relevant configuration files and new certificates and will start communicating with the Guest Tools Service.

OS Customization

Nutanix provides native OS customization capabilities leveraging CloudInit and Sysprep. CloudInit is a package which handles bootstrapping of Linux cloud servers. This allows for the early initialization and customization of a Linux instance. Sysprep is a OS customization for Windows.

Some typical uses include:

  • Setting Hostname
  • Installing packages
  • Adding users / key management
  • Custom scripts
Supported Configurations

The solution is applicable to Linux guests running on AHV, including versions below (list may be incomplete, refer to documentation for a fully supported list):

  • Hypervisors:
    • AHV
  • Operating Systems:
    • Linux - most modern distributions
    • Windows - most modern distributions

In order for CloudInit to be used the following are necessary:

  • CloudInit package must be installed in Linux image

Sysprep is available by default in Windows installations.

Package Installation

CloudInit can be installed (if not already) using the following commands:

Red Hat Based Systems (CentOS, RHEL)

yum -y install CloudInit

Debian Based Systems (Ubuntu)

apt-get -y update; apt-get -y install CloudInit

Sysprep is part of the base Windows installation.

Image Customization

To leverage a custom script for OS customization, a check box and inputs is available in Prism or the REST API. This option is specified during the VM creation or cloning process:

Custom Script - Input Options
Custom Script - Input Options

Nutanix has a few options for specifying the custom script path:

  • ADSF Path
    • Use a file which has been previously upload to ADSF
  • Upload a file
    • Upload a file which will be used
  • Type or paste script
    • CloudInit script or Unattend.xml text

Nutanix passes the user data script to CloudInit or Sysprep process during first boot by creating a CD-ROM which contains the script. Once the process is complete we will remove the CD-ROM.

Input formatting

The platform supports a good amount of user data input formats, I've identified a few of the key ones below:

User-Data Script (CloudInit - Linux)

A user-data script is a simple shell script that will be executed very late in the boot process (e.g. "rc.local-like").

The scripts will begin similar to any bash script: "#!".

Below we show an example user-data script:

touch /tmp/fooTest
mkdir /tmp/barFolder

Include File (CloudInit - Linux)

The include file contains a list of urls (one per line). Each of the URLs will be read and they will be processed similar to any other script.

The scripts will begin with: "#include".

Below we show an example include script:


Cloud Config Data (CloudInit - Linux)

The cloud-config input type is the most common and specific to CloudInit.

The scripts will begin with: "#cloud-config"

Below we show an example cloud config data script:

#cloud-config # Set hostname hostname: foobar # Add user(s) users: - name: nutanix sudo: ['ALL=(ALL) NOPASSWD:ALL'] ssh-authorized-keys: - ssh-rsa: <PUB KEY> lock-passwd: false passwd: <PASSWORD> # Automatically update all of the packages package_upgrade: true package_reboot_if_required: true # Install the LAMP stack packages: - httpd - mariadb-server - php - php-pear - php-mysql # Run Commands after execution runcmd: - systemctl enable httpd

Validating CloudInit Execution

CloudInit log files can be found in /var/log/cloud-init.log and cloud-init-output.log.

Unattend.xml (Sysprep - Windows)

The unattend.xml file is the input file Sysprep uses for image customization on boot, you can read more here: LINK

The scripts will begin with: "<?xml version="1.0" ?>".

Below we show an example unattend.xml file:

<?xml version="1.0" ?> <unattend xmlns="urn:schemas-microsoft-com:unattend"> <settings pass="windowsPE"> <component name="Microsoft-Windows-Setup" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" processorArchitecture="x86"> <WindowsDeploymentServices> <Login> <WillShowUI>OnError</WillShowUI> <Credentials> <Username>username</Username> <Domain></Domain> <Password>my_password</Password> </Credentials> </Login> <ImageSelection> <WillShowUI>OnError</WillShowUI> <InstallImage> <ImageName>Windows Vista with Office</ImageName> <ImageGroup>ImageGroup1</ImageGroup> <Filename>Install.wim</Filename> </InstallImage> <InstallTo> <DiskID>0</DiskID> <PartitionID>1</PartitionID> </InstallTo> </ImageSelection> </WindowsDeploymentServices> <DiskConfiguration> <WillShowUI>OnError</WillShowUI> <Disk> <DiskID>0</DiskID> <WillWipeDisk>false</WillWipeDisk> <ModifyPartitions> <ModifyPartition> <Order>1</Order> <PartitionID>1</PartitionID> <Letter>C</Letter> <Label>TestOS</Label> <Format>NTFS</Format> <Active>true</Active> <Extend>false</Extend> </ModifyPartition> </ModifyPartitions> </Disk> </DiskConfiguration> </component> <component name="Microsoft-Windows-International-Core-WinPE" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" processorArchitecture="x86"> <SetupUILanguage> <WillShowUI>OnError</WillShowUI> <UILanguage>en-US</UILanguage> </SetupUILanguage> <UILanguage>en-US</UILanguage> </component> </settings> </unattend>

Karbon (Container Services)

Nutanix provides the ability to leverage persistent containers on the Nutanix platform using Kubernetes (currently). It was previously possible to run Docker on Nutanix platform; however, data persistence was an issue given the ephemeral nature of containers.

Container technologies like Docker are a different approach to hardware virtualization. With traditional virtualization each VM has its own Operating System (OS) but they share the underlying hardware. Containers, which include the application and all its dependencies, run as isolated processes that share the underlying Operating System (OS) kernel.

The following table shows a simple comparison between VMs and Containers:

Metric Virtual Machines (VM) Containers
Virtualization Type Hardware-level virtualization OS kernel virtualization
Overhead Heavyweight Lightweight
Provisioning Speed Slower (seconds to minutes) Real-time / fast (us to ms)
Performance Overhead Limited performance Native performance
Security Fully isolated (more secure) Process-level isolation (less secure)
Supported Configurations

The solution is applicable to the configurations below (list may be incomplete, refer to documentation for a fully supported list):

  • AHV
    Container System(s)*:
  • Docker 1.13

*As of 4.7, the solution only supports storage integration with Docker based containers. However, any other container system can run as a VM on the Nutanix platform.

Container Services Constructs

The following entities compose Karbon Container Services:

  • Nutanix Docker Machine Driver: Handles Docker container host provisioning via Docker Machine and the AOS Image Service
  • Nutanix Docker Volume Plugin: Responsible for interfacing with AOS Volumes to create, mount, format and attach volumes to the desired container

The following entities compose Docker (note: not all are required):

  • Docker Image: The basis and image for a container
  • Docker Registry: Holding space for Docker Images
  • Docker Hub: Online container marketplace (public Docker Registry)
  • Docker File: Text file describing how to construct the Docker image
  • Docker Container: Running instantiation of a Docker Image
  • Docker Engine: Creates, ships and runs Docker containers
  • Docker Swarm: Docker host clustering / scheduling platform
  • Docker Daemon: Handles requests from Docker Client and does heavy lifting of building, running and distributing containers
  • Docker Store: Marketplace for trusted and enterprise ready containers

The Nutanix solution currently leverages Docker Engine running in VMs which are created using Docker Machine. These machines can run in conjunction with normal VMs on the platform.

Docker - High-level Architecture
Docker - High-level Architecture

Nutanix has developed a Docker Volume Plugin which will create, format and attach a volume to container(s) using the AOS Volumes feature. This allows the data to persist as a container is power cycled / moved.

Data persistence is achieved by using the Nutanix Volume Plugin which will leverage AOS Volumes to attach a volume to the host / container:

Docker - Volumes
Docker - Volumes

In order for Container Services to be used the following are necessary:

  • Nutanix cluster must be AOS 4.7 or later
  • A CentOS 7.0+ or a Rhel 7.2+ OS image with the iscsi-initiator-utils package installed must be downloaded and exist as an image in the AOS Image Service
  • The Nutanix Data Services IP must be configured
  • Docker Toolbox must be installed on the client machine used for configuration
  • Nutanix Docker Machine Driver must be in client's PATH
Docker Host Creation

Assuming all pre-requisites have been met the first step is to provision the Nutanix Docker Hosts using Docker Machine:

docker-machine -D create -d nutanix \
--nutanix-username <PRISM_USER> --nutanix-password <PRISM_PASSWORD> \
--nutanix-endpoint <CLUSTER_IP>:9440 --nutanix-vm-image <DOCKER_IMAGE_NAME> \
--nutanix-vm-network <NETWORK_NAME> \
--nutanix-vm-cores <NUM_CPU> --nutanix-vm-mem <MEM_MB> \

The following figure shows a high-level overview of the backend workflow:

Docker - Host Creation Workflow
Docker - Host Creation Workflow

The next step is to SSH into the newly provisioned Docker Host(s) via docker-machine ssh:

docker-machine ssh <DOCKER_HOST_NAME>

To install the Nutanix Docker Volume Plugin run:

docker plugin install ntnx/nutanix_volume_plugin PRISM_IP= DATASERVICES_IP= PRISM_PASSWORD= PRISM_USERNAME= DEFAULT_CONTAINER= --alias nutanix

After that runs you should now see the plugin enabled:

[root@DOCKER-NTNX-00 ~]# docker plugin ls ID Name Description Enabled 37fba568078d nutanix:latest Nutanix volume plugin for docker true

Docker Container Creation

Once the Nutanix Docker Host(s) have been deployed and the volume plugin has been enabled, you can provision containers with persistent storage.

A volume using the AOS Volumes can be created using the typical Docker volume command structure and specifying the Nutanix volume driver. Example usage below:

docker volume create \
<VOLUME_NAME> --driver nutanix
docker volume create PGDataVol --driver nutanix

To following command structure can be used to create a container using the created volume. Example usage below:

docker run -d --name <CONTAINER_NAME> \
-p <START_PORT:END_PORT> --volume-driver nutanix \

docker run -d --name postgresexample -p 5433:5433 --volume-driver nutanix -v PGDataVol:/var/lib/postgresql/data postgres:latest

The following figure shows a high-level overview of the backend workflow:

Docker - Container Creation Workflow
Docker - Container Creation Workflow

You now have a container running with persistent storage!

Backup and Disaster Recovery

Nutanix provides native backup and disaster recovery (DR) capabilities allowing users to backup, restore and DR VM(s) and objects running on DSF to both on-premise or cloud environments (Xi). As of AOS 5.11 Nutanix released a feature called Leap which abstracts a lot of these concepts. For more information on Leap, refer to the 'Leap' chapter in the 'Book of Prism'.

We will cover the following items in the following sections:

  • Implementation Constructs
  • Protecting Entities
  • Backup and Restore
  • Replication and DR

NOTE: Though Nutanix provides native options for backup and dr, traditional solutions (e.g. Commvault, Rubrik, etc.) can also be used, leveraging some of the native features the platform provides (VSS, snapshots, etc.).

Implementation Constructs

Within Nutanix Backup and DR, there are a few key constructs:

Protection Domain (PD)
  • Key Role: Macro group of VMs and/or files to protect
  • Description: A group of VMs and/or files to be replicated together on a desired schedule.  A PD can protect a full container or you can select individual VMs and/or files
Pro tip

Create multiple PDs for various services tiers driven by a desired RPO/RTO.  For file distribution (e.g. golden images, ISOs, etc.) you can create a PD with the files to replication.

Consistency Group (CG)
  • Key Role: Subset of VMs/files in PD to be crash-consistent
  • Description: VMs and/or files which are part of a Protection Domain which need to be snapshotted in a crash-consistent manner.  This ensures that when VMs/files are recovered, they come up in a consistent state.  A protection domain can have multiple consistency groups.
Pro tip

Group dependent application or service VMs in a consistency group to ensure they are recovered in a consistent state (e.g. App and DB)

Snapshot Schedule
  • Key Role: Snapshot and replication schedule
  • Description: Snapshot and replication schedule for VMs in a particular PD and CG
Pro tip

The snapshot schedule should be equal to your desired RPO

Retention Policy
  • Key Role: Number of local and remote snapshots to keep
  • Description: The retention policy defines the number of local and remote snapshots to retain.  NOTE: A remote site must be configured for a remote retention/replication policy to be configured.
Pro tip

The retention policy should equal the number of restore points required per VM/file

Remote Site
  • Key Role: A remote Nutanix cluster
  • Description: A remote Nutanix cluster which can be leveraged as a target for backup or DR purposes.
Pro tip

Ensure the target site has ample capacity (compute/storage) to handle a full site failure.  In certain cases replication/DR between racks within a single site can also make sense.

The following figure shows a logical representation of the relationship between a PD, CG, and VM/Files for a single site:

DR Construct Mapping
DR Construct Mapping
Policy Based DR & Run Books

Policy based DR and run books extends upon the capabilities defined in VM based DR (PDs, CGs, etc.) and abstracts things into a policy driven model. This simplifies configuration by focusing on the items of interest (e.g. RPO, retention, etc.) and assigning to categories instead of directly to VMs. This also allows for a "default policy" that can apply to all VMs.

NOTE: These policies are configured via Prism Central (PC).

Protecting Entities

You can protect Entities (VMs, VGs, Files), using the following workflow:

From the Data Protection page, select + Protection Domain -> Async DR:

DR - Async PD
DR - Async PD

Specify a PD name and click 'Create'

DR - Create PD
DR - Create PD

Select entities to protect:

DR - Async PD
DR - Async PD

Click 'Protect Selected Entities'

DR - Protect Entities
DR - Protect Entities

The protect entities will now be displayed under 'Protected Entities'

DR - Protected Entities
DR - Protected Entities

Click 'Next', then click 'Next Schedule' to create a snapshot and replication schedule

Enter the desired snapshot frequency, retention and any remote sites for replication

DR - Create Schedule
DR - Create Schedule

Click 'Create Schedule' to complete the schedule completion.

Multiple Schedules

It is possible to create multiple snapshot / replication schedules. For example, if you want to have a local backup schedule occurring hourly and another schedule which replicated to a remote site daily.

It’s important to mention that a full container can be protected for simplicity. However, the platform provides the ability to protect down to the granularity of a single VM and/or file level.

Backup and Restore

Nutanix backup capabilities leverage the native DSF snapshot capabilities and are invoked by Cerebro and performed by Stargate. These snapshot capabilities are zero copy to ensure efficient storage utilization and low overhead. You can read more on Nutanix snapshots in the 'Snapshots and Clones' section.

Typical backup and restore operations include:

  • Snapshot: Create a restore point and replicate (if necessary)
  • Restore: Restore VM(s) / File(s) from a previous snapshot (replaces original objects)
  • Clone: Similar to restore but does not replace original objects (creates new objects as desired snapshot)

From the Data Protection Page, you can see the protection domains (PD) previously created in the 'Protecting Entities' section.

DR - View PDs
DR - View PDs

Once you're selected a target PD you can see the various options:

DR - PD Actions
DR - PD Actions

If you click 'Take Snapshot' you can take an ad-hoc snapshot of the selected PD and replicate to a remote site if necessary:

DR - Take Snapshot
DR - Take Snapshot

You can also 'Migrate' the PD which will fail over the entities to a remote site:

DR - Migrate
DR - Migrate

In the event of a migrate (controlled failover), the system will take a new snapshot, replicate then promote the other site with the newly created snap.

Pro tip

With AOS 5.0 and above you can now leverage a single node replication target data protection.

You can also view the PD snapshot(s) in the table below:

DR - Local Snapshots
DR - Local Snapshots

From here you can restore or clone a PD snapshot:

DR - Restore Snapshot
DR - Restore Snapshot

If you choose to 'Create new entities' that will be like cloning the snapshot of the PD to new entities with the desired prefixes. Otherwise 'Overwrite existing entities' will replace the current entities with those at the time of the snapshot.

Storage only backup target

For backup / archival only purposes, it is possible to configure a storage only Nutanix cluster as a remote site which will act as a backup target. This will allow data to be replicated to / from the storage only cluster.

App Consistent Snapshots

Nutanix provides native VmQueisced Snapshot Service (VSS) capabilities for queiscing OS and application operations which ensure an application consistent snapshot is achieved.

VmQueisced Snapshot Service (VSS)

VSS is typically a Windows specific term for Volume Shadow Copy Service. However, since this solution applies to both Windows and Linux we've modified the term to VmQueisced Snapshot Service.

Supported Configurations

The solution is applicable to both Windows and Linux guests, including versions below (list may be incomplete, refer to documentation for a fully supported list):

  • Hypervisors:
    • ESX
    • AHV
  • Windows
    • 2008R2, 2012, 2012R2
  • Linux
    • CentOS 6.5/7.0
    • RHEL 6.5/7.0
    • OEL 6.5/7.0
    • Ubuntu 14.04+
    • SLES11SP3+

In order for Nutanix VSS snapshots to be used the following are necessary:

  • Nutanix Platform
    • Cluster Virtual IP (VIP) must be configured
  • Guest OS / UVM
    • NGT must be installed
    • CVM VIP must be reachable on port 2074
  • Disaster Recovery Configuration
    • UVM must be in PD with 'Use application consistent snapshots' enabled

As of 4.6 this is achieved using the native Nutanix Hardware VSS provider which is installed as part of the Nutanix Guest Tools package. You can read more on the guest tools in the 'Nutanix Guest Tools' section.

The following image shows a high-level view of the VSS architecture:

Nutanix Hardware VSS Provider

You can perform an application consistent snapshot by following the normal data protection workflow and selecting 'Use application consistent snapshots' when protecting the VM.

Enabling/Disabling Nutanix VSS

When NGT is enabled for a UVM, the Nutanix VSS snapshot capability is enabled by default. However, you can turn off this capability with the following command:

ncli ngt disable-applications application-names=vss_snapshot vm_id=<VM_ID>

Windows VSS Architecture

The Nutanix VSS solution is integrated with the Windows VSS framework. The following shows a high-level view of the architecture:

Nutanix VSS - Windows Architecture
Nutanix VSS - Windows Architecture

Once NGT is installed you can see the NGT Agent and VSS Hardware Provider services:

VSS Hardware Provider
VSS Hardware Provider
Linux VSS Architecture

The Linux solution works similar to the Windows solution, however scripts are leveraged instead of the Microsoft VSS framework as it doesn't exist in Linux distros.

The Nutanix VSS solution is integrated with the Windows VSS framework. The following shows a high-level view of the architecture:

Nutanix VSS - Linux Architecture
Nutanix VSS - Linux Architecture

The pre-freeze and post-thaw scripts are located in the following directories:

  • Pre-freeze: /sbin/pre_freeze
  • Post-thaw: /sbin/post-thaw
Eliminating ESXi Stun

ESXi has native app consistent snapshot support using VMware guest tools. However, during this process, delta disks are created and ESXi "stuns" the VM in order to remap the virtual disks to the new delta files which will handle the new write IO. Stuns will also occur when a VMware snapshot is deleted.

During this stun process the VM its OS cannot execute any operations and is essentially in a "stuck" state (e.g. pings will fail, no IO). The duration of the stun will depend on the number of vmdks and speed of datastore metadata operations (e.g. create new delta disks, etc.)

By using Nutanix VSS we completely bypass the VMware snapshot / stun process and have little to no impact to performance or VM / OS availability.

Replication and Disaster Recovery (DR)

For a visual explanation, you can watch the following video: LINK

Nutanix provides native DR and replication capabilities, which build upon the same features explained in the Snapshots & Clones section.  Cerebro is the component responsible for managing the DR and replication in DSF.  Cerebro runs on every node and a Cerebro leader is elected (similar to NFS leader) and is responsible for managing replication tasks.  In the event the CVM acting as Cerebro leader fails, another is elected and assumes the role.  The Cerebro page can be found on <CVM IP>:2020. The DR function can be broken down into a few key focus areas:

  • Replication Topologies
  • Replication Lifecycle
  • Global Deduplication
Replication Topologies

Traditionally, there are a few key replication topologies: Site to site, hub and spoke, and full and/or partial mesh.  Contrary to traditional solutions which only allow for site to site or hub and spoke, Nutanix provides a fully mesh or flexible many-to-many model.

Example Replication Topologies
Example Replication Topologies

Essentially, this allows the admin to determine a replication capability that meets their company's needs.

Replication Lifecycle

Nutanix replication leverages the Cerebro service mentioned above.  The Cerebro service is broken into a “Cerebro Leader”, which is a dynamically elected CVM, and Cerebro Workers, which run on every CVM.  In the event where the CVM acting as the “Cerebro Leader” fails, a new “Leader” is elected.

The Cerebro Leader is responsible for managing task delegation to the local Cerebro Workers as well as coordinating with remote Cerebro Leader(s) when remote replication is occurring.

During a replication, the Cerebro Leader will figure out which data needs to be replicated, and delegate the replication tasks to the Cerebro Workers which will then tell Stargate which data to replicate and to where.

Replicated data is protected at multiple layers throughout the process. Extent reads on the source are checksummed to ensure consistency for source data (similar to how any DSF read occurs) and the new extent(s) are checksummed at the target (similar to any DSF write). TCP provides consistency on the network layer.

The following figure shows a representation of this architecture:

Replication Architecture
Replication Architecture

It is also possible to configure a remote site with a proxy which will be used as a bridgehead for all coordination and replication traffic coming from a cluster.

Pro tip

When using a remote site configured with a proxy, always utilize the cluster IP as that will always be hosted by the Prism Leader and available, even if CVM(s) go down.

The following figure shows a representation of the replication architecture using a proxy:

Replication Architecture - Proxy
Replication Architecture - Proxy

In certain scenarios, it is also possible to configure a remote site using a SSH tunnel where all traffic will flow between two CVMs.

This should only be used for non-production scenarios and the cluster IPs should be used to ensure availability.

The following figure shows a representation of the replication architecture using a SSH tunnel:

Replication Architecture - SSH Tunnel
Replication Architecture - SSH Tunnel
Global Deduplication

As explained in the Elastic Deduplication Engine section above, DSF has the ability to deduplicate data by just updating metadata pointers. The same concept is applied to the DR and replication feature.  Before sending data over the wire, DSF will query the remote site and check whether or not the fingerprint(s) already exist on the target (meaning the data already exists).  If so, no data will be shipped over the wire and only a metadata update will occur. For data which doesn’t exist on the target, the data will be compressed and sent to the target site.  At this point, the data existing on both sites is usable for deduplication.

The following figure shows an example three site deployment where each site contains one or more protection domains (PD):

Replication Deduplication
Replication Deduplication

Fingerprinting must be enabled on the source and target container / vstore for replication deduplication to occur.


Building upon the traditional asynchronous (async) replication capabilities mentioned previously; Nutanix has introduced support for near synchronous replication (NearSync).

NearSync provides the best of both worlds: zero impact to primary I/O latency (like async replication) in addition to a very low RPO (like sync replication (metro)). This allows users have a very low RPO without having the overhead of requiring synchronous replication for writes.

This capability uses a new snapshot technology called light-weight snapshot (LWS). Unlike the traditional vDisk based snapshots used by async, this leverages markers and is completely OpLog based (vs. vDisk snapshots which are done in the Extent Store).

Mesos is a new service added to manage the snapshot layer and abstract the complexities of the full/incremental snapshots. Cerebro continues to manage the high-level constructs and policies (e.g. consistency groups, etc.) whereas Mesos is responsible for interacting with Stargate and controlling the LWS lifecycle.

The following figure shows an example of the communication between the NearSync components:

NearSync Components
NearSync Component Interaction

When a user configures a snapshot frequency <= 15 minutes, NearSync is automatically leveraged. Upon this, an initial seed snapshot is taken then replicated to the remote site(s). Once this completes in < 60 minutes (can be the first or n later), another seed snapshot is immedatly taken and replicated in addition to LWS snapshot replication starting. Once the second seed snapshot finishes replication, all already replicated LWS snapshots become valid and the system is in stable NearSync.

The following figure shows an example timeline from enabling NearSync to execution:

NearSync Replication Lifecycle
NearSync Replication Lifecycle

During a steady run state vDisk snapshots are taken every hour. Rather than sending the snapshot over to the remote site in addition to the LWS, the remote site composes the vDisk snapshot based upon the prior vDisk snapshot and the LWS from that time.

In the event NearSync falls out of sync (e.g. network outage, WAN latency, etc.) causing the LWS replication to take > 60 minutes, the system will automatically switch back to vDisk based snapshots. When one of these completes in < 60 minutes, the system will take another snapshot immediately as well as start replicating LWS. Once the full snapshot completes, the LWS snapshots become valid and the system is in stable NearSync. This process is similar to the initial enabling of NearSync.

When a LWS based snap is restored (or cloned), the system will take a clone of the latest vDisk snapshot and apply the LWS incrementally until the desired LWS is reached.

The following figure shows an example of how a LWS based snapshot is restored:

vDisk Restore from LWS
vDisk Restore from LWS

Metro Availability

Nutanix provides native “stretch clustering” capabilities which allow for a compute and storage cluster to span multiple physical sites.  In these deployments, the compute cluster spans two locations and has access to a shared pool of storage.

This expands the VM HA domain from a single site to between two sites providing a near 0 RTO and a RPO of 0.

In this deployment, each site has its own Nutanix cluster, however the containers are “stretched” by synchronously replicating to the remote site before acknowledging writes.

The following figure shows a high-level design of what this architecture looks like:

Metro Availability - Normal State
Metro Availability - Normal State

In the event of a site failure, an HA event will occur where the VMs can be restarted on the other site. The failover process is typically a manual process. With the AOS 5.0 release a Metro Witness can be configured which can automate the failover. The witness can be downloaded via the Portal and is configured via Prism.

The following figure shows an example site failure:

Metro Availability - Site Failure
Metro Availability - Site Failure

In the event where there is a link failure between the two sites, each cluster will operate independently.  Once the link comes back up, the sites will be re-synchronized (deltas-only) and synchronous replication will start occurring.

The following figure shows an example link failure:

Metro Availability - Link Failure
Metro Availability - Link Failure

Cloud Connect

Building upon the native DR / replication capabilities of DSF, Cloud Connect extends this capability into cloud providers (currently Amazon Web Services, Microsoft Azure).  NOTE: This feature is currently limited to just backup / replication.

Very similar to creating a remote site to be used for native DR / replication, a “cloud remote site” is just created.  When a new cloud remote site is created, Nutanix will automatically spin up a single node Nutanix cluster in EC2 (currently m1.xlarge) or Azure Virtual Machines (currently D3) to be used as the endpoint.

The cloud instance is based upon the same Acropolis code-base leveraged for locally running clusters.  This means that all of the native replication capabilities (e.g., global deduplication, delta based replications, etc.) can be leveraged.

In the case where multiple Nutanix clusters are leveraging Cloud Connect, they can either A) share the same instance running in the region, or B) spin up a new instance.

Storage for cloud instances is done using a "cloud disk" which is a logical disk backed by S3 (AWS) or BlobStore (Azure). Data is stored as the usual egroups which are files on the object stores.

The following figure shows a logical representation of a “remote site” used for Cloud Connect:

Cloud Connect - Region
Cloud Connect Region

Since a cloud based remote site is similar to any other Nutanix remote site, a cluster can replicate to multiple regions if higher availability is required (e.g., data availability in the case of a full region outage):

Cloud Connect - Multi-region
Cloud Connect Multi-region

The same replication / retention policies are leveraged for data replicated using Cloud Connect.  As data / snapshots become stale, or expire, the cloud cluster will clean up data as necessary.

If replication isn’t frequently occurring (e.g., daily or weekly), the platform can be configured to power up the cloud instance(s) prior to a scheduled replication and down after a replication has completed.

Data that is replicated to any cloud region can also be pulled down and restored to any existing, or newly created Nutanix cluster which has the cloud remote site(s) configured:

Cloud Connect - Restore
Cloud Connect - Restore


Important Pages

These are advanced Nutanix pages besides the standard user interface that allow you to monitor detailed stats and metrics.  The URLs are formatted in the following way: http://<Nutanix CVM IP/DNS>:<Port/path (mentioned below)>  Example: http://MyCVM-A:2009  NOTE: if you’re on a different subnet IPtables will need to be disabled on the CVM to access the pages.

2009 Page

This is a Stargate page used to monitor the back end storage system and should only be used by advanced users.  I’ll have a post that explains the 2009 pages and things to look for.

2009/latency Page

This is a Stargate page used to monitor the back end latency.

2009/vdisk_stats Page

This is a Stargate page used to show various vDisk stats including histograms of I/O sizes, latency, write hits (e.g., OpLog, eStore), read hits (cache, SSD, HDD, etc.) and more.

2009/h/traces Page

This is the Stargate page used to monitor activity traces for operations.

2009/h/vars Page

This is the Stargate page used to monitor various counters.

2010 Page

This is the Curator page which is used for monitoring Curator runs.

2010/master/control Page

This is the Curator control page which is used to manually start Curator jobs

2011 Page

This is the Chronos page which monitors jobs and tasks scheduled by Curator.

2020 Page

 This is the Cerebro page which monitors the protection domains, replication status and DR.

2020/h/traces Page

This is the Cerebro page used to monitor activity traces for PD operations and replication.

2030 Page

This is the main Acropolis page and shows details about the environment hosts, any currently running tasks and networking details.

2030/sched Page

This is an Acropolis page used to show information about VM and resource scheduling used for placement decisions.  This page shows the available host resources and VMs running on each host.

2030/tasks Page

This is an Acropolis page used to show information about Acropolis tasks and their state.  You can click on the task UUID to get detailed JSON about the task.

2030/vms Page

This is an Acropolis page used to show information about Acropolis VMs and details about them.  You can click on the VM Name to connect to the console.

Cluster Commands

Check cluster status

Description: Check cluster status from the CLI

cluster status

Check local CVM service status

Description: Check a single CVM's service status from the CLI

genesis status

Check upgrade status


Perform manual / cli upgrade

download .tar.gz into ~/tmp

tar xzf .tar.gz

cd ~/tmp

./install/bin/cluster -i ./install upgrade

Node(s) upgrade
Hypervisor upgrade status

Description: Check hypervisor upgrade status from the CLI on any CVM


Detailed logs (on every CVM)


Restart cluster service from CLI

Description: Restart a single cluster service from the CLI

Stop service

cluster stop <Service Name>

Start stopped services

cluster start  #NOTE: This will start all stopped services

Start cluster service from CLI

Description: Start stopped cluster services from the CLI

Start stopped services

cluster start  #NOTE: This will start all stopped services


Start single service

Start single service: cluster start  <Service Name>

Restart local service from CLI

Description: Restart a single cluster service from the CLI

Stop Service

genesis stop <Service Name>

Start Service

cluster start

Start local service from CLI

Description: Start stopped cluster services from the CLI

cluster start #NOTE: This will start all stopped services

Cluster add node from cmdline

Description: Perform cluster add-node from CLI

ncli cluster discover-nodes | egrep "Uuid" | awk '{print $4}' | xargs -I UUID ncli cluster add-node node-uuid=UUID

Find cluster id

Description: Find the cluster ID for the current cluster

zeus_config_printer | grep cluster_id

Open port

Description: Enable port through IPtables

sudo vi /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport <PORT> -j ACCEPT
sudo service iptables restart

Check for Shadow Clones

Description: Displays the shadow clones in the following format:  name#id@svm_id

vdisk_config_printer | grep '#'

Reset Latency Page Stats

Description: Reset the Latency Page (<CVM IP>:2009/latency) counters

allssh "wget"

Find vDisk information

Description: Find vDisk information and details including name, id, size, iqn and others


Find Number of vDisks

Description: Find the current number of vDisks (files) on DSF

vdisk_config_printer | grep vdisk_id | wc -l

Get detailed vDisk information

Description: Displays a provided vDisks egroup IDs, size, transformation and savings, garbage and replica placement

vdisk_usage_printer -vdisk_id=<VDISK_ID>

Start Curator scan from CLI

Description: Starts a Curator scan from the CLI

# Full Scan
allssh "wget -O - "http://localhost:2010/master/api/client/StartCuratorTasks?task_type=2";"

# Partial Scan
allssh "wget -O - "http://localhost:2010/master/api/client/StartCuratorTasks?task_type=3";"

# Refresh Usage
allssh "wget -O - "http://localhost:2010/master/api/client/RefreshStats";"

Check under replicated data via CLI

Description: Check for under replicated data using curator_cli

curator_cli get_under_replication_info summary=true

Compact ring

Description: Compact the metadata ring

allssh "nodetool -h localhost compact"

Find NOS version

Description: Find the NOS  version (NOTE: can also be done using NCLI)

allssh "cat /etc/nutanix/release_version"

Find CVM version

Description: Find the CVM image version

allssh "cat /etc/nutanix/svm-version"

Manually fingerprint vDisk(s)

Description: Create fingerprints for a particular vDisk (For dedupe)  NOTE: dedupe must be enabled on the container

vdisk_manipulator –vdisk_id=<vDisk ID> --operation=add_fingerprints

Manually fingerprint all vDisk(s)

Description: Create fingerprints for all vDisk(s) (For dedupe)  NOTE: dedupe must be enabled on the container

for vdisk in `vdisk_config_printer | grep vdisk_id | awk {'print $2'}`; do vdisk_manipulator -vdisk_id=$vdisk --operation=add_fingerprints; done

Echo Factory_Config.json for all cluster nodes

Description: Echos the factory_config.jscon for all nodes in the cluster

allssh "cat /etc/nutanix/factory_config.json"

Upgrade a single Nutanix node’s NOS version

Description: Upgrade a single node's NOS version to match that of the cluster

~/cluster/bin/cluster -u <NEW_NODE_IP> upgrade_node

 List files (vDisk) on DSF

Description: List files and associated information for vDisks stored on DSF


Get help text

Nfs_ls --help

Install Nutanix Cluster Check (NCC)

Description: Installs the Nutanix Cluster Check (NCC) health script to test for potential issues and cluster health

Download NCC from the Nutanix Support Portal (

SCP .tar.gz to the /home/nutanix directory

Untar NCC .tar.gz

tar xzmf <ncc .tar.gz file name> --recursive-unlink

Run install script

./ncc/bin/ -f <ncc .tar.gz file name>

Create links

source ~/ncc/ncc_completion.bash
echo "source ~/ncc/ncc_completion.bash" >> ~/.bashrc

Run Nutanix Cluster Check (NCC)

Description: Runs the Nutanix Cluster Check (NCC) health script to test for potential issues and cluster health.  This is a great first step when troubleshooting any cluster issues.

Make sure NCC is installed (steps above)

Run NCC health checks

ncc health_checks run_all

List tasks using progress monitor cli

progress_monitor_cli -fetchall

Remove task using progress monitor cli

progress_monitor_cli --entity_id=<ENTITY_ID> --operation=<OPERATION> --entity_type=<ENTITY_TYPE> --delete
# NOTE: operation and entity_type should be all lowercase with k removed from the begining

Metrics and Thresholds

The following section will cover specific metrics and thresholds on the Nutanix back end.  More updates to these coming shortly!


More coming soon!

Troubleshooting & Advanced Administration

Find Acropolis logs

Description: Find Acropolis logs for the cluster

allssh "cat ~/data/logs/Acropolis.log"

Find cluster error logs

Description: Find ERROR logs for the cluster

allssh "cat ~/data/logs/<COMPONENT NAME or *>.ERROR"

Example for Stargate

allssh "cat ~/data/logs/Stargate.ERROR"

Find cluster fatal logs

Description: Find FATAL logs for the cluster

allssh "cat ~/data/logs/<COMPONENT NAME or *>.FATAL"

Example for Stargate

allssh "cat ~/data/logs/Stargate.FATAL"

Using the 2009 Page (Stargate)

In most cases Prism should be able to give you all of the information and data points you require.  However, in certain scenarios, or if you want some more detailed data you can leverage the Stargate aka 2009 page.  The 2009 page can be viewed by navigating to <CVM IP>:2009.

Accessing back-end pages

If you're on a different network segment (L2 subnet) you'll need to add a rule in IP tables to access any of the back-end pages.

At the top of the page is the overview details which show various details about the cluster:

2009 Page - Stargate Overview
2009 Page - Stargate Overview

In this section there are two key areas I look out for, the first being the I/O queues that shows the number of admitted / outstanding operations.

The figure shows the queues portion of the overview section:

2009 Page - Stargate Overview - Queues
2009 Page - Stargate Overview - Queues

The second portion is the unified cache details that shows information on cache sizes and hit rates.

The figure shows the unified cache portion of the overview section:

2009 Page - Stargate Overview - Unified Cache
2009 Page - Stargate Overview - Unified Cache
Pro tip

In ideal cases the hit rates should be above 80-90%+ if the workload is read heavy for the best possible read performance.

NOTE: these values are per Stargate / CVM

The next section is the 'Cluster State' that shows details on the various Stargates in the cluster and their disk usages.

The figure shows the Stargates and disk utilization (available/total):

2009 Page - Cluster State - Disk Usage
2009 Page - Cluster State - Disk Usage

The next section is the 'NFS Worker' section which will show various details and stats per vDisk.

The figure shows the vDisks and various I/O details:

2009 Page - NFS Worker - vDisk Stats
2009 Page - NFS Worker - vDisk Stats
Pro tip

When looking at any potential performance issues I always look at the following:

  1. Avg. latency
  2. Avg. op size
  3. Avg. outstanding

For more specific details the vdisk_stats page holds a plethora of information.

Using the 2009/vdisk_stats Page

The 2009 vdisk_stats page is a detailed page which provides even further data points per vDisk.  This page includes details and a histogram of items like randomness, latency histograms, I/O sizes and working set details.

You can navigate to the vdisk_stats page by clicking on the 'vDisk Id' in the left hand column.

The figure shows the section and hyperlinked vDisk Id:

2009 Page - Hosted vDisks
2009 Page - Hosted vDisks

This will bring you to the vdisk_stats page which will give you the detailed vDisk stats.  NOTE: These values are real-time and can be updated by refreshing the page.

The first key area is the 'Ops and Randomness' section which will show a breakdown of whether the I/O patterns are random or sequential in nature.

The figure shows the 'Ops and Randomness' section:

2009 Page - vDisk Stats - Ops and Randomness
2009 Page - vDisk Stats - Ops and Randomness

The next area shows a histogram of the frontend read and write I/O latency (aka the latency the VM / OS sees).

The figure shows the 'Frontend Read Latency' histogram:

2009 Page - vDisk Stats - Frontend Read Latency
2009 Page - vDisk Stats - Frontend Read Latency

The figure shows the 'Frontend Write Latency' histogram:

2009 Page - vDisk Stats - Frontend Write Latency
2009 Page - vDisk Stats - Frontend Write Latency

The next key area is the I/O size distribution that shows a histogram of the read and write I/O sizes.

The figure shows the 'Read Size Distribution' histogram:

2009 Page - vDisk Stats - Read I/O Size
2009 Page - vDisk Stats - Read I/O Size

The figure shows the 'Write Size Distribution' histogram:

2009 Page - vDisk Stats - Write I/O Size
2009 Page - vDisk Stats - Write I/O Size

The next key area is the 'Working Set Size' section which provides insight on working set sizes for the last 2 minutes and 1 hour.  This is broken down for both read and write I/O.

The figure shows the 'Working Set Sizes' table:

2009 Page - vDisk Stats - Working Set
2009 Page - vDisk Stats - Working Set

The 'Read Source' provides details on which tier or location the read I/Os are being served from.

The figure shows the 'Read Source' details:

2009 Page - vDisk Stats - Read Source
2009 Page - vDisk Stats - Read Source
Pro tip

If you're seeing high read latency take a look at the read source for the vDisk and take a look where the I/Os are being served from.  In most cases high latency could be caused by reads coming from HDD (Estore HDD).

The 'Write Destination' section will show where the new write I/O are coming in to.

The figure shows the 'Write Destination' table:

2009 Page - vDisk Stats - Write Destination
2009 Page - vDisk Stats - Write Destination
Pro tip

Random I/Os will be written to the Oplog, sequential I/Os will bypass the Oplog and be directly written to the Extent Store (Estore).

Another interesting data point is what data is being up-migrated from HDD to SSD via ILM.  The 'Extent Group Up-Migration' table shows data that has been up-migrated in the last 300, 3,600 and 86,400 seconds.

The figure shows the 'Extent Group Up-Migration' table:

2009 Page - vDisk Stats - Extent Group Up-Migration
2009 Page - vDisk Stats - Extent Group Up-Migration

Using the 2010 Page (Curator)

The 2010 page is a detailed page for monitoring the Curator MapReduce framework.  This page provides details on jobs, scans, and associated tasks. 

You can navigate to the Curator page by navigating to http://<CVM IP>:2010.  NOTE: if you're not on the Curator Leader click on the IP hyperlink after 'Curator Leader: '.  

The top of the page will show various details about the Curator Leader including uptime, build version, etc.

The next section is the 'Curator Nodes' table that shows various details about the nodes in the cluster, the roles, and health status.  These will be the nodes Curator leverages for the distributed processing and delegation of tasks.

The figure shows the 'Curator Nodes' table:

2010 Page - Curator Nodes
2010 Page - Curator Nodes

The next section is the 'Curator Jobs' table that shows the completed or currently running jobs.  

There are two main types of jobs which include a partial scan which is eligible to run every 60 minutes and a full scan which is eligible to run every 6 hours.  NOTE: the timing will be variable based upon utilization and other activities.

These scans will run on their periodic schedules however can also be triggered by certain cluster events.

Here are some of the reasons for a jobs execution:

  • Periodic (normal state)
  • Disk / Node / Block failure
  • ILM Imbalance
  • Disk / Tier Imbalance

The figure shows the 'Curator Jobs' table:

2010 Page - Curator Jobs
2010 Page - Curator Jobs

The table shows some of the high-level activities performed by each job:

Activity Full Scan Partial Scan
Disk Balancing X X
Compression X X
Deduplication X  
Erasure Coding X  
Garbage Cleanup X  

Clicking on the 'Execution id' will bring you to the job details page which displays various job stats as well as generated tasks.

The table at the top of the page will show various details on the job including the type, reason, tasks and duration.

The next section is the 'Background Task Stats' table which displays various details on the type of tasks, quantity generated and priority.

The figure shows the job details table:

2010 Page - Curator Job - Details
2010 Page - Curator Job - Details

The figure shows the 'Background Task Stats' table:

2010 Page - Curator Job - Tasks
2010 Page - Curator Job - Tasks

The next section is the 'MapReduce Jobs' table that shows the actual MapReduce jobs started by each Curator job.  Partial scans will have a single MapReduce Job, full scans will have four MapReduce Jobs.

The figure shows the 'MapReduce Jobs' table:

2010 Page - MapReduce Jobs
2010 Page - MapReduce Jobs

Clicking on the 'Job id' will bring you to the MapReduce job details page which displays the tasks status, various counters and details about the MapReduce job.

The figure shows a sample of some of the job counters:

2010 Page - MapReduce Job - Counters
2010 Page - MapReduce Job - Counters

The next section on the main page is the 'Queued Curator Jobs' and 'Last Successful Curator Scans' section. These tables show when the periodic scans are eligible to run and the last successful scan's details.

The figure shows the 'Queued Curator Jobs' and 'Last Successful Curator Scans' section:

2010 Page - Queued and Successful Scans
2010 Page - Queued and Successful Scans

Advanced CLI Information

Prism should provide all that is necessary in terms of normal troubleshooting and performance monitoring. However, there may be cases where you want to get more detailed information which is exposed on some of the backend pages mentioned above, or the CLI.


The vdisk_config_printer command will display a list of vdisk information for all vdisk on the cluster.

I've highlighted some of the important fields below:

  • Vdisk ID
  • Vdisk name
  • Parent vdisk ID (if clone or snapshot)
  • Vdisk size (Bytes)
  • Container id
  • To remove bool (to be cleaned up by curator scan)
  • Mutability state (mutable if active r/w vdisk, immutable if snapshot)

The following shows example command output:

nutanix@NTNX-13SM35210012-A-CVM:~$ vdisk_config_printer | more ... vdisk_id: 1014400 vdisk_name: "NFS:1314152" parent_vdisk_id: 16445 vdisk_size: 40000000000 container_id: 988 to_remove: true creation_time_usecs: 1414104961926709 mutability_state: kImmutableSnapshot closest_named_ancestor: "NFS:852488" vdisk_creator_loc: 7 vdisk_creator_loc: 67426 vdisk_creator_loc: 4420541 nfs_file_name: "d12f5058-f4ef-4471-a196-c1ce8b722877" may_be_parent: true parent_nfs_file_name_hint: "d12f5058-f4ef-4471-a196-c1ce8b722877" last_modification_time_usecs: 1414241875647629 ...

vdisk_usage_printer -vdisk_id=<VDISK ID>

The vdisk_usage_printer is used to get detailed information for a vdisk, its extents and egroups.

I've highlighted some of the important fields below:

  • Egroup ID
  • Egroup extent count
  • Untransformed egroup size
  • Transformed egroup size
  • Transform ratio
  • Transformation type(s)
  • Egroup replica locations (disk/cvm/rack)

The following shows example command output:

nutanix@NTNX-13SM35210012-A-CVM:~$ vdisk_usage_printer -vdisk_id=99999 Egid # eids UT Size T Size ... T Type Replicas(disk/svm/rack) 1256878 64 1.03 MB 1.03 MB ... D,[73 /14/60][184108644/184108632/60] 1256881 64 1.03 MB 1.03 MB ... D,[73 /14/60][152/7/25] 1256883 64 1.00 MB 1.00 MB ... D,[73 /14/60][184108642/184108632/60] 1055651 4 4.00 MB 4.00 MB ... none[76 /14/60][184108643/184108632/60] 1056604 4 4.00 MB 4.00 MB ... none[74 /14/60][184108642/184108632/60] 1056605 4 4.00 MB 4.00 MB ... none[73 /14/60][152/7/25] ...

NOTE: Notice the egroup size for deduped vs. non-deduped egroups (1 vs. 4MB). As mentioned in the 'Data Structure' section, for deduped data, a 1MB egroup size is preferred to negate any potential fragmention cause by de-duplicating the data.

curator_cli display_data_reduction_report

The curator_cli display_data_reduction_report is used to get detailed information on the storage savings per container by transform (e.g. clone, snap, dedup, compression, erasure coding, etc.)

I've highlighted some of the important fields below:

  • Container ID
  • Technique (transform applied)
  • Pre reduction Size
  • Post reduction size
  • Saved space
  • Savings ratio

The following shows example command output:

CVM:~$ curator_cli display_data_reduction_report Using curator leader: Using execution id 68188 of the last successful full scan +---------------------------------------------------------------------------+ | Container| Technique | Pre | Post | Saved | Ratio | | Id | | Reduction | Reduction | | | +---------------------------------------------------------------------------+ | 988 | Clone | 4.88 TB | 2.86 TB | 2.02 TB | 1.70753 | | 988 | Snapshot | 2.86 TB | 2.22 TB | 656.92 GB | 1.28931 | | 988 | Dedup | 2.22 TB | 1.21 TB | 1.00 TB | 1.82518 | | 988 | Compression | 1.23 TB | 1.23 TB | 0.00 KB | 1 | | 988 | Erasure Coding | 1.23 TB | 1.23 TB | 0.00 KB | 1 | | 26768753 | Clone | 764.26 GB | 626.25 GB | 138.01 GB | 1.22038 | | 26768753 | Snapshot | 380.87 GB | 380.87 GB | 0.00 KB | 1 | | 84040 | Snappy | 810.35 GB | 102.38 GB | 707.97 GB | 7.91496 | | 6853230 | Snappy | 3.15 TB | 1.09 TB | 2.06 TB | 2.88713 | | 12199346 | Snappy | 872.42 GB | 109.89 GB | 762.53 GB | 7.93892 | | 12736558 | Snappy | 9.00 TB | 1.13 TB | 7.87 TB | 7.94087 | | 15430780 | Snappy | 1.23 TB | 89.37 GB | 1.14 TB | 14.1043 | | 26768751 | Snappy | 339.00 MB | 45.02 MB | 293.98 MB | 7.53072 | | 27352219 | Snappy | 1013.8 MB | 90.32 MB | 923.55 MB | 11.2253 | +---------------------------------------------------------------------------+

curator_cli get_vdisk_usage lookup_vdisk_ids=<COMMA SEPARATED VDISK ID(s)>

The curator_cli display_data_reduction_report is used to get detailed information on the storage savings per container per transform (e.g. clone, snap, dedup, compression, erasure coding, etc.)

I've highlighted some of the important fields below:

  • Vdisk ID
  • Exclusive usage (Data referred to by only this vdisk)
  • Logical uninherited (Data written to vdisk, may be inherited by a child in the event of clone)
  • Logical dedup (Amount of vdisk data that has been deduplicated)
  • Logical snapshot (Data not shared across vdisk chains)
  • Logical clone (Data shared across vdisk chains)

The following shows example command output:

Using curator leader: VDisk usage stats: +------------------------------------------------------------------------+ | VDisk Id | Exclusive | Logical | Logical | Logical | Logical | | | usage | Uninherited | Dedup | Snapshot | Clone | +------------------------------------------------------------------------+ | 254244142 | 596.06 MB | 529.75 MB | 6.70 GB | 11.55 MB | 214 MB | | 15995052 | 599.05 MB | 90.70 MB | 7.14 GB | 0.00 KB | 4.81 MB | | 203739387 | 31.97 GB | 31.86 GB | 24.3 MB | 0.00 KB | 4.72 GB | | 22841153 | 147.51 GB | 147.18 GB | 0.00 KB | 0.00 KB | 0.00 KB | ...

curator_cli get_egroup_access_info

The curator_cli get_egroup_access_info is used to get detailed information on the number of egroups in each bucket based upon last access (read) / modify ([over]write). This information can be used to estimate the number of egroups which might be eligible candidates to leverage erasure coding.

I've highlighted some of the important fields below:

  • Container ID
  • Access \ Modify (secs)

The following shows example command output:

Using curator leader: Container Id: 988 +----------------------------------------------------------------------------.. | Access \ Modify (secs) | [0,300) | [300,3600) | [3600,86400) | [86400,60480.. +----------------------------------------------------------------------------.. | [0,300) | 345 | 1 | 0 | 0 .. | [300,3600) | 164 | 817 | 0 | 0 .. | [3600,86400) | 4 | 7 | 3479 | 7 .. | [86400,604800) | 0 | 0 | 81 | 7063 .. | [604800,2592000) | 0 | 0 | 15 | 22 .. | [2592000,15552000) | 1 | 0 | 0 | 10 .. | [15552000,inf) | 0 | 0 | 0 | 1 .. +----------------------------------------------------------------------------.. ...

Book of AHV


Node Architecture

In AHV deployments, the Controller VM (CVM) runs as a VM and disks are presented using PCI passthrough.  This allows the full PCI controller (and attached devices) to be passed through directly to the CVM and bypass the hypervisor.  AHV is based upon CentOS KVM. Full hardware virtualization is used for guest VMs (HVM).

AHV Node
AHV Node

The AHV is built upon the CentOS KVM foundation and extends its base functionality to include features like HA, live migration, etc. 

AHV is validated as part of the Microsoft Server Virtualization Validation Program and is validated to run Microsoft OS and applications.

KVM Architecture

Within KVM there are a few main components:

  • KVM-kmod
    • KVM kernel module
  • Libvirtd
    • An API, daemon and management tool for managing KVM and QEMU.  Communication between AOS and KVM / QEMU occurs through libvirtd.
  • Qemu-kvm
    • A machine emulator and virtualizer that runs in userspace for every Virtual Machine (domain).  In AHV it is used for hardware-assisted virtualization and VMs run as HVMs.

The following figure shows the relationship between the various components:

KVM Component Relationship
KVM Component Relationship

Communication between AOS and KVM occurs via Libvirt. 

Processor generation compatibility

Similar to VMware's Enhanced vMotion Capability (EVC) which allows VMs to move between different processor generations; AHV will determine the lowest processor generation in the cluster and constrain all QEMU domains to that level. This allows mixing of processor generations within an AHV cluster and ensures the ability to live migrate between hosts.

Configuration Maximums and Scalability

The following configuration maximums and scalability limits are applicable:

  • Maximum cluster size: N/A – same as Nutanix cluster size
  • Maximum vCPUs per VM: Number of physical cores per host
  • Maximum memory per VM: Min of 4TB or available physical node memory
  • Maximum virtual disk size: 9EB* (Exabyte)
  • Maximum VMs per host: N/A – Limited by memory
  • Maximum VMs per cluster: N/A – Limited by memory

*AHV does not have a traditional storage stack like ESXi / Hyper-V; all disks are passed to the VM(s) as raw SCSI block devices. This means the maximum virtual disk size is limited by the maximum DSF vDisk size (9 Exabytes).


AHV leverages Open vSwitch (OVS) for all VM networking.  VM networking is configured through Prism / ACLI and each VM nic is connected into a tap interface.

The following figure shows a conceptual diagram of the OVS architecture:

Open vSwitch Network Overview
Open vSwitch Network Overview

In the prior image you see a few types of components:

Open vSwitch (OVS)

OVS is an open source software switch implemented in the Linux kernel and designed to work in a multiserver virtualization environment. By default, OVS behaves like a layer-2 learning switch that maintains a MAC address table. The hypervisor host and VMs connect to virtual ports on the switch.

OVS supports many popular switch features, including VLAN tagging, Link Aggregation Control Protocol (LACP), port mirroring, and quality of service (QoS), to name a few. Each AHV server maintains an OVS instance, and all OVS instances combine to form a single logical switch. Constructs called bridges manage the switch instances residing on the AHV hosts.


Bridges act as virtual switches to manage network traffic between physical and virtual network interfaces. The default AHV configuration includes an OVS bridge called br0 and a native Linux bridge called virbr0. The virbr0 Linux bridge carries management and storage communication between the CVM and AHV host. All other storage, host, and VM network traffic flows through the br0 OVS bridge. The AHV host, VMs, and physical interfaces use “ports” for connectivity to the bridge.


Ports are logical constructs created in a bridge that represent connectivity to the virtual switch. Nutanix uses several port types, including internal, tap, VXLAN, and bond:

  • An internal port—with the same name as the default bridge (br0)—provides access for the AHV host.
  • Tap ports act as bridge connections for virtual NICs presented to VMs.
  • VXLAN ports are used for the IP address management functionality provided by Acropolis.
  • Bonded ports provide NIC teaming for the physical interfaces of the AHV host.

Bonded ports aggregate the physical interfaces on the AHV host. By default, a bond named br0-up is created in bridge br0. After the node imaging process, all interfaces are placed within a single bond, which is a requirement for the foundation imaging process. Changes to the default bond, br0-up, often rename this to bond0. Nutanix recommends using the name br0-up to quickly identify the interface as the bridge br0 uplink.

OVS bonds allow for several load-balancing modes, including active-backup, balance-slb and balance-tcp. LACP can also be activated for a bond. The “bond_mode” setting is not specified during installation and therefore defaults to active-backup, which is the recommended configuration.

Uplink Load Balancing

Briefly mentioned in the prior section, it is possible to balance traffic across bond uplinks.

The following bond modes are available:

  • active-backup
    • Default configuration which transmits all traffic over a single active adapter. If the active adapter becomes unavailable, another adapter in the bond will become active. Limits throughput to a single nic's bandwidth. (Recommended)
  • balance-slb
    • Balances each VM's nic across adapters in the bond (e.g. VM A nic 1 - eth0 / nic 2 - eth1). Limits VM per-nic throughput to a single nic's bandwidth, however a VM with x nics can leverage x * adapter bandwidth (assuming x is the same for the number of VM nics and physical uplink adapters in the bond). NOTE: has caveats for multicast traffic
  • balance-tcp / LACP
    • Balances each VM nic's TCP session across adapters in the bond. Limits per-nic throughput to the maximum bond bandwidth (number of physical uplink adapters * speed). Requires link aggregation and used when LACP is required.

You can find additional information on bonds in the AHV Networking guide (LINK).

VM NIC Types

AHV supports the following VM network interface types:

  • Access (default)
  • Trunk (4.6 and above)

By default VM nics will be created as Access interfaces (similar to what you'd see with a VM nic on a port group), however it is possible to expose a trunked interface up to the VM's OS. Trunked NICs send the primary VLAN untagged, and all additional VLANs as tags to the same vNIC on the VM. This is useful to bring multiple networks to a VM without adding vNICs.

A trunked interface can be added with the following command:

vm.nic_create <VM_NAME> vlan_mode=kTrunked trunked_networks=<ALLOWED_VLANS> network=<NATIVE_VLAN>


vm.nic_create fooVM vlan_mode=kTrunked trunked_networks=10,20,30 network=vlan.10

Service Chaining

AHV Service chaining allows us to intercept all traffic and forward to a packet processor (NFV, appliance, virtual appliance, etc.) functions trasparently as part of the network path.

Common uses for service chaining:

  • Firewall (e.g. Palo Alto, ete.)
  • Load balancer (e.g. F5, Netscaler, etc.)
  • IDS/IPS/network monitors (e.g. packet capture)

Within service chaining there are two types of way:

Service chain - Packet Processors
Service chain - Packet Processors
  • Inline packet processor
    • Intercepts packets inline as they flow through OVS
    • Can modify and allow/deny packet
    • Common uses: firewalls and load balancers
  • Tap packet processor
    • Inspects packets as they flow, can only read as it's a tap into the packet flow
    • Common uses: IDS/IPS/network monitor

Any service chaining is done after the Flow - Microsegmentation rules are applied and before the packet leaves the local OVS. This occurs in the network function bridge (

Service Chain - Flow
Service Chain - Flow

NOTE: it is possible to string together multiple NFV / packet processors in a single chain.

How It Works

Storage I/O Path

AHV does not leverage a traditional storage stack like ESXi or Hyper-V. All disk(s) are passed to the VM(s) as raw SCSI block devices. This keeps the I/O path lightweight and optimized.


AOS abstracts kvm, virsh, qemu, libvirt, and iSCSI from the end-user and handles all backend configuration. This allows the user to focus higher up the stack on the VMs via Prism / ACLI. The following is for informational purposes only and it is not recommended to manually mess with virsh, libvirt etc.

Each AHV host runs an iSCSI redirector which regularly checks the health of Stargates throughout the cluster using NOP commands.

In the iscsi_redirector log (located in /var/log/ on the AHV host), you can see each Stargate's health:

2017-08-18 19:25:21,733 - INFO - Portal is up ... 2017-08-18 19:25:25,735 - INFO - Portal is up 2017-08-18 19:25:26,737 - INFO - Portal is up

NOTE: The local Stargate is shown via its internal address.

In the following you can see the iscsi_redirector is listening on

[root@NTNX-BEAST-1 ~]# netstat -tnlp | egrep tcp.*3261 Proto ... Local Address Foreign Address State PID/Program name ... tcp ...* LISTEN 8044/python ...

QEMU is configured with the iSCSI redirector as the iSCSI target portal.  Upon a login request, the redirector will perform an iSCSI login redirect to a healthy Stargate (preferably the local one).

iSCSI Multi-pathing - Normal State
iSCSI Multi-pathing - Normal State

Looking at the domain's XML we can see the configuration:

<devices> ... <disk type='network' device='lun'> <driver name='qemu' type='raw' cache='none' error_policy='report' io='native'/> <source protocol='iscsi' name=''> <host name='' port='3261'/> </source> <backingStore/> <target dev='sda' bus='scsi'/> <boot order='1'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> ... </devices>

The preferred controller type is virtio-scsi (default for SCSI devices). IDE devices, while possible, are not recommended for most scenarios. In order for virtio to be used with Windows the virtio drivers, Nutanix mobility drivers, or Nutanix guest tools must be installed. Modern Linux distros ship with virtio pre-installed.


... <controller type='scsi' index='0' model='virtio-scsi'> <driver max_sectors='2048'/> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> ...

In the event where the active Stargate goes down (thus failing to respond to the NOP OUT command), the iSCSI redirector will mark the local Stargate as unhealthy.  When QEMU retries the iSCSI login, the redirector will redirect the login to another healthy Stargate.

iSCSI Multi-pathing - Local CVM Down
iSCSI Multi-pathing - Local CVM Down

Once the local CVM's Stargate comes back up (and begins responding to the NOP OUT commands), the remote Stargate will quiesce then kill all connections to remote iSCSI sessions.  QEMU will then attempt an iSCSI login again and will be redirected to the local Stargate.

iSCSI Multi-pathing - Local CVM Back Up
iSCSI Multi-pathing - Local CVM Back Up

Traditional I/O Path

Like every hypervisor and OS there is a mix of user and kernel space components which interact to perform a common activity. Prior to reading the following, it is recommended to read the 'User vs. Kernel Space' section to learn more about how each interact with eachother.

When a VM performs an I/O it will perform the following (some steps have been excluded for clarity):

  1. VM's OS perform SCSI command(s) to virtual device(s)
  2. Virtio-scsi takes those requests and places them in the guest's memory
  3. Requests are handled by the QEMU main loop
  4. Libiscsi inspects each request and forwards
  5. Network layer forwards requests to local CVM (or externally if local is unavailable)
  6. Stargate handles request(s)

The following shows this sample flow:

AHV VirtIO Data Path - Classic
AHV VirtIO Data Path - Classic

Looking at the domain's XML we can see it is using the qemu-kvm emulator:

... <devices> <emulator>/usr/libexec/qemu-kvm</emulator> ...

Looking at an AHV host, you can see qemu-kvm has established sessions with a healthy Stargate using the local bridge and IPs. For external communication, the external host and Stargate IPs will be used. NOTE: There will be one session per disk device (look at PID 24845)

[root@NTNX-BEAST-1 log]# netstat -np | egrep tcp.*qemu Proto ... Local Address Foreign Address State PID/Program name tcp ... ESTABLISHED 25293/qemu-kvm tcp ... ESTABLISHED 23198/qemu-kvm tcp ... ESTABLISHED 24845/qemu-kvm tcp ... ESTABLISHED 24845/qemu-kvm ...

Now in this path there are a few inefficiencies as the main loop is single threaded and libiscsi inspects every SCSI command.

Frodo I/O Path (aka AHV Turbo Mode)

As storage technologies continue to evolve and become more efficient, so must we. Given the fact that we fully control AHV and the Nutanix stack this was an area of opportunity.

In short Frodo is a heavily optimized I/O path for AHV that allows for higher throughput, lower latency and less CPU overhead.

Pro tip

Frodo is enabled by default on VMs powered on after AOS 5.5.X.

When a VM performs an I/O it will perform the following (some steps have been excluded for clarity):

  1. VM's OS perform SCSI command(s) to virtual device(s)
  2. Virtio-scsi takes those requests and places them in the guest's memory
  3. Requests are handled by Frodo
  4. Custom libiscsi appends iscsi header and forwards
  5. Network layer forwards requests to local CVM (or externally if local is unavailable)
  6. Stargate handles request(s)

The following shows this sample flow:

AHV VirtIO Data Path - Frodo
AHV VirtIO Data Path - Frodo

The following path does looks similar to the traditional I/O except for a few key differences:

  • Qemu main loop is replaced by Frodo (vhost-user-scsi)
  • Frodo exposes multiple virtual queues (VQs) to the guest (one per vCPU)
  • Leverages multiple threads for multi-vCPU VMs
  • Libiscsi is replaced by our own much more lightweight version

To the guest it will notice that it now has multiple queues for the disk device(s), other than that it'll just see the performance improvements. In some cases we've seen a CPU overhead reduction of 25% to perform the I/O and performance increases of up to 3x compared to Qemu! Comparing to another hypervisor we've seen CPU overhead to perform I/Os drop by up to 3x.

Looking at an AHV host, you will see a frodo process for each VM (qemu-kvm process) running:

[root@drt-itppc03-1 ~]# ps aux | egrep frodo ... /usr/libexec/qemu-kvm ... -chardev socket,id=frodo0,fd=3 \ -device vhost-user-scsi-pci,chardev=frodo0,num_queues=16... ... /usr/libexec/frodo ... -t ...

Looking at the domain's XML we can see it is using frodo:

... <devices> <emulator>/usr/libexec/frodo</emulator> ...

Pro tip

To take advantage of Frodo's multiple threads / connections, you must have >= 2 vCPU for a VM when it is powered on.

It can be characterized by the following:

  • 1 vCPU UVM:
    • 1 Frodo thread / session per disk device
  • >= 2 vCPU UVM:
    • 2 Frodo threads / sessions per disk device

In the following, you can see Frodo has established sessions with a healthy Stargate using the local bridge and IPs. For external communication, the external host and Stargate IPs will be used.

[root@NTNX-BEAST-1 log]# netstat -np | egrep tcp.*frodo Proto ... Local Address Foreign Address State PID/Program name tcp ... ESTABLISHED 42957/frodo tcp ... ESTABLISHED 42957/frodo tcp ... ESTABLISHED 42957/frodo tcp ... ESTABLISHED 42957/frodo ...

IP Address Management

The Acropolis IP address management (IPAM) solution provides the ability to establish a DHCP scope and assign addresses to VMs.  This leverages VXLAN and OpenFlow rules to intercept the DHCP request and respond with a DHCP response.

Here we show an example DHCP request using the Nutanix IPAM solution where the Acropolis Leader is running locally:

IPAM - Local Acropolis Leader
IPAM - Local Acropolis Leader

If the Acropolis Leader is running remotely, the same VXLAN tunnel will be leveraged to handle the request over the network. 

IPAM - Remote Acropolis Leader
IPAM - Remote Acropolis Leader

Traditional DHCP / IPAM solutions can also be leveraged in an ‘unmanaged’ network scenario.

VM High Availability (HA)

AHV VM HA is a feature built to ensure VM availability in the event of a host or block outage. In the event of a host failure the VMs previously running on that host will be restarted on other healthy nodes throughout the cluster. The Acropolis Leader is responsible for restarting the VM(s) on the healthy host(s).

The Acropolis Leader tracks host health by monitoring its connections to the libvirt on all cluster hosts:

HA - Host Monitoring
HA - Host Monitoring

Once the libvirt connection goes down, the countdown to the HA restart is initiated. Should libvirt connection fail to be re-established within the timeout, Acropolis will restart VMs that were running on the disconnected host. When this occurs, VMs should be restarted within 120 seconds.

In the event the Acropolis Leader becomes partitioned, isolated or fails a new Acropolis Leader will be elected on the healthy portion of the cluster. If a cluster becomes partitioned (e.g X nodes can't talk to the other Y nodes) the side with quorum will remain up and VM(s) will be restarted on those hosts.

There are two main modes for VM HA:

  • Default
    • This mode requires no configuration and is included by default when installing an AHV-based Nutanix cluster. When an AHV host becomes unavailable, the VMs that were running on the failed AHV host restart on the remaining hosts, depending on the available resources. Not all of the failed VMs restart if the remaining hosts do not have sufficient resources.
  • Guarantee
    • This nondefault configuration reserves space throughout the AHV hosts in the cluster to guarantee that all failed VMs can restart on other hosts in the AHV cluster during a host failure. To enable Guarantee mode, select the Enable HA check box, as shown in the figure below. A message then appears displaying the amount of memory reserved and how many AHV host failures can be tolerated.

Resource Reservations

When using the Guarantee mode for VM HA, the system will reserve host resources for VMs. The amount of resources which are reserved is summarized by the following:

  • If all containers are RF2 (FT1)
    • One "host" worth of resources
  • If any containers are RF3 (FT2)
    • Two "hosts" worth of resources

When hosts have uneven memory capacities the system will use the largest host's memory capacity when determining how much to reserve per host.

Post 5.0 Resource Reservations

Prior to 5.0, we supported both host and segment based reserevations. With 5.0 and later we now only support a segment based reservation which is automatically implemented when the Guarantee HA mode is selected.

Reserve segments distributes the resource reservation across all hosts in a cluster. In this scenario, each host will share a portion of the reservation for HA. This ensures the overall cluster has enough failover capacity to restart VM(s) in the event of a host failure.

The figure shows an example scenario with reserved segments:

HA - Reserved Segment
HA - Reserved Segment

In the event of a host failure VM(s) will be restarted throughout the cluster on the remaining healthy hosts:

HA - Reserved Segment - Fail Over
HA - Reserved Segment - Fail Over
Reserved segment(s) calculation

The system will automatically calculate the total number of reserved segments and per host reservation.

Finding reservations reduces to a well known set of problems called Knapsack. The optimal solution is NP-hard (exponential), but heuristic solutions can come close to optimal for the common case. We implement one such algorithm called MTHM. Nutanix will continue improving its placement algorithms.


More coming soon!

Important Pages

More coming soon!

Command Reference

Enable 10GbE links only on OVS

Description: Enable 10g only on bond0 for local host

manage_ovs --interfaces 10g update_uplinks

Description: Show ovs uplinks for full cluster

allssh "manage_ovs --interfaces 10g update_uplinks"

Show OVS uplinks

Description: Show ovs uplinks for local host

manage_ovs show_uplinks

Description: Show ovs uplinks for full cluster

allssh "manage_ovs show_uplinks"

Show OVS interfaces

Description: Show ovs interfaces for local host

manage_ovs show_interfaces

Show interfaces for full cluster

allssh "manage_ovs show_interfaces"

Show OVS switch information

Description: Show switch information

ovs-vsctl show

List OVS bridges

Description: List bridges

ovs-vsctl list br

Show OVS bridge information

Description: Show OVS port information

ovs-vsctl list port br0
ovs-vsctl list port <bond>

Show OVS interface information

Description: Show interface information

ovs-vsctl list interface br0

Show ports / interfaces on bridge

Description: Show ports on a bridge

ovs-vsctl list-ports br0

Description: Show ifaces on a bridge

ovs-vsctl list-ifaces br0

Create OVS bridge

Description: Create bridge

ovs-vsctl add-br <bridge>

Add ports to bridge

Description: Add port to bridge

ovs-vsctl add-port <bridge> <port>

Description: Add bond port to bridge

ovs-vsctl add-bond <bridge> <port> <iface>

Show OVS bond details

Description: Show bond details

ovs-appctl bond/show <bond>


ovs-appctl bond/show bond0

Set bond mode and configure LACP on bond

Description: Enable LACP on ports

ovs-vsctl set port <bond> lacp=<active/passive>

Description: Enable on all hosts for bond0

for i in `hostips`;do echo $i; ssh $i source /etc/profile > /dev/null 2>&1; ovs-vsctl set port bond0 lacp=active;done

Show LACP details on bond

Description: Show LACP details

ovs-appctl lacp/show <bond>

Set bond mode

Description: Set bond mode on ports

ovs-vsctl set port <bond> bond_mode=<active-backup, balance-slb, balance-tcp>

Show OpenFlow information

Description: Show OVS openflow details

ovs-ofctl show br0

Description: Show OpenFlow rules

ovs-ofctl dump-flows br0

Get QEMU PIDs and top information

Description: Get QEMU PIDs

ps aux | grep qemu | awk '{print $2}'

Description: Get top metrics for specific PID

top -p <PID>

Get active Stargate for QEMU processes

Description: Get active Stargates for storage I/O for each QEMU processes

netstat –np | egrep tcp.*qemu

Metrics and Thresholds

More coming soon!

Troubleshooting & Advanced Administration

Check iSCSI Redirector Logs

Description: Check iSCSI Redirector Logs for all hosts

for i in `hostips`; do echo $i; ssh root@$i cat /var/log/iscsi_redirector;done

Example for single host

Ssh root@<HOST IP>
Cat /var/log/iscsi_redirector

Monitor CPU steal (stolen CPU)

Description: Monitor CPU steal time (stolen CPU)

Launch top and look for %st (bold below)

Cpu(s):  0.0%us, 0.0%sy,  0.0%ni, 96.4%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st

Monitor VM network resource stats

Description: Monitor VM resource stats

Launch virt-top


Go to networking page

2 – Networking

Book of vSphere


Node Architecture

In ESXi deployments, the Controller VM (CVM) runs as a VM and disks are presented using VMDirectPath I/O.  This allows the full PCI controller (and attached devices) to be passed through directly to the CVM and bypass the hypervisor.

ESXi Node Architecture
ESXi Node Architecture

Configuration Maximums and Scalability

The following configuration maximums and scalability limits are applicable:

  • Maximum cluster size: 64
  • Maximum vCPUs per VM: 128
  • Maximum memory per VM: 4TB
  • Maximum virtual disk size: 62TB
  • Maximum VMs per host: 1,024
  • Maximum VMs per cluster: 8,000 (2,048 per datastore if HA is enabled)

NOTE: As of vSphere 6.0

Pro tip

When doing benchmarking on ESXi hosts, always test with the ESXi host power policy set to 'High performance'. This will disable and P- and C- states and will make sure the test results aren't artificially limited.


Each ESXi host has a local vSwitch which is used for intra-host communication between the Nutanix CVM and host. For external communication and VMs a standard vSwitch (default) or dvSwitch is leveraged.

The local vSwitch (vSwitchNutanix) is for local communication between the Nutanix CVM and ESXi host. The host has a vmkernel interface on this vSwitch (vmk1 - and the CVM has an interface bound to a port group on this internal switch (svm-iscsi-pg - This is the primary storage communication path.

The external vSwitch can be a standard vSwitch or a dvSwitch. This will host the external interfaces for the ESXi host and CVM as well as the port groups leveraged by VMs on the host. The external vmkernel interface is leveraged for host management, vMotion, etc. The external CVM interface is used for communication to other Nutanix CVMs. As many port groups can be created as required assuming the VLANs are enabled on the trunk.

The following figure shows a conceptual diagram of the vSwitch architecture:

ESXi vSwitch Network Overview
ESXi vSwitch Network Overview
Uplink and Teaming policy

It is recommended to have dual ToR switches and uplinks across both switches for switch HA. By default the system will have uplink interfaces in active/passive mode. For upstream switch architectures that are capable of having active/active uplink interfaces (e.g. vPC, MLAG, etc.) that can be leveraged for additional network throughput.

How It Works

Array Offloads – VAAI

The Nutanix platform supports the VMware APIs for Array Integration (VAAI), which allows the hypervisor to offload certain tasks to the array.  This is much more efficient as the hypervisor doesn’t need to be the 'man in the middle'. Nutanix currently supports the VAAI primitives for NAS, including the ‘full file clone’, ‘fast file clone’, and ‘reserve space’ primitives.  Here’s a good article explaining the various primitives: 

For both the full and fast file clones, a DSF 'fast clone' is done, meaning a writable snapshot (using re-direct on write) for each clone that is created.  Each of these clones has its own block map, meaning that chain depth isn’t anything to worry about. The following will determine whether or not VAAI will be used for specific scenarios:

  • Clone VM with Snapshot –> VAAI will NOT be used
  • Clone VM without Snapshot which is Powered Off –> VAAI WILL be used
  • Clone VM to a different Datastore/Container –> VAAI will NOT be used
  • Clone VM which is Powered On  –> VAAI will NOT be used

These scenarios apply to VMware View:

  • View Full Clone (Template with Snapshot) –> VAAI will NOT be used
  • View Full Clone (Template w/o Snapshot) –> VAAI WILL be used
  • View Linked Clone (VCAI) –> VAAI WILL be used

You can validate VAAI operations are taking place by using the ‘NFS Adapter’ Activity Traces page.

CVM Autopathing aka

In this section, I’ll cover how CVM 'failures' are handled (I’ll cover how we handle component failures in future update).  A CVM 'failure' could include a user powering down the CVM, a CVM rolling upgrade, or any event which might bring down the CVM. DSF has a feature called autopathing where when a local CVM becomes unavailable, the I/Os are then transparently handled by other CVMs in the cluster. The hypervisor and CVM communicate using a private network on a dedicated vSwitch (more on this above).  This means that for all storage I/Os, these are happening to the internal IP addresses on the CVM (  The external IP address of the CVM is used for remote replication and for CVM communication.

The following figure shows an example of what this looks like:

ESXi Host Networking
ESXi Host Networking

In the event of a local CVM failure, the local addresses previously hosted by the local CVM are unavailable.  DSF will automatically detect this outage and will redirect these I/Os to another CVM in the cluster over 10GbE.  The re-routing is done transparently to the hypervisor and VMs running on the host.  This means that even if a CVM is powered down, the VMs will still continue to be able to perform I/Os to DSF. Once the local CVM is back up and available, traffic will then seamlessly be transferred back and served by the local CVM.

The following figure shows a graphical representation of how this looks for a failed CVM:

ESXi Host Networking - Local CVM Down
ESXi Host Networking - Local CVM Down


Important Pages

More coming soon!

Command Reference

ESXi cluster upgrade

Description: Perform an automated upgrade of ESXi hosts using the CLI and custom offline bundle
# Upload upgrade offline bundle to a Nutanix CVM
# Log in to Nutanix CVM
# Perform upgrade

cluster --md5sum=<bundle_checksum> --bundle=</path/to/offline_bundle> host_upgrade

# Example

cluster --md5sum=bff0b5558ad226ad395f6a4dc2b28597 --bundle=/tmp/ host_upgrade

Restart ESXi host services

Description: Restart each ESXi hosts services in a incremental manner

for i in `hostips`;do ssh root@$i " restart";done

Display ESXi host nics in ‘Up’ state

Description: Display the ESXi host's nics which are in a 'Up' state

for i in `hostips`;do echo $i && ssh root@$i esxcfg-nics -l | grep Up;done

Display ESXi host 10GbE nics and status

Description: Display the ESXi host's 10GbE nics and status

for i in `hostips`;do echo $i && ssh root@$i esxcfg-nics -l | grep ixgbe;done

Display ESXi host active adapters

Description: Display the ESXi host's active, standby and unused adapters

for i in `hostips`;do echo $i &&  ssh root@$i "esxcli network vswitch standard policy failover get --vswitch-name vSwitch0";done

Display ESXi host routing tables

Description: Display the ESXi host's routing tables

for i in `hostips`;do ssh root@$i 'esxcfg-route -l';done

Check if VAAI is enabled on datastore

Description: Check whether or not VAAI is enabled/supported for a datastore

vmkfstools -Ph /vmfs/volumes/<Datastore Name>

Set VIB acceptance level to community supported

Description: Set the vib acceptance level to CommunitySupported allowing for 3rd party vibs to be installed

esxcli software acceptance set --level CommunitySupported

Install VIB

Description: Install a vib without checking the signature

esxcli software vib install --viburl=/<VIB directory>/<VIB name> --no-sig-check

# OR

esxcli software vib install --depoturl=/<VIB directory>/<VIB name> --no-sig-check

Check ESXi ramdisk space

Description: Check free space of ESXi ramdisk

for i in `hostips`;do echo $i; ssh root@$i 'vdf -h';done

Clear pynfs logs

Description: Clears the pynfs logs on each ESXi host

for i in `hostips`;do echo $i; ssh root@$i '> /pynfs/pynfs.log';done

Metrics and Thresholds

More coming soon!

Troubleshooting & Advanced Administration

More coming soon!

Book of Hyper-V


When a Nutanix Hyper-V cluster is created we automatically join the Hyper-V hosts to the specified Windows Active Directory domain. These hosts are then put into a failover cluster for VM HA. When this is complete there will be AD objects for each individual Hyper-V host and the failover cluster.

Node Architecture

In Hyper-V deployments, the Controller VM (CVM) runs as a VM and disks are presented using disk passthrough.

Hyper-V Node Architecture
Hyper-V Node Architecture

Configuration Maximums and Scalability

The following configuration maximums and scalability limits are applicable:

  • Maximum cluster size: 64
  • Maximum vCPUs per VM: 64
  • Maximum memory per VM: 1TB
  • Maximum virtual disk size: 64TB
  • Maximum VMs per host: 1,024
  • Maximum VMs per cluster: 8,000

NOTE: As of Hyper-V 2012 R2


Each Hyper-V host has a internal only virtual switch which is used for intra-host communication between the Nutanix CVM and host. For external communication and VMs a external virtual switch (default) or logical switch is leveraged.

The internal switch (InternalSwitch) is for local communication between the Nutanix CVM and Hyper-V host. The host has a virtual ethernet interface (vEth) on this internal switch ( and the CVM has a vEth on this internal switch ( This is the primary storage communication path.

The external vSwitch can be a standard virtual switch or a logical switch. This will host the external interfaces for the Hyper-V host and CVM as well as the logical and VM networks leveraged by VMs on the host. The external vEth interface is leveraged for host management, live migration, etc. The external CVM interface is used for communication to other Nutanix CVMs. As many logical and VM networks can be created as required assuming the VLANs are enabled on the trunk.

The following figure shows a conceptual diagram of the virtual switch architecture:

Hyper-V Virtual Switch Network Overview
Hyper-V Virtual Switch Network Overview
Uplink and Teaming policy

It is recommended to have dual ToR switches and uplinks across both switches for switch HA. By default the system will have the LBFO team in switch independent mode which doesn't require any special configuration.

How It Works

Array Offloads – ODX

The Nutanix platform supports the Microsoft Offloaded Data Transfers (ODX), which allow the hypervisor to offload certain tasks to the array.  This is much more efficient as the hypervisor doesn’t need to be the 'man in the middle'. Nutanix currently supports the ODX primitives for SMB, which include full copy and zeroing operations.  However, contrary to VAAI which has a 'fast file' clone operation (using writable snapshots), the ODX primitives do not have an equivalent and perform a full copy.  Given this, it is more efficient to rely on the native DSF clones which can currently be invoked via nCLI, REST, or PowerShell CMDlets. Currently ODX IS invoked for the following operations:

  • In VM or VM to VM file copy on DSF SMB share
  • SMB share file copy

Deploy the template from the SCVMM Library (DSF SMB share) – NOTE: Shares must be added to the SCVMM cluster using short names (e.g., not FQDN).  An easy way to force this is to add an entry into the hosts file for the cluster (e.g.     nutanix-130).

ODX is NOT invoked for the following operations:

  • Clone VM through SCVMM
  • Deploy template from SCVMM Library (non-DSF SMB Share)
  • XenDesktop Clone Deployment

You can validate ODX operations are taking place by using the ‘NFS Adapter’ Activity Traces page (yes, I said NFS, even though this is being performed via SMB).  The operations activity show will be ‘NfsWorkerVaaiCopyDataOp‘ when copying a vDisk and ‘NfsWorkerVaaiWriteZerosOp‘ when zeroing out a disk.


Important Pages

More coming soon!

Command Reference

Execute command on multiple remote hosts

Description: Execute a PowerShell on one or many remote hosts

$targetServers = "Host1","Host2","Etc"
Invoke-Command -ComputerName  $targetServers {

Check available VMQ Offloads

Description: Display the available number of VMQ offloads for a particular host

gwmi –Namespace "root\virtualization\v2" –Class Msvm_VirtualEthernetSwitch | select elementname, MaxVMQOffloads

Disable VMQ for VMs matching a specific prefix

Description: Disable VMQ for specific VMs

$vmPrefix = "myVMs"
Get-VM | Where {$_.Name -match $vmPrefix} | Get-VMNetworkAdapter | Set-VMNetworkAdapter -VmqWeight 0

Enable VMQ for VMs matching a certain prefix

Description: Enable VMQ for specific VMs

$vmPrefix = "myVMs"
Get-VM | Where {$_.Name -match $vmPrefix} | Get-VMNetworkAdapter | Set-VMNetworkAdapter -VmqWeight 1

Power-On VMs matching a certain prefix

Description: Power-On VMs matching a certain prefix

$vmPrefix = "myVMs"
Get-VM | Where {$_.Name -match $vmPrefix -and $_.StatusString -eq "Stopped"} | Start-VM

Shutdown VMs matching a certain prefix

Description: Shutdown VMs matching a certain prefix

$vmPrefix = "myVMs"
Get-VM | Where {$_.Name -match $vmPrefix -and $_.StatusString -eq "Running"}} | Shutdown-VM -RunAsynchronously

Stop VMs matching a certain prefix

Description: Stop VMs matching a certain prefix

$vmPrefix = "myVMs"
Get-VM | Where {$_.Name -match $vmPrefix} | Stop-VM

Get Hyper-V host RSS settings

Description: Get Hyper-V host RSS (recieve side scaling) settings


Check Winsh and WinRM connectivity

Description: Check Winsh and WinRM connectivity / status by performing a sample query which should return the computer system object not an error

allssh "winsh "get-wmiobject win32_computersystem"

Metrics and Thresholds

More coming soon!

Troubleshooting & Advanced Administration

More coming soon!

Part 2: Services

Book of Test Drive

As the saying goes "seeing is believing" which I couldn't agree with more. This document serves as a basis of foundational knowledge covering how the product works and its architecture. Essentially it goes into detail on the conceptual nature of the product.

However, to demonstrate true understanding, one may argue you need a mix of both conceptual learning in addition to hands on experience, as visualized below:

Test Drive - Conceptual Architecture
Test Drive - Conceptual Architecture

Nutanix Test Drive is the service which allows people to experience the Nutanix products in action. This is starting as a product focused on experiencing the core product but will eventually turn into the experience for any and all Nutanix products (e.g. Core, Frame, Beam, etc.)

Simply put, Test Drive is synonymous with experiencing Nutanix.


Want to try it out? Click on the link below to take it for a spin!

Test Drive

Some Background

Our first attempt at a "trial experience" started with something we called Community Edition (CE). CE allowed users to install Nutanix software on limited set of hardware. While this was good for some individuals who liked to be hands on and tinker, it didn't allow us to fully achieve our goal: allowing everyone to quick experience the Nutanix Platform.

From these learnings we set the following requirements for Test Drive:

  1. It must allow people to rapidly experience the Nutanix Platform
  2. It must guide them through the product and activities

Based upon those two key requirements, it was clear the experience needed to consist of two core items: the environment and the guide.

Experience Components

There are two core components of the Test Drive experience:

  • Environment
    • Where the experience is occuring
    • Could be hosted "somewhere" or a local environment
    • May include data / configuration specific to the experience taking place
  • Guide
    • What is walking you through the experience
    • Could be a workbook, an in-Prism guide, an instructor, etc.

The following image shows these two components:

Test Drive - High-Level Architecture
Test Drive - High-Level Architecture
The Experience

To start a Test Drive you can launch it from the MyNutanix page or navigate to the Test Drive main page on (

Once you're in to your Test Drive environment you can select from a series of common themes:

Test Drive - Themes
Test Drive - Themes

Once you've chosen a theme you can see the various activities (and sub-activities) that are part of that theme:

Test Drive - Activities
Test Drive - Activities

Upon selecting an activity the guide will walk you through the activity's steps overlayed onto Prism:

Test Drive - Prism Guide
Test Drive - Prism Guide

Continue through the guide until the activity is completed. Upon completion a new activity can be launched:

Test Drive - Activity Complete
Test Drive - Activity Complete

The above covers some of the ideas around Test Drive and what we're trying to achieve. Simply put, we are proud of the Nutanix platform and want to allow anyone / everyone to try it out. The following sections will cover some of the Test Drive hosting environments that can be used (more coming soon).

Test Drive on GCP

Test Drive on GCP is one of the Test Drive environments that can be used which runs in GCP using virtual Nutanix clusters.

The idea for virtual / nested clusters originated in our Engineering organization where we leverage a great deal of nested virtualization / AHV for development and QA purposes. This allows us to greatly over-subscribe resources and test larger scales without requiring tons of excess capacity. This was originally referred to as "NullHV" internally.

Test Drive on GCP takes this concept of nested AHV and builds it on top of GCP's nested virtualization capability (LINK). This allows us to run Nutanix software on GCP and provide massive scalability with a completely on-demand infrastructure.

The following image shows the conceptual Nutanix Test Drive "Cluster":

Test Drive - GCP Environment
Test Drive - GCP Environment

A virtual Nutanix Cluster is created by running a pair of native GCP VMs for the AHV host and the CVM. These VMs work together to simulate a single Nutanix node. Similar to a typical deployment the CVM handles storage and has locally attached SSD, where the AHV host provides the compute (CPU/MEM) to the UVMs. In cases where PC is needed, another PC VM will be spun up to manage the environment.

Book of Nutanix Clusters

Nutanix Clusters on AWS

Nutanix Clusters on AWS provides on-demand clusters running in target cloud environments using bare metal resources. This allows for true on-demand capacity with the simplicity of the Nutanix platform you know. Once provisioned the cluster appears like any traditional AHV cluster, just running in a cloud providers datacenters.

Supported Configurations

The solution is applicable to the configurations below (list may be incomplete, refer to documentation for a fully supported list):

    Core Use Case(s):
  • On-Demand / burst capacity
  • Backup / DR
  • Cloud Native
  • Geo Expansion / DC consolidation
  • App migration
  • Etc.
    Management interfaces(s):
  • Nutanix Clusters Portal - Provisioning
  • Prism Central (PC) - Nutanix Management
  • AWS Console - AWS Management
    Supported Environment(s):
  • Cloud:
    • AWS (EA)
  • EC2 Metal Instance Types:
    • i3.metal
    • m5d.metal
    • z1d.metal
  • Part of AOS
    Compatible Features:
  • AOS Features
  • AWS Services
Key terms / Constructs

The following key items are used throughout this section and defined in the following:

  • Nutanix Clusters Portal
    • The Nutanix Clusters Portal is responsible for handling cluster provisioning requests and interacting with AWS and the provisioned hosts. It creates cluster specific details and handles the dynamic CloudFormation stack creation.
  • Region
    • A geographic landmass or area where multiple Availability Zones (sites) are located. A region can have two or more AZs. These can include regions like US-East-1 or US-West-1.
  • Availability Zone (AZ)
    • An AZ consists of one or more discrete datacenters inter-connected by low latency links. Each site has it's own redundant power, cooling, network, etc. Comparing these to a traditional colo or datacenter, these would be considered more resilient as a AZ can consist of multiple independent datacenters. These can include sites like US-East-1a or US-West-1a.
  • VPC
    • A logically isolated segment of the AWS cloud for tenants. Provides a mechanism to to secure and isolate environment from others. Can be exposed to the internet or other private network segments (other VPCs, or VPNs).
  • S3
    • Amazon's object service which provides persistent object storage accessed via the S3 API. This is used for archival / restore.
  • EBS
    • Amazon's volume / block service which provides persistent volumes that can be attached to AMIs.
  • Cloud Formation Template (CFT)
    • A Cloud Formation Template simplifies provisioning, but allowing you to define a "stack" of resources and dependencies. This stack can then be provisioned as a whole instead of each individual resource.

From a high-level the Nutanix Clusters Portal is the main interface for provisioning Nutanix Clusters on AWS and interacting with AWS.

The provisioning process can be summarized with the following high-level steps:

  1. Create cluster in Nutanix Clusters Portal
  2. Deployment specific inputs (e.g. Region, AZ, Instance type, VPC/Subnets, etc.)
  3. The Nutanix Cluster Orchestrator creates associated resources
  4. Host agent in Nutanix AMI checks-in with Nutanix Clusters on AWS
  5. Once all hosts as up, cluster is created

The following shows a high-level overview of the Nutanix Clusters on AWS interaction:

Nutanix Clusters on AWS - Overview
Nutanix Clusters on AWS - Overview

The following shows a high-level overview of a the inputs taken by the cluster orchestrator and some created resources:

Nutanix Clusters on AWS - Cluster Orchestrator Inputs
Nutanix Clusters on AWS - Cluster Orchestrator Inputs

The following shows a high-level overview of a node in AWS:

Nutanix Clusters on AWS - Node Architecture
Nutanix Clusters on AWS - Node Architecture

Given the hosts are bare metal, we have full control over storage and network resources similar to a typical on-premise deployment. For the CVM and AHV host boot, EBS volumes are used. NOTE: certain resources like EBS interaction run through the AWS Nitro card which appears as a NVMe controller in the AHV host.

Placement policy

Nutanix Clusters on AWS uses a partition placement policy with 7 partitions by default. Hosts are striped across these partitions which correspond with racks in Nutanix. This ensures you can have 1-2 full "rack" failures and still maintain availability.

The following shows a high-level overview of the partition placement strategy and host striping:

Nutanix Clusters on AWS - Partition Placement
Nutanix Clusters on AWS - Partition Placement

In cases where multiple node types are leveraged (e.g. i3.metal and m5d.metal, etc.), each node type has its own 7 partitions which nodes are striped across.

The following shows a high-level overview of the partition placement strategy and host striping when multiple instance types are used:

Nutanix Clusters on AWS - Partition Placement (Multi)
Nutanix Clusters on AWS - Partition Placement (Multi)

Storage for Nutanix Clusters on AWS can be broken down into two core areas:

  1. Core / Active
  2. Hibernation

Core storage is the exact same as you'd expect on any Nutanix cluster, passing the "local" storage devices to the CVM to be leveraged by Stargate.

Instance Storage

Given that the "local" storage is backed by the AWS instance store, which isn't fully resilient in the event of a power outage / node failure additional considerations must be handled.

For example, in a local Nutanix cluster in the event of a power outage or node failure, the storage is persisted on the local devices and will come back when the node / power comes back online. In the case of the AWS instance store, this is not the case.

In most cases it is highly unlikely that a full AZ will lose power / go down, however for sensitive workloads it is recommended to:

  • Leverage a backup solution to persist to S3 or any durable storage
  • Replicate data to another Nutanix cluster in a different AZ/Region/Cloud (on-prem or remote)

One unique ability with Nutanix Clusters on AWS is the ability to "hibernate" a cluster allowing you to persist the data while spinning down the EC2 compute instances. This could be useful for cases where you don't need the compute resources and don't want to continue paying for them, but want to persist the data and have the ability to restore at a later point.

When a cluster is hibernated, the data will be backed up from the cluster to S3. Once the data is backed up the EC2 instances will be terminated. Upon a resume / restore, new EC2 instances will be provisioned and data will be loaded into the cluster from S3.


Networking can be broken down into a few core areas:

  • Host / Cluster Networking
  • Guest / UVM Networking
  • WAN / L3 Networking
Native vs. Overlay

Instead of running our own overlay network, we decided to run natively on AWS subnets, this allows VMs running on the platform to natively communicate with AWS services with zero performance degradation.

Nutanix Clusters on AWS are provisioned into an AWS VPC, the following shows a high-level overview of an AWS VPC:

Nutanix Clusters on AWS - AWS VPC
Nutanix Clusters on AWS - AWS VPC
New vs. Default VPC

AWS will create a default VPC/Subnet/Etc. with a ip scheme for each region.

It is recommended to create a new VPC with associated subnets, NAT/Internet Gateways, etc. that fits into your corporate IP scheme. This is important if you ever plan to extend networks between VPCs (VPC peering), or to your existing WAN. I treat this as I would any site on the WAN.

Host Networking

The hosts running on baremetal in AWS are traditional AHV hosts, and thus leverage the same OVS based network stack.

The following shows a high-level overview of a AWS AHV host's OVS stack:

Nutanix Clusters on AWS - OVS Architecture
Nutanix Clusters on AWS - OVS Architecture

The OVS stack is relatively the same as any AHV host except for the addition of the L3 uplink bridge.

For UVM (Guest VM) networking, VPC subnets are used. A UVM network can be created during the cluster creation process or via the following steps:

From the AWS VPC dashboard, click on 'subnets' then click on 'Create Subnet' and input the network details:

Nutanix Clusters on AWS - Create Subnet
Nutanix Clusters on AWS - OVS Architecture

NOTE: the CIDR block should be a subset of the VPC CIDR range.

The subnet will inherit the route table from the VPC:

Nutanix Clusters on AWS - Route Table
Nutanix Clusters on AWS - Route Table

In this case you can see any traffic in the peered VPC will go over the VPC peering link and any external traffic will go over the internet gateway.

Once complete, you will see the network is available in Prism.

WAN / L3 Networking

In most cases deployments will not be just in AWS and will need to communicate with the external world (Other VPCs, Internet or WAN).

For connecting VPCs (in the same or different regions), you can use VPC peering which allows you to tunnel between VPCs. NOTE: you will need to ensure you follow WAN IP scheme best practices and there are no CIDR range overlaps between VPCs / subnets.

The following shows a VPC peering connection between a VPC in the eu-west-1 and eu-west-2 regions:

Nutanix Clusters on AWS - VPC Peering
Nutanix Clusters on AWS - VPC Peering

The route table for each VPC will then route traffic going to the other VPC over the peering connection (this will need to exist on both sides if communication needs to be bi-directional):

Nutanix Clusters on AWS - Route Table
Nutanix Clusters on AWS - Route Table

For network expansion to on-premise / WAN, either a VPN gateway (tunnel) or AWS Direct Connect can be leveraged.


Given these resources are running in a cloud outside our full control security, data encryption and compliance is a very critical consideration.

The recommendations can be characterized with the following:

  • Enable data encryption
  • Only use private subnets (no public IP assignment)
  • Lock down security groups and allowed ports / IP CIDR blocks
  • For more granular security, leverage Flow
Usage and Configuration

The following sections cover how to configure and leverage Nutanix Clusters on AWS.

The high-level process can be characterized into the following high-level steps:

  1. Create AWS Account(s)
  2. Configure AWS network resources (if necessary)
  3. Provision cluster(s) via Nutanix Clusters Portal
  4. Leverage cluster resources once provisioning is complete

More to come!

Book of Storage Services

Volumes (Block Services)

The Nutanix Volumes feature (previously know as Acropolis Volumes) exposes back-end DSF storage to external consumers (guest OS, physical hosts, containers, etc.) via iSCSI.

This allows any operating system to access DSF and leverage its storage capabilities.  In this deployment scenario, the OS is talking directly to Nutanix bypassing any hypervisor. 

Core use-cases for Volumes:

  • Shared Disks
    • Oracle RAC, Microsoft Failover Clustering, etc.
  • Disks as first-class entities
    • Where execution contexts are ephemeral and data is critical
    • Containers, OpenStack, etc.
  • Guest-initiated iSCSI
    • Bare-metal consumers
    • Exchange on vSphere (for Microsoft Support)
Qualified Operating Systems

The solution is iSCSI spec compliant, the qualified operating systems are just those of which have been validated by QA.

  • Microsoft Windows Server 2008 R2, 2012 R2
  • Redhat Enterprise Linux 6.0+
Volumes Constructs

The following entities compose Volumes:

  • Data Services IP: Cluster wide IP address used for iSCSI login requests (Introduced in 4.7)
  • Volume Group: iSCSI target and group of disk devices allowing for centralized management, snapshotting, and policy application
  • Disk(s): Storage devices in the Volume Group (seen as LUNs for the iSCSI target)
  • Attachment: Allowing a specified initiator IQN access to the volume group
  • Secret(s): Secret used for CHAP/Mutual CHAP authentication

NOTE: On the backend, a VG’s disk is just a vDisk on DSF.


Before we get to configuration, we need to configure the Data Services IP which will act as our central discovery / login portal.

We'll set this on the 'Cluster Details' page (Gear Icon -> Cluster Details):

Volumes - Data Services IP
Volumes - Data Services IP

This can also be set via NCLI / API:

ncli cluster edit-params external-data- services-ip-address=<DATA SERVICES IP ADDRESS>

Target Creation

To use Volumes, the first thing we'll do is create a 'Volume Group' which is the iSCSI target.

From the 'Storage' page click on '+ Volume Group' on the right hand corner:

Volumes - Add Volume Group
Volumes - Add Volume Group

This will launch a menu where we'll specify the VG details:

Volumes - Add VG Details
Volumes - Add VG Details

Next we'll click on '+ Add new disk' to add any disk(s) to the target (visible as LUNs):

A menu will appear allowing us to select the target container and size of the disk:

Volumes - Add Disk
Volumes - Add Disk

Click 'Add' and repeat this for however many disks you'd like to add.

Once we've specified the details and added disk(s) we'll attach the Volume Group to a VM or Initiator IQN. This will allow the VM to access the iSCSI target (requests from an unknown initiator are rejected):

Volumes - Add Initiator IQN / VM
Volumes - Initiator IQN / VM

Click 'Save' and the Volume Group configuration is complete!

This can all be done via ACLI / API as well:

# Create VG

vg.create <VG Name>

# Add disk(s) to VG

Vg.disk_create <VG Name> container=<CTR Name> create_size=<Disk size, e.g. 500G>

# Attach initiator IQN to VG

Vg.attach_external <VG Name> <Initiator IQN>

Path High-Availability (HA)

As mentioned previously, the Data Services IP is leveraged for discovery. This allows for a single address that can be leveraged without the need of knowing individual CVM IP addresses.

The Data Services IP will be assigned to the current iSCSI leader. In the event that fails, a new iSCSI leader will become elected and assigned the Data Services IP. This ensures the discovery portal will always remain available.

The iSCSI initiator is configured with the Data Services IP as the iSCSI target portal. Upon a login request, the platform will perform an iSCSI login redirect to a healthy Stargate.

Volumes - Login Redirect
Volumes - Login Redirect

In the event where the active (affined) Stargate goes down, the initiator retries the iSCSI login to the Data Services IP, which will then redirect to another healthy Stargate.

Volumes - Failure Handling
Volumes - Failure Handling

If the affined Stargate comes back up and is stable, the currently active Stargate will quiesce I/O and kill the active iSCSI session(s). When the initiator re-attempts the iSCSI login, the Data Services IP will redirect it to the affined Stargate.

Volumes - Failback
Volumes - Failback
Health Monitoring and Defaults

Stargate health is monitored using Zookeeper for Volumes, using the exact same mechanism as DSF.

For failback, the default interval is 120 seconds. This means once the affined Stargate is healthy for 2 or more minutes, we will quiesce and close the session. Forcing another login back to the affined Stargate.

Given this mechanism, client side multipathing (MPIO) is no longer necessary for path HA. When connecting to a target, there's now no need to check 'Enable multi-path' (which enables MPIO):

Volumes - No MPIO
Volumes - No MPIO

The iSCSI protocol spec mandates a single iSCSI session (TCP connection) per target, between initiator and target. This means there is a 1:1 relationship between a Stargate and a target.

As of 4.7, 32 (default) virtual targets will be automatically created per attached initiator and assigned to each disk device added to the volume group (VG). This provides an iSCSI target per disk device. Previously this would have been handled by creating multiple VGs with a single disk each.

When looking at the VG details in ACLI/API you can see the 32 virtual targets created for each attachment:

attachment_list { external_initiator_name: "" target_params { num_virtual_targets: 32 } }

Here we've created a sample VG with 3 disks devices added to it. When performing a discovery on my client I can see an individual target for each disk device (with a suffix in the format of '-tgt[int]'):

Volumes - Virtual Target
Volumes - Virtual Target

This allows each disk device to have its own iSCSI session and the ability for these sessions to be hosted across multiple Stargates, increasing scalability and performance:

Volumes - Multi-Path
Volumes - Multi-Path

Load balancing occurs during iSCSI session establishment (iSCSI login), for each target.

Active Path(s)

You can view the active Stargate(s) hosting the virtual target(s) with the following command (will display CVM IP for hosting Stargate):

# Windows
Get-NetTCPConnection -State Established -RemotePort 3205

# Linux
iscsiadm -m session -P 1

As of 4.7 a simple hash function is used to distribute targets across cluster nodes. In 5.0 this is integrated with the Dynamic Scheduler which will re-balance sessions if necesary. We will continue to look at the algorithm and optimize as necessary. It is also possible to set a preferred node which will be used as long as it is in a healthy state.


Volumes supports the SCSI UNMAP (TRIM) command in the SCSI T10 specification. This command is used to reclaim space from deleted blocks.

Files (File Services)

The Nutanix Files feature allows users to leverage the Nutanix platform as a highly available file server. This allows for a single namespace where users can store home directories and files.

Supported Configurations

The solution is applicable to the configurations below (list may be incomplete, refer to documentation for a fully supported list):

    Core Use Case(s):
  • Home folders / user profiles
  • Filer storage
  • Media server
    Management interfaces(s):
  • Prism Element (PE)
  • AHV
  • ESXi (AOS 5.0 and beyond)
  • Prism
    Compatible Features:
  • Nutanix Snapshots and DR
  • File level snapshots including Windows Previous Version (WPV)
  • Self Service Restore
  • CFT Backups
    File Protocols:
  • CIFS 2.1
  • NFS v4
  • NFS v3 (as of AFS 3.5)
Files Constructs

This feature is composed of a few high-level constructs:

  • File Server
    • High-level namespace. Each file server will have its own set of Files VMs (FSVM) which are deployed
  • Share
    • Share exposed to users. A file server can have multiple shares (e.g. departmental shares, etc.)
  • Folder
    • Folders for file storage. Folders are sharded across FSVMs

The figure shows the high-level mapping of the constructs:

Files Mapping
Files Mapping

The Nutanix Files feature follows the same methodology for distribution as the Nutanix platform to ensure availability and scale. A minimum of 3 FSVMs will be deployed as part of the File Server deployment.

The figure shows a detailed view of the components:

Files Detail
Files Detail

The FSVMs are combined into a logical file server instance sometimes referred to as a Files cluster. You can create multiple Files clusters within a single Nutanix cluster. The FSVMs are transparently deployed as part of the configuration process.

The figure shows a detailed view of FSVMs on the AOS platform:

Files Detail
FSVM Deployment Arch
Authentication and Authorization

The Nutanix Files feature is fully integrated into Microsoft Active Directory (AD) and DNS. This allows all of the secure and established authentication and authorization capabilities of AD to be leveraged. All share permissions, user and group management is done using the traditional Windows MMC for file management.

As part of the installation process the following AD / DNS objects will be created:

  • AD Computer Account for File Server
  • AD Service Principal Name (SPN) for File Server and each FSVM
  • DNS entry for File Server pointing to all FSVM(s)
  • DNS entry for each FSVM
AD Privileges for File Server Creation

A user account with the domain admin or equivalent privileges must be used to deploy the File Service feature as AD and DNS objects are created.

High-Availability (HA)

Each FSVM leverages the Volumes API for its data storage which is accessed via in-guest iSCSI. This allows any FSVM to connect to any iSCSI target in the event of a FSVM outage.

The figure shows a high-level overview of the FSVM storage:

FSVM Storage
FSVM Storage

To provide for path availability DM-MPIO is leveraged within the FSVM which will have the active path set to the local CVM by default:


In the event where the local CVM becomes unavailable (e.g. active path down), DM-MPIO will activate one of the failover paths to a remote CVM which will then takeover IO.

FSVM MPIO Failover
FSVM MPIO Failover

When the local CVM comes back and is healthy it will be marked as the active path to provide for local IO.

In a normal operating environment each FSVM will be communicating with its own VG for data storage with passive connections to the others. Each FSVM will have an IP which clients use to communicate with the FSVM as part of the DFS referral process. Clients do not need to know each individual FSVM's IP as the DFS referral process will connect them to the correct IP hosting their folder(s).

FSVM Normal Operation
FSVM Normal Operation

In the event of a FSVM "failure" (e.g. maintenance, power off, etc.) the VG and IP of the failed FSVM will be taken over by another FSVM to ensure client availability.

The figure shows the transfer of the failed FSVM's IP and VG:

FSVM Failure Scenario
FSVM Failure Scenario

When the failed FSVM comes back and is stable, it will re-take its IP and VG and continue to serve client IO.

Objects (Object Services)

The Nutanix Objects feature provides highly scalable and durable object services via an S3 compliant API (More Information on S3: LINK). Given Nutanix Objects is deployed on top of the Nuatnix platform, it can take advantage of AOS features like deduplication, compression, replication and more. Objects was introduced in AOS 5.11.

Supported Configurations

The solution is applicable to the configurations below (list may be incomplete, refer to documentation for a fully supported list):

    Core Use Case(s):
  • Backups
  • Big data/analytics
    Management interfaces(s):
  • Prism Central (PC)
  • N/A - Runs on Nutanix MSP (Dependent on MSP supported Hypervisors)
  • LCM
    Compatible Features:
  • TBI
    Object Protocols:
  • S3 (version 4)
Nutanix Microservices Platform (MSP)

Nutanix Objects leverages the Nutanix Microservices Platform (MSP) and is one of the first core services to do so.

Nutanix MSP provides a common framework and services to deploy the Objects component's associated containers and platform services like Identity and Access Management (IAM) and Load Balancing (LB).

Key terms

The following key terms are used throughout this section and defined in the following:

  • Bucket
    • An organization unit exposed to users and contains the objects (think share to a file on a file server). A deployment can, and typically will, have multiple buckets (e.g. departmental, compartmental, etc.)
  • Object
    • The actual unit (blob) of storage and item interfaced with via the API (GET/PUT).
  • S3
    • The term used to describe the original object service Amazon Web Services (AWS) introduced. Now is used synonomysously for an object service. S3 also is used to define the object API which is highly leveraged throughout projects.

The figure shows the high-level mapping of the conceptual structure:

Objects - Hierarchy
Objects - Hierarchy
Objects Constructs

This feature is composed of a few high-level constructs:

  • Load Balancer
    • The load balancer is part of the Nutanix MSP and serves as a proxy for service and data requests. This ensures high-availability for the service and load balancing among the Objects containers.
  • Service Manager
    • The service manager serves as the endpoint for all UI requests and manages object store instances. It is also responsible for collecting stats from instances.
  • Metadata Server
    • The metadata server is responsible to containing all the meta information around a Nutanix Objects deployment (e.g. buckets, objects, etc.). Leverages ChakrDB which is a RocksDB based Key-Value store developed by Nutanix. ChakrDB uses Nutanix ABS for storage.
  • Object Controller
    • The Object Controller is responsible for managing object data and coordinates metadata updates with the Metadata Server. It interfaces with Stargate via the Storage Proxy API.
  • Region Manager
    • The Region Manager is responsible to managing all of the object storage information (e.g. Region) on DSF.
  • Region
    • A region provides the high-level mapping between an object and the corresponding locations on Nutanix vDisk(s). Similar to a vDisk ID, offset and length.
  • Atlas Service
    • The Atlas Service is responsible for object lifecycle policy enforcement and performing garbage collection.

The figure shows a detailed view of the Objects service architecture:

Objects - Architecture
Objects - Architecture

The Objects specific components are highlighted in Nutanix Green. With objects there's no concept of an "overwrite" hence the CxxD vs. CRUD (Create/Replace/Update/Delete). The commonly employed method for an object "overwrite" is to create a new revision or create a new object and point to the new object.

Object Storage and I/O

An object is stored in logical constructs called regions. A region is a fixed segment of space on a vDisk.

The figure shows an example of the relationship between a vDisk and region:

Objects - vDisk Region
Objects - vDisk Region

Smaller objects may fit in a chunk of a single region (region id, offset, length), whereas larger objects may get striped across regions. When a large object is striped across multiple regions these regions can be hosted on multiple vDisks allowing multiple Stargates to be leveraged concurrently.

The figure shows an example of the relationship between a object, chunk and region:

Objects - Object Chunk
Objects - Object Chunk

The object services feature follows the same methodology for distribution as the Nutanix platform to ensure availability and scale. A minimum of 3 object VMs will be deployed as part of the Objects deployment.

Book of Network Services

Flow (Microsegmentation)

Flow is a distributed stateful firewall that enables granular network monitoring and enforcement between entities running on the AHV platform as well as external things they communicate with.

Supported Configurations

The solution is applicable to the configurations below (list may be incomplete, refer to documentation for a fully supported list):

    Core Use Case(s):
  • Microsegmentation
    Management interfaces(s):
  • Prism Central (PC)
    Supported Environment(s):
  • On-Premise:
    • AHV (As of AOS 5.10)
  • Cloud:
  • Part of AOS
    Compatible Features:
  • Service Chaining
  • Calm
  • Epoch

The configuration is done via Prism Central by defining policies and assigning to categories. This allows the configuration to be done in a central place and pushed to many Nutanix clusters. Each AHV host implements the rules using OpenFlow.

Implementation Constructs

Within Nutanix Flow, there are a few key constructs:


Categories are used to define groups of entities which policies and enforcement are applied to. They typically apply, but are not limited to: environment, application type, application tier, etc.

  • Category: Key/Value "Tag"
  • Examples: app | tier | group | location | subnet | etc.

For example, a VM providing production database services may have the following assigned categories:

  • AppTier: Database
  • AppType: MySQL
  • Environment: Production

These categories can then be leveraged by policies to determine what rules / actions to apply (also leveraged outside of the Flow context).

Security Rule

Security rule(s) are the defined rules and determine what is allowed between defined categories.

Flow - Microsegmentation - Rules
Flow - Microsegmentation - Rules

There are a few types of security rules:

  • App Rule
    • This is your common rule allowing you to define what transport (TCP/UDP), Port, and source/destination is allowed/denied.
    • [Allow|Deny] Transport: Port(s) [To|From]
    • Example: Allow TCP 8080 from Category:Tier:Web to Category:Tier:App
  • Isolation Rule
    • Deny traffic between two categories, allow traffic within category
    • Example: seperate tenant A from tenant B, clone environment and allow to run in parallel without affecting normal network communication.
  • Quarantine Rule
    • Deny All traffic for specified VM(s)/categories
    • Example: VMs A,B,C infected with a virus, isolate them to stop the virus from further infecting the network

The following shows an example utilizing Flow - Microsegmentation to control traffic in a sample application:

Flow - Microsegmentation - Example Application
Flow - Microsegmentation - Example Application

Enforcement determines what action is taken when a rule is matched. With AHV Flow - Microsegmentation there are two types of enforcement:

  • Apply
    • Enforce the policy by allowing defined flows and dropping all others.
  • Monitor
    • Allow all flows, but highlight any packets that would have violated the policy in the policy visualization page.

Flow - Microsegmentation rules are the first applied to a packet once it leaves the UVM. This occurs in the microsegmentation bridge (br.microseg):

Flow - Microsegmentation - Flow
Flow - Microsegmentation - Flow

Book of Backup / DR Services

Leap (Policy Driven DR / Run Books)

Test Drive

For those who are interested in getting hands on, take it for a spin with Nutanix Test Drive!

The Nutanix Leap feature provides policy driven backup, DR and run book automation services configured via Prism Central (PC). This capability builds upon and extends the native DR and replications features that have been availble in AOS and configured in PE for years. For more information on the actual back-end mechanism being leveraged for replication, etc. refer to the 'Backup and Disaster Recovery (DR)' section in the 'Book of AOS'. Leap was introduced in AOS 5.10.

Supported Configurations

The solution is applicable to the configurations below (list may be incomplete, refer to documentation for a fully supported list):

    Core Use Case(s):
  • Policy based backups and replication
  • DR run book automation
  • DRaaS (via Xi)
    Management interfaces(s):
  • Prism Central (PC)
    Supported Environment(s):
  • On-Premise:
    • AHV (As of AOS 5.10)
  • Cloud:
    • Xi (As of AOS 5.10)
  • Part of AOS
    Compatible Features:
  • AOS BC/DR features
Key terms

The following key terms are used throughout this section and defined in the following:

  • Recovery Point Objective (RPO)
    • Refers to the acceptable data loss in the event of a failure. For example, if you want an RPO of 1 hour, you'd take a snapshot every 1 hour. In the event of a restore, you'd be restoring data as of up to 1 hour ago. For synchronous replication typically an RPO of 0 is achieved.
  • Recovery Time Objective (RTO)
    • Recovery time objective. Refers to the period of time from failure event to restored service. For example, if a failure occurs and you need things to be back up and running in 30 minutes, you'd have an RTO of 30 minutes.
  • Recovery Point
    • A restoration point aka snapshot.
Implementation Constructs

Within Nutanix Leap, there are a few key constructs:

Protection Policy
  • Key Role: Backup/Replication policy for assigned categories
  • Description: A protection policy defines the RPO (snap frequency), recovery location (remote cluster / Xi), snapshot retention (local vs. remote cluster), and associated categories. With Protection Policies everything is applied at the category level (with a default that can apply to any/all). This is different from Protection Domains where you have to select VM(s).

The following image shows the structure of the Nutanix Leap Protection Policy:

Leap - Protection Policy
Leap - Protection Policy
Recovery Plan
  • Key Role: DR run book
  • Description: A Recovery Plan is a run book that defines the power on sequencing (can specify categories or VMs) and network mapping (primary vs. recovery and test failover / failback). This is most synonymous with what people would leverage SRM for. NOTE: a Protection Policy must be configured before a Recovery Plan can be configured. This is necessary as the data must exist at the recovery site in order for it to be recovered.

The following image shows the structure of the Nutanix Leap Recovery Plan:

Leap - Recovery Plan
Leap - Recovery Plan
Linear Retention Policy
  • Key Role: Recovery Point retention policy
  • Description: A linear retention policy specifies the number of recovery points to retain. For example, if the RPO is 1 hour and your retention is set to 10, you'd keep 10 hours (10 x 1 hour) of recovery points (snaps).
Roll-up Retention Policy
  • Key Role: Recovery Point retention policy
  • Description: A roll-up retention policy will "roll-up" snaps dependent on the RPO and retention duration. For example, if the RPO is 1 hour and your retention is set to 5 days it'll keep 1 day of hourly and 4 days of daily recovery points. The logic can be characterized as follows: If retention is n days, keep 1 day of RPO and n-1 days of daily recovery points. If retention is n weeks, keep 1 day of RPO and 1 week of daily and n-1 weeks of weekly recovery points. If retention is n months, keep 1 day of RPO and 1 week of daily and 1 month of weekly and n-1 months of monthly recovery points. If retention is n years, keep 1 day of RPO and 1 week of daily and 1 month of weekly and n-1 months of monthly recovery points.
Linear vs. roll-up retention

Use linear policies for small RPO windows with shorter retention periods or in cases where you always need to be able to recover to a specific RPO window.

Use roll-up policies for anything with a longer retention period. They're more flexible and automatically handle snapshot aging / pruning while still providing granular RPOs for the first day.

The following shows a high-level overview of the Leap constructs:

Leap - Overview
Leap - Overview

The following shows how Leap can replicate between on-premise and Xi:

Leap - topology
Leap - Topology
Usage and Configuration

The following sections cover how to configure and leverage Leap.

The high-level process can be characterized into the following high-level steps:

  1. Connect to Availability Zones (AZs)
  2. Configure Protection Policies
  3. Configure Recovery Plan(s)
  4. Perform/Test Failover & Failback
Connect Availability Zone(s)

The first step is connecting to an AZ which can be a Xi AZ or another PC. NOTE: As of 5.11 you will need at least 2 PCs deployed (1 for each site).

In PC, search for 'Availability Zones' or navigate to 'Administration' -> 'Availability Zones':

Leap - Connect to Availability Zone
Leap - Connect to Availability Zone

Click on 'Connect to Availability Zone' and select the AZ Type ('Xi' or 'Physical Location' aka PC instance):

Leap - Connect to Availability Zone
Leap - Connect to Availability Zone

Input credentials for PC or Xi and click 'Connect':

Leap - Connect to Availability Zone
Leap - Connect to Availability Zone

The connected AZ will now be displayed and be available.

Configure Protection Policies

In PC, search for 'Protection Policies' or navigate to 'Policies' -> 'Protection Policies':

Leap - Protection Policies
Leap - Protection Policies

Click on 'Create Protection Policy':

Leap - Protection Policy
Leap - Create Protection Policy

Input details for the name, recovery location, RPO and retention policy (describe previously):

Leap - Protection Policy Inputs
Leap - Protection Policy Inputs

NOTE: for Xi you don't need select a 'Target Cluster':

Leap - Protection Policy Inputs - Xi
Leap - Protection Policy Inputs - Xi

Next we'll select the categories for the policy to apply to:

Leap - Protection Categories
Leap - Protection Policy Categories

Click 'Save' and you will now see the newly created Protection Policy:

Leap - Protection Policies
Leap - Protection Policies
Configure Recovery Plans

In PC, search for 'Recovery Plans' or navigate to 'Policies' -> 'Recovery Plans':

Leap - Recovery Plans
Leap - Recovery Plans

On the first launch you will be greeted with a screen to create the first Recovery Plan:

Leap - Create Recovery Plan
Leap - Create Recovery Plan

Select the 'Recovery Location' using the drop down:

Leap - Select Recovery Location
Leap - Select Recovery Location

NOTE: This can be either a Xi AZ or Physical AZ (PC with corresponding managed clusters).

Input the Recovery Plan name and description and click 'Next':

Leap - Recovery Plan - Naming
Leap - Recovery Plan - Naming

Next click on 'Add Entities' and specify the power on sequence:

Leap - Recovery Plan - Power On Sequence
Leap - Recovery Plan - Power On Sequence

Search for VMs or Categories to add to each stage:

Leap - Recovery Plan - Power On Sequence
Leap - Recovery Plan - Power On Sequence

Once the power on sequence looks good with the stages, click 'Next':

Leap - Recovery Plan - Power On Sequence
Leap - Recovery Plan - Power On Sequence
Power On Sequencing

When determining the power on sequence you will want to stage things as follows:

  • Stage 0: Core services (AD, DNS, etc.)
  • Stage 1: Services dependent on Stage 0 services, and required for Stage 2 services (e.g. DB Tier)
  • Stage 2: Services dependent on Stage 1 services, and required for Stage 3 services (e.g. App Tier)
  • Stage 3: Services dependent on Stage 2 services, and required for Stage 4 services (e.g. Web Tier)
  • Stage 4-N: Repeat based upon dependencies

We will now map the network between our source and target environments:

Leap - Recovery Plan - Network Mapping
Leap - Recovery Plan - Network Mapping
Failover / Failback Networks

In most cases you will want to use a non-routable or isolated network for your test networks. This will ensure you don't have any issues with duplicate SIDs, arp entries, etc.

Mine (Backup Solution)

Coming soon!

Book of Orchestration Services

Calm (Orchestration / Automation)

Coming soon!

Book of Governance Services

Beam (Cost Governance / Compliance)

Coming soon!

Part 3: Scenarios

In the section we will cover various use cases, considerations and sample implementations with the Nutanix platform.

Scenario: secure analytics platform

In this scenario we will design a secure analytics platform ingesting data from various external and internal sources and ETL'ing into a data lake where analytics are performed. This is a scenario I am personally familiar with as I worked on an internal project called Project Earth doing just this.

Given the scope of this being very large and proprietary, only the security aspects of the platform will be discussed in the first iteration.

At a Glance
  • Key Requirements
    • Environment must be highly secured through all layers (network, application, etc.)
    • Access must be scoped to specific enclaves / users
    • Must consume data from both internal and external sources
    • Configuration must be automated
    • User management / RBAC must be 100% automated
    • Data must be encrypted
  • Solution Components
    • Front-End: Tableau
    • Service Catalog: Service Now
    • Approvals: Slack / Email -> ServiceNow
    • Configuration Automation: Puppet
    • Database: Apache MySQL / extensible
    • ESB / ETL: Custom + Apache Kafka
    • Analytics: Custom
  • Platform
    • Hypervisor: Nutanix AHV
    • Encryption: Nutanix software encryption
    • Microsegmentation: Nutanix Flow
    • Compute + Storage + Network (virtual): Nutanix

The following shows a high-level view of the solution layers and data flows:

Scenario - Secure Analytics Platform
Scenario - Secure Analytics Platform
Security Architecture

As mentioned in the 'Security and Encryption' section, security occurs at multple levels ranging from data to systems to people. In the following we will cover how we hardened each of these components at a high-level.

Networking & Communication

When it comes to networking and communication we need to ensure only known / secure enclaves were able to get access to the systems and data flows outbound are restricted.

We achieved this using a few items in alignment:

  • All configurations are 100% automated using Puppet
  • All policies are whitelist only
  • Only truted enclaves are allowed inbound on specific ports
  • All developer access flowed through a single jump box
  • MySQL users / grants were scoped to specific user / IP addresses combinations
  • Firewalld rules secured the Linux firewall
  • Nutanix Flow secured the virtual / physical network layer

The following shows the Flow policies for the dev/staging/production environments:

Scenario - Secure Analytics Platform - Flow Policies
Scenario - Secure Analytics Platform - Flow Policies

Only specific ports / protocols were allowed between application tiers and inbound:

Scenario - Secure Analytics Platform - Flow Policy Detail
Scenario - Secure Analytics Platform - Flow Policy Detail

Categories were leverated to specify app tiers and environments. Only certain ports were allowed between:

Scenario - Secure Analytics Platform - Policy Categories
Scenario - Secure Analytics Platform - Policy Categories

Here's a sample look at a Flow policy for dev which shows the allowed inbound sources. It also highlights the blocked connections which coincidentally were from an internal pentesting tool:

Scenario - Secure Analytics Platform - Flow Policy Detail
Scenario - Secure Analytics Platform - Flow Policy Detail
Systems and Configuration

When it comes to the stack there were a few core layers:

  • Application / Services
  • VMs / Containers
  • Infrastrucutre (Nutanix)

The full stack was 100% automated using Puppet, Nutanix SCMA and environment templates. This allowed us to ensure security / configuration baselines and adherance to them. This also allowed us to simply update packages if any security vulnerabilities were found.

Within the Nutanix platform the native SCMA was leveraged (enabled by defualt) which ensures a STIG'd / secure configuration for the CVMs and hosts. Cluster lockdown mode was enabled (recommended) to force key based access.


With any platform that is integrating with multiple systems, secret management is a very important item. Initially we started with using encrypted yaml (eyaml) within Puppet but eventually moved this to a more secure / manageable hiera backend. There are multiple options here like HashiCorp Vault, etc.

Data Encryption

Data encryption is important to ensure an attacker can't make any sense of the data if they were to steal or become in posession of it.

Native Nutanix software-based encryption was leveraged to provide data encryption. In scenarios where a key manager isn't available, the local key manager (LKM) can be leveraged. If using an external key manager (EKM) it is recommended to rotate keys, this occurs yearly with the LKM by default.

Scenario - Secure Analytics Platform - Data Encryption
Scenario - Secure Analytics Platform - Data Encryption
Data Scoping and RBAC

One of the most important things once the "stack" has been hardened, is ensuring only specified individuals get access to the data they should have access to and all access / requests / granting is fully auditible. From my perspective the only way to accurately do this is through a fully automated system where any granting of access, approvals, etc. has to flow through the automation and nothing can be done manually. This is exactly what we did with Project Earth, even as admins, we can't override access for users.

If we break this down there are a few core stages:

  • Requesting / inheriting access
  • Approving / denying access
  • Validation & Q/A
  • Revocation
  • Auditing

For all requests and ticketing we leverage ServiceNow (SNOW). Within SNOW we created a custom catalog in which users could request access to specific types of data. The available types of "roles" / data available are automatically generated and published to SNOW whenever new data sources / roles are created.

Once requested, it would go to their manager for approval and then two other approvers who must approve before any access is granted. This "dual key" approval ensured proper checks and balances. Another key point here is that the time which people could request access for was limited and could be revoked at any time. Upon expiration / revocation, membership was automatically removed.

Once the request was approved the role assignment / group membership was fully automated.

The following figure shows a high-level view of the flow:

Scenario - Secure Analytics Platform - Flow Policy Detail
Scenario - Secure Analytics Platform - RBAC / Role Assignment

For validation we have checks to ensure members of a group match the "approved" state in SNOW. All authentication / access requests are logged and stored in a central logging system. Using this system we could look for anomalous access or things out of the ordinary.

Change Control

A key for auditibility is proper change control throughout all aspects of the system. For requests / approvals those were all stored in SNOW. All other items whether it be Puppet, custom logic and code, etc. were all kept in a hardened source control system with only specific developers / keys have access. Any modifications/ new code first went through a code review process / security validation. Once reviewed / approved the changes were put into "purgatory" until validation in dev / staging environments were coplete. Any modifications to production systems are done using automation to ensure human error potential is minimized.

Scenario: multi-site DR / replication

Coming soon!

Part 4: Integrations


OpenStack is an open source platform for managing and building clouds.  It is primarily broken into the front-end (dashboard and API) and infrastructure services (compute, storage, etc.).

The OpenStack and Nutanix solution is composed of a few main components:

  • OpenStack Controller (OSC)
    • An existing, or newly provisioned VM or host hosting the OpenStack UI, API and services. Handles all OpenStack API calls. In an OVM deployment this can be co-located with the AOS OpenStack Drivers.
  • AOS OpenStack Driver
    • Responsible for taking OpenStack RPCs from the OpenStack Controller and translates them into native AOS API calls. This can be deployed on the OpenStack Controller, the OVM (pre-installed), or on a new VM.
  • AOS OpenStack Services VM (OVM)
    • VM with AOS drivers that is responsible for taking OpenStack RPCs from the OpenStack Controller and translates them into native AOS API calls.

The OpenStack Controller can be an existing VM / host, or deployed as part of the OpenStack on Nutanix solution. The OVM is a helper VM which is deployed as part of the Nutanix OpenStack solution.

The client communicates with the OpenStack Controller using their expected methods (Web UI / HTTP, SDK, CLI or API) and the OpenStack controller communicates with the OVM which translates the requests into native AOS REST API calls using the OpenStack Driver.

The figure shows a high-level overview of the communication:

OpenStack + AOS OpenStack Driver
OpenStack + AOS OpenStack Driver

This allows for the best of both worlds, the goodness of the OpenStack Portal and APIs, without the complex OpenStack infrastructure and associated management. All back-end infrastructure services (compute, storage, network) leverage the native Nutanix services. No need to deploy Nova Compute hosts, etc. The platform exposes APIs for these services which the controller communicates with then translates them into native AOS API calls. Also, given the simplified deployment model, the full OpenStack + Nutanix solution can be up in less than 30 minutes.

Supported OpenStack Controllers

The current solution (as of 4.5.1) requires an OpenStack Controller on version Kilo or later.

The table shows a high-level conceptual role mapping:

Item Role OpenStack Controller OVM Nutanix cluster Prism
Tenant Dashboard User interface and API X
Admin Dashboard Infra monitoring and ops X X
Orchestration Object CRUD and lifecycle management X
Quotas Resource controls and limits X
Users, Groups and Roles Role based access control (RBAC) X
SSO Single-sign on X
Platform Integration OpenStack to Nutanix integration X
Infrastructure Services Target infrastructure (compute, storage, network) X
OpenStack Components

OpenStack is composed of a set of components which are responsible for serving various infrastructure functions. Some of these functions will be hosted by the OpenStack Controller and some will be hosted by the OVM.

The table shows the core OpenStack components and role mapping:

Component Role OpenStack Controller OVM
Keystone Identity service X
Horizon Dashboard and UI X
Nova Compute X
Swift Object storage X X
Cinder Block storage X
Glance Image service X X
Neutron Networking X
Heat Orchestration X
Others All other components X

The figure shows a more detailed view of the OpenStack components and communication:

OpenStack + Nutanix API Communication
OpenStack + Nutanix API Communication

In the following sections we will go through some of the main OpenStack components and how they are integrated into the Nutanix platform.


Nova is the compute engine and scheduler for the OpenStack platform. In the Nutanix OpenStack solution each OVM acts as a compute host and every Nutanix cluster will act as a single hypervisor host eligible for scheduling OpenStack instances. The OVM runs the Nova-compute service.

You can view the Nova services using the OpenStack portal under 'Admin'->'System'->'System Information'->'Compute Services'.

The figure shows the Nova services, host and state:

OpenStack Nova Services
OpenStack Nova Services

The Nova scheduler decides which compute host (i.e. OVM) to place the instances based upon the selected availability zone. These requests will be sent to the selected OVM which will forward the request to the target host's (i.e. Nutanix cluster) Acropolis scheduler. The Acropolis scheduler will then determine optimal node placement within the cluster. Individual nodes within a cluster are not exposed to OpenStack.

You can view the compute and hypervisor hosts using the OpenStack portal under 'Admin'->'System'->'Hypervisors'.

The figure shows the OVM as the compute host:

OpenStack Compute Host
OpenStack Compute Host

The figure shows the Nutanix cluster as the hypervisor host:

OpenStack Hypervisor Host
OpenStack Hypervisor Host

As you can see from the previous image the full cluster resources are seen in a single hypervisor host.


Swift is an object store used to store and retrieve files. This is currently only leveraged for backup / restore of snapshots and images.


Cinder is OpenStack's volume component for exposing iSCSI targets. Cinder leverages the Acropolis Volumes API in the Nutanix solution. These volumes are attached to the instance(s) directly as block devices (as compared to in-guest).

You can view the Cinder services using the OpenStack portal under 'Admin'->'System'->'System Information'->'Block Storage Services'.

The figure shows the Cinder services, host and state:

OpenStack Cinder Services
OpenStack Cinder Services
Glance / Image Repo

Glance is the image store for OpenStack and shows the available images for provisioning. Images can include ISOs, disks, and snapshots.

The Image Repo is the repository storing available images published by Glance. These can be located within the Nutanix environment or by an external source. When the images are hosted on the Nutanix platform, they will be published to the OpenStack controller via Glance on the OVM. In cases where the Image Repo exists only on an external source, Glance will be hosted by the OpenStack Controller and the Image Cache will be leveraged on the Nutanix cluster(s).

Glance is enabled on a per-cluster basis and will always exist with the Image Repo. When Glance is enabled on multiple clusters the Image Repo will span those clusters and images created via the OpenStack Portal will be propagated to all clusters running Glance. Those clusters not hosting Glance will cache the images locally using the Image Cache.

Pro tip

For larger deployments Glance should run on at least two Nutanix clusters per site. This will provide Image Repo HA in the case of a cluster outage and ensure the images will always be available when not in the Image Cache.

When external sources host the Image Repo / Glance, Nova will be responsible for handling data movement from the external source to the target Nutanix cluster(s). In this case the Image Cache will be leveraged on the target Nutanix cluster(s) to cache the image locally for any subsequent provisioning requests for the image.


Neutron is the networking component of OpenStack and responsible for network configuration. The OVM allows network CRUD operations to be performed by the OpenStack portal and will then make the required changes in Acropolis.

You can view the Neutron services using the OpenStack portal under 'Admin'->'System'->'System Information'->'Network Agents'.

The figure shows the Neutron services, host and state:

OpenStack Neutron Services + Currently only Local and VLAN network types are supported.
OpenStack Neutron Services

Neutron will assign IP addresses to instances when they are booted. In this case Acropolis will receive a desired IP address for the VM which will be allocated. When the VM performs a DHCP request the Acropolis Leader will respond to the DHCP request on a private VXLAN as usual with AHV.

Supported Network Types

Currently only Local and VLAN network types are supported.

The Keystone and Horizon components run in an OpenStack Controller which interfaces with the OVM. The OVM(s) have an OpenStack Driver which is responsible for translating the OpenStack API calls into native Acropolis API calls.

Design and Deployment

For large scale cloud deployments it is important to leverage a delivery topology that will be distributed and meet the requirements of the end-users while providing flexibility and locality.

OpenStack leverages the following high-level constructs which are defined below:

  • Region
    • A geographic landmass or area where multiple Availability Zones (sites) are located. These can include regions like US-Northwest or US-West.
  • Availability Zone (AZ)
    • A specific site or datacenter location where cloud services are hosted. These can include sites like US-Northwest-1 or US-West-1.
  • Host Aggregate
    • A group of compute hosts, can be a row, aisle or equivalent to the site / AZ.
  • Compute Host
    • An OVM which is running the nova-compute service.
  • Hypervisor Host
    • A Nutanix cluster (seen as a single host).

The figure shows the high-level relationship of the constructs:

OpenStack - Deployment Layout
OpenStack - Deployment Layout

The figure shows an example application of the constructs:

OpenStack - Deployment Layout - Example
OpenStack - Deployment Layout - Example

You can view and manage hosts, host aggregates and availability zones using the OpenStack portal under 'Admin'->'System'->'Host Aggregates'.

The figure shows the host aggregates, availability zones and hosts:

OpenStack Host Aggregates and Availability Zones
OpenStack Host Aggregates and Availability Zones
Services Design and Scaling

For larger deployments it is recommended to have multiple OVMs connected to the OpenStack Controller abstracted by a load balancer. This allows for HA and of the OVMs as well as distribution of transactions. The OVM(s) don't contain any state information allowing them to be scaled.

The figure shows an example of scaling OVMs for a single site:

OpenStack - OVM Load Balancing
OpenStack - OVM Load Balancing

One method to achieve this for the OVM(s) is using Keepalived and HAproxy.

For environments spanning multiple sites the OpenStack Controller will talk to multiple OVMs across sites.

The figure shows an example of the deployment across multiple sites:

OpenStack - Multi-Site
OpenStack - Multi-Site

The OVM can be deployed as a standalone RPM on a CentOS / Redhat distro or as a full VM. The OVM can be deployed on any platform (Nutanix or non-Nutanix) as long as it has network connectivity to the OpenStack Controller and Nutanix Cluster(s).

The VM(s) for the OVM can be deployed on a Nutanix AHV cluster using the following steps. If the OVM is already deployed you can skip past the VM creation steps. You can use the full OVM image or use an existing CentOS / Redhat VM image.

First we will import the provided OVM disk image to Nutanix cluster. This can be done by copying the disk image over using SCP or by specifying a URL to copy the file from. We will cover importing this using the Images API. Note: It is possible to deploy this VM anywhere, not necessarily on a Nutanix cluster.

To import the disk image using Images API, run the following command:

image.create <IMAGE_NAME> source_url=<SOURCE_URL> container=<CONTAINER_NAME>

Next create the Acropolis VM for the OVM by running the following ACLI commands on any CVM:

vm.create <VM_NAME> num_vcpus=2 memory=16G
vm.disk_create <VM_NAME> clone_from_image=<IMAGE_NAME>
vm.nic_create <VM_NAME> network=<NETWORK_NAME>
vm.on <VM_NAME>

Once the VM(s) have been created and powered on, SSH to the OVM(s) using the provided credentials.


Help txt can be displayed by running the following command on the OVM:

ovmctl --help

The OVM supports two deployment modes:

  • OVM-allinone
    • OVM includes all Acropolis drivers and OpenStack controller
  • OVM-services
    • OVM includes all Acropolis drivers and communicates with external/remote OpenStack controller

Both deployment modes will be covered in the following sections. You can use in any mode and also switch between modes.


The following steps cover the OVM-allinone deployment. Start by SSHing to the OVM(s) to run the following commands.

# Register OpenStack Driver service
ovmctl --add ovm --name <OVM_NAME> --ip <OVM_IP> --netmask <NET_MASK> --gateway <DEFAULT_GW> --domain <DOMAIN> --nameserver <DNS>

# Register OpenStack Controller
ovmctl --add controller --name <OVM_NAME> --ip <OVM_IP>

# Register Nutanix cluster(s) (run for each cluster to add)
ovmctl --add cluster --name <CLUSTER_NAME> --ip <CLUSTER_IP> --username <PRISM_USER> --password <PRISM_PASSWORD>

The following values are used as defaults:
Number of VCPUs per core = 4
Container name = default
Image cache = disabled, Image cache URL = None

Next we'll verify the configuration using the following command:

ovmctl --show

At this point everything should be up and running, enjoy.


The following steps cover the OVM-services deployment. Start by SSHing to the OVM(s) to run the following commands.

# Register OpenStack Driver service
ovmctl --add ovm --name <OVM_NAME> --ip <OVM_IP>

# Register OpenStack Controller
ovmctl --add controller --name <OS_CONTROLLER_NAME> --ip <OS_CONTROLLER_IP> --username <OS_CONTROLLER_USERNAME> --password <OS_CONTROLLER_PASSWORD>

The following values are used as defaults:
Authentication: auth_strategy = keystone, auth_region = RegionOne
auth_tenant = services, auth_password = admin
Database: db_{nova,cinder,glance,neutron} = mysql, db_{nova,cinder,glance,neutron}_password = admin
RPC: rpc_backend = rabbit, rpc_username = guest, rpc_password = guest

# Register Nutanix cluster(s) (run for each cluster to add)
ovmctl --add cluster --name <CLUSTER_NAME> --ip <CLUSTER_IP> --username <PRISM_USER> --password <PRISM_PASSWORD>

The following values are used as defaults:
Number of VCPUs per core = 4
Container name = default
Image cache = disabled, Image cache URL = None

If non-default passwords were used for the OpenStack controller deployment, we'll need to update those:

# Update controller passwords (if non-default are used)
ovmctl --update controller --name <OS_CONTROLLER_NAME> --auth_nova_password <> --auth_glance_password <> --auth_neutron_password <> --auth_cinder_password <> --db_nova_password <> --db_glance_password <> --db_neutron_password <> --db_cinder_password <>

Next we'll verify the configuration using the following command:

ovmctl --show

Now that the OVM has been configured, we'll configure the OpenStack Controller to know about the Glance and Neutron endpoints.

Log in to the OpenStack controller and enter the keystonerc_admin source:

# enter keystonerc_admin
source ./keystonerc_admin

First we will delete the existing endpoint for Glance that is pointing to the controller:

# Find old Glance endpoint id (port 9292)
keystone endpoint-list # Remove old keystone endpoint for Glance
keystone endpoint-delete <GLANCE_ENDPOINT_ID>

Next we will create the new Glance endpoint that will point to the OVM:

# Find Glance service id
keystone service-list | grep glance
# Will look similar to the following:
| 9e539e8dee264dd9a086677427434982 | glance | image |

# Add Keystone endpoint for Glance
keystone endpoint-create \
--service-id <GLANCE_SERVICE_ID> \
--publicurl http://<OVM_IP>:9292 \
--internalurl http://<OVM_IP>:9292 \
--region <REGION_NAME> \
--adminurl http://<OVM_IP>:9292

Next we will delete the existing endpoint for Neutron that is pointing to the controller:

# Find old Neutron endpoint id (port 9696)
keystone endpoint-list # Remove old keystone endpoint for Neutron
keystone endpoint-delete <NEUTRON_ENDPOINT_ID>

Next we will create the new Neutron endpoint that will point to the OVM:

# Find Neutron service id
keystone service-list | grep neutron
# Will look similar to the following:
| f4c4266142c742a78b330f8bafe5e49e | neutron | network |

# Add Keystone endpoint for Neutron
keystone endpoint-create \
--service-id <NEUTRON_SERVICE_ID> \
--publicurl http://<OVM_IP>:9696 \
--internalurl http://<OVM_IP>:9696 \
--region <REGION_NAME> \
--adminurl http://<OVM_IP>:9696

After the endpoints have been created we will update the Nova and Cinder configuration files with new OVM IP of Glance host.

First we will edit Nova.conf which is located at /etc/nova/nova.conf and edit the following lines:

# Default glance hostname or IP address (string value)

# Default glance port (integer value)
# A list of the glance api servers available to nova. Prefix
# with https:// for ssl-based glance api servers.
# ([hostname|ip]:port) (list value)

Now we will disable nova-compute on the OpenStack controller (if not already):

systemctl disable openstack-nova-compute.service
systemctl stop openstack-nova-compute.service
service openstack-nova-compute stop

Next we will edit Cinder.conf which is located at /etc/cinder/cinder.conf and edit the following items:

# Default glance host name or IP (string value)
# Default glance port (integer value)
# A list of the glance API servers available to cinder
# ([hostname|ip]:port) (list value)

We will also comment out lvm enabled backends as those will not be leveraged:

# Comment out the following lines in cinder.conf

Now we will disable cinder volume on the OpenStack controller (if not already):

systemctl disable openstack-cinder-volume.service
systemctl stop openstack-cinder-volume.service
service openstack-cinder-volume stop

Now we will disable glance-image on the OpenStack controller (if not already):

systemctl disable openstack-glance-api.service
systemctl disable openstack-glance-registry.service
systemctl stop openstack-glance-api.service
systemctl stop openstack-glance-registry.service
service openstack-glance-api stop
service openstack-glance-registry stop

After the files have been edited we will restart the Nova and Cinder services to take the new configuration settings. The services can be restarted with the following commands below or by running the scripts which are available for download.

# Restart Nova services
service openstack-nova-api restart
service openstack-nova-consoleauth restart
service openstack-nova-scheduler restart
service openstack-nova-conductor restart
service openstack-nova-cert restart
service openstack-nova-novncproxy restart

# OR you can also use the script which can be downloaded as part of the helper tools:

# Restart Cinder
service openstack-cinder-api restart
service openstack-cinder-scheduler restart
service openstack-cinder-backup restart

# OR you can also use the script which can be downloaded as part of the helper tools:

Troubleshooting & Advanced Administration
Key log locations
Component Key Log Location(s)
Keystone /var/log/keystone/keystone.log
Horizon /var/log/horizon/horizon.log
Nova /var/log/nova/nova-api.log
Swift /var/log/swift/swift.log
Cinder /var/log/cinder/api.log
Glance /var/log/glance/api.log
Neutron /var/log/neutron/server.log

Logs marked with * are on the OVM only.

Pro tip

Check NTP if a service is seen as state 'down' in OpenStack Manager (Admin UI or CLI) even though the service is running in the OVM. Many services have a requirement for time to be in sync between the OpenStack Controller and OVM.

Command Reference

Load Keystone source (perform before running other commands)

source keystonerc_admin

List Keystone services

keystone service-list

List Keystone endpoints

keystone endpoint-list

Create Keystone endpoint

keystone endpoint-create \
--service-id=<SERVICE_ID> \
--publicurl=http://<IP:PORT> \
--internalurl=http://<IP:PORT> \
--region=<REGION_NAME> \

List Nova instances

nova list

Show instance details

nova show <INSTANCE_NAME>

List Nova hypervisor hosts

nova hypervisor-list

Show hypervisor host details

nova hypervisor-show <HOST_ID>

List Glance images

glance image-list

Show Glance image details

glance image-show <IMAGE_ID>


Thank you for reading The Nutanix Bible!  Stay tuned for many more upcoming updates and enjoy the Nutanix platform!