Physical Node Removal - Nutanix Cluster

 

Permanently remove a physical node / host from a Nutanix cluster are two scenarios in node removal.

1. Permanently Removing an online node
2. Removing an offline / not-responsive node

At least 30% free space must be available to avoid filling any disk beyond 95%. 

Some Points to consider before initiating node removal:

1. Sufficient Disk space available on other nodes in the cluster
2. User Virtual Machine relocation (if required)
3. Any software upgrade should not be running
4. Checklist on verifying cluster health status
5. Data resiliency is “OK” (green) in Prism

1. There is a chance that DRS will place VM on the node you are trying to evacuate so you may want to change the setting to manual or partial
2. Run a complete “ncc report” either from prism or CVM cli:                                 
3. ncc health_checks run_all

Depending on the size of data, node removal can be lengthy process, which involves relocating data from the node to other healthy nodes in the cluster. Node removal also removes entries from the cluster configuration before permanently marking the node removed.

You can initiate Node Removal process from within Prism > Hardware > Select Host > Remove Host or from ncli.

Node removal process, when initiated, factors in cluster load and runs on a low priority - ensuring no impact to the cluster while relocating data to different nodes across the cluster.

To remove a node from a Nutanix Cluster, please follow the official Nutanix KB:

Removing Unreachable or Powered-off Nodes From Nutanix Clusters

You want to unmount the datastore from the node and you will find that the datastore will not unmount. Reason for this is that the datastore is being used a heartbeat datastore. You can just disable HA on the cluster. There are several ways you can do the unmounting of the datastores. I prefer to do it via PRISM.

• Go to Storage > Table and select the datastore. Click update
• Select Mount/Unmount on the following ESX hosts and deselect host in question.
  
  
  
  
• Click OK when prompted.
  

Now the datastores are unmounted you can proceed with removing the nodes from the cluster

• Go to Hardware > Table and select your host.Click remove host.
• Click OK when prompted
  
  
  
  
• The removal process will start and you can follow progress under tasks. This is a time consuming process.
  
  
  
  
• Once the process is complete you should see a decrease in available dataspace and the host will be no longer visible.

or NCLI 

Evicting the node

---

To evict the node, you just execute an ncli command with the node ID confirmed in earlier steps.

ncli host remove-start id={node id}

You can check on the status of the job with another ncli command

ncli host get-remove-status

If you want a more in-depth view of the tasks going on, there is a more specific ncli command (this technically covers any tasks at hand, not just removal)

ncli progress-monitor list

Prism will also reflect the running tasks