Nutanix Move – Offline / Public Repo with non VM-To-Move Network access.

Over the a couple of days we have been migrating some vm:s from a customers onprem vSphere 6.5 environment to Nutanix AHV. And we’ve run in to some issues.

EDIT 2023-02-06:
I have publiced a Nutanix Move Repo on Git for you to just clone. Se this post

First, let’s discuss the networking part of Nutanix move. Our Nutanix-Move appliance is placed in a public DMZ at our location without internet access, for us to gain access to our customers VMware environment we have to setup a VPN between our two locations.

If you review the official documentation from nutanix on witch ports that have to be open between the esxi host(s) and move, and from move to AHV cluster Below ports to open is suggested in the KB:

Firewall Port Requirements

We set up the appropriate rules from Move to ESXI source, and from move to AHV cluster, using only the Cluster Virtual IP.

We successfully added the ESXI as source and the AVH cluster as destination.

Next step was to migrate a test VM from source to destination. So we started by adding a Migration Job in Nutanix Move. The detailed steps to create a project in nutanix move are not in the scope for this article.

First stop we ran in to was that we where unable to use the Automatic feature because of ESXI limitations to insert the drivers trough VMware-tools, so we chose to run the Manual prepare of each individual VM trough Powershell.

So, we clicked the manual option in move and got the powershell-script shown below (note, the ip numbers are figurated):

powershell.exe -command {Set-ExecutionPolicy Bypass -Scope Process -Force;
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }; 
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor 
[Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12; $scriptPath = 
(New-Object System.Net.WebClient).DownloadString('https://' + '192.168.0.2' + 
'/resources/uvm/win/esx_setup_uvm.ps1'); $retainIP = $true; $installVirtio = $true; 
$setSanPolicy = $true; $uninstallVMwareTools = $true; Invoke-Command -ScriptBlock 
([scriptblock]::Create($scriptPath)) -ArgumentList 
'192.168.0.2',$retainIP,$installVirtio,$setSanPolicy,$uninstallVMwareTools}

When we tried to run this script, we found that the Virtual Machine was not able to download the artifacts/payload containing IPretain-scrips and VirtIO drivers etc, because there was no HTTPs-access(tcp443) to our move-server. Because of security restrictions from our networking team, they did not allow communication from the VMs/ESXI servers at the customers site to our Move VM at our premise.

So, how can you solve this problem?. There are two possible solutions.

1. Manually install the VirtIO-msi driver on the Guest-VM to be migrated.
   Pros: Easy, fast
   Cons: You loose the features for retain IP, SAN Policy, and Removal of VMware-tools (Move 4.0 feature).
2. Set up a repo witch is accessible from the customers location, either a local repo in the customers network, or a public repo accessible over the internet.
   Pros: Can still use the Retain IP, SAN Policy and removal of VMware tools. Can be reused for other projects if set up for access over the internet.
   Cons: Complex to set up first time.

We chose to go for option two, because we need the features mentioned above.

Solution

So, now off to the solution.

First, we have to reverse engineer the powershell-script that you get in the move UI. As you can se in the code snippet in the previous section, the script downloads a script from the move VM named esx_setup_uvm.ps1.

DownloadString(‘https://’ + ‘192.168.0.2’ + ‘/resources/uvm/win/esx_setup_uvm.ps1’)

So the first step was to download that from a jumpstation that have https access to the move-vm by enetering the url link https://move-ip/resources/uvm/win/esx_setup_uvm.ps1.

Open the script in your favorite Code-based editor.

In the script, we can follow the steps that the script uses to download various artifacts (payloads), and also we the URLs it’s using to download those artifacts.

If we go to line 271 we se the below output:

As we can see here, the script downloads the different files from the URLS below from the Nutanix Move VM:

https://move-ip/resources/Nutanix-VirtIO-latest-stable-x86.msi
https://move-ip/resources/Nutanix-VirtIO-latest-stable.msi
https://move-ip/resources/Nutanix-VirtIO-sha1.cer
https://move-ip/resources/Nutanix-VirtIO-sha2.cer
https://move-ip/resources/Nutanix-VirtIO-sha1-balloon-driver.cer
https://move-ip/resources/Nutanix-VirtIO-scsi-passthrough-driver.cer
https://move-ip/resources/setSANPolicy.bat
https://move-ip/resources/uvm/win/cleanUp/cleanup_installation.ps1
https://move-ip/resources/uvm/win/cleanUp/scheduleTargetCleanup.bat
https://move-ip/resources/retainIP.bat
https://move-ip/resources/wmi-net-util.exe
https://move-ip/resources/TaskRetainIPOnStartConfig.xml
https://move-ip/resources/uninstallVMwareTools.bat

So, once again, off to a jumbox with HTTPs access to the Nutanix Move VM, and i used chrome do simply download those files from the Nutanix VM.

After that, i set up a Ubuntu 20.04 VM with Apache 2 (detailed steps for this is not within the scope of this post)
Install Apache on Ubuntu
Enable SSL on Apache

We uploaded the files to the Ubuntu Server with the the exact folder structure of the Nutanix Move VM. Then we publish the server using HTTPs on the internet, and limited the firewall rules to only allow the customers on-prem public ip as source.

From the source VMs we’re trying to migrate we get the following result:

So now we run a updated version of the Powershell script that move give us during the Option of Manual Preparation (note the IP numbers 90.90.90.90):

powershell.exe -command {Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12; $scriptPath = (New-Object System.Net.WebClient).DownloadString('https://' + '90.90.90.90' + '/resources/uvm/win/esx_setup_uvm.ps1'); $retainIP = $true; $installVirtio = $true; $setSanPolicy = $true; $uninstallVMwareTools = $true; Invoke-Command -ScriptBlock ([scriptblock]::Create($scriptPath)) -ArgumentList '90.90.90.90',$retainIP,$installVirtio,$setSanPolicy,$uninstallVMwareTools}

Now the script runs as expected and downloads the payload, installs the scripts, and the VirtIO drivers

Now we return to Nutanix Move Web interface. And start the migration.

During migration, we found that move also tries to communicate with each individual CVM. Se below log output

<133>date=2021-05-07 time=11:56:55 devname="<firewallname>" devid="xxxxxxxxx" 
logid="0000000013" type="traffic" subtype="forward" level="notice" vd="<virtual domain>" 
eventtime=1620381415670353879 tz="+0200" srcip=<MoveIP> srcport=757 srcintf="<source-vlan>" 
srcintfrole="undefined" dstip=<CVM IP> dstport=2049 dstintf=<destination-vlan>" 
dstintfrole="undefined" sessionid=3917345744 proto=6 action="deny" policyid=0 
policytype="policy" service="NFS" dstcountry="Reserved" srccountry="Sweden" trandisp="noop" 
duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 
crlevel="high" mastersrcmac="00:00:00:00:00:00" srcmac="00:00:00:00:00:00" srcserver=0

As a result the Move VM was stuck at “Mounting Container”. We added all our CVMs in the rules allowing communication on port 2049 from the Move VM to The AHV cluster.

Now the migration started off as expected:

Conclusion

This was the best way to solve this kind of problem when you don’t have the opportunity to access the move VM directly from within the source VMs, we can use it for other customers/projects.

One thing i was looking for during this project, was a published repo somewhere online or where the files reside on the Move appliance. Any sudgestions where to download the Full repo are much appreciated

Happy migrations of your VMs :)