Adventures in Upgrading an x3650 M2 from VMWare ESXi 6.5 to 6.7

I'm trying to squeeze out a bit more life from some IBM x3650 and x3550 M2's. These systems are quite old, but still pack enough power to run meaningful workloads when the upgrades are maxed out. There is one last upgrade you can do that will allow you to unlock the ability to run VMware ESXi 6.7 on a IBM x3650/x3550 M2.

The Plan

It's a multi-step process to get from point A to B, but not difficult. It took about 2 hours worth of effort including research, product purchases, and the troubleshooting. With this guide you can save some time on the research and troubleshooting.

Upgrade system firmware
Replace CPU's
Upgrade VMware ESXi

Update Firmware

It goes without saying that you should be running the latest firmware on these systems. The last time a firmware update was released for these systems was December 2013. Below are the versions of firmware that worked when I did my CPU upgrade:

IMM YUOOG2C-1.42 12/16/2013
UEFI D6E161A-1.19 11/29/2013
DSA DSYTC5C-9.42 11/05/2013

IBM (and Lenovo) make the update process effortless. Before Lenovo took over we would usually download a tool called IBM Bootable Media Creator, BoMC for short. The "Bo" in that acronym was probably used to not conflict with the "BMC" acronym witch is for the baseboard management controller on the system itself.

IBM has discontinued maintaining that tool. The last version of that tool is 9.64 and will not work with any modern versions of Windows. I also doubt it would be able to download the updates since URL's have changed in the last couple of years.

Fortunately both IBM and Lenovo have somewhat of a better track record at supporting older products. Lenovo now publishes a tool called "Lenovo XClarity Essentials Bootable Media Creator". As of this writing, the link to obtain that tool can be found at https://sysmgt.lenovofiles.com/help/topic/bomc_lenovo/bomc_t_downloading_bomc.html. If the link is broken, use a search engine to find the latest one.

Lenovo Bootable Media Creator Progress Screen

Download and run the tool to create an update ISO image that you can mount using the remote media tool. Power on the server, boot from the virtual CD-ROM, follow the prompts, and let the system update itself. The amount of time this takes depends on how many items need updating. It took my system roughly 30 minutes to go through it.

Replacing the CPU

VMware ESXi will not run on the stock Xeon 5500 series CPU's that where shipped with the M2's. The minimum that will be accepted by the installer is a 5600 series CPU. It just so happens that the socket type is the same for both CPU's and the newer UEFI firmware allows the M2's to recognize the 5600 series CPU's.

Upgrading the CPU is a fairly simple process if you've worked with computer hardware in the past. Get yourself some thermal paste and a couple of new Xeon 56xx series CPU's. The existing heat sinks and system fans provided by IBM will work just fine. Of course this is assuming you have a properly cooled environment. The new processors do use slightly more power and therefor let own more heat.

For my upgrade I decided to go all the way and get two Intel Xeon X5670 @ 2.93GHz CPU for each of my M2's.

For thermal compound I use nothing but the best, Noctua NT-H2. It includes cleaning wipes. It may seem superficial, but it extremely convenient to just have them. You should have some napkins or paper towel handy to dry things off.

As with all the newer IBM (and now Lenovo) hardware, the designs are well thought out. Just pop the cover, pull a few levers, and swap out the CPU's. Clean off the thermal compound from the old CPU's and heatsinks. Also wipe the new CPU surfaces to get a nice clean seal. You only need a single cleaning wipe per server if you went with the Noctua option. Use a single sheet of paper towel to dry the surfaces of the heatsink and CPU's.

Of course be careful! Pay attention, take it slow, and be patient. Intel in their wisdom have shifted the designs so the motherboard CPU socket is the most delicate piece. Damaging the pins on the CPU socket mean that you have to replace the whole board or server.

TIP: Remove the old CPUs first, then clean off the old thermal compound far away from the server.

After placing your new CPU's into the socket, apply some thermal compound. Follow the instructions provided by the paste you purchased. Mine said to use 5 dots with the center dot being slightly larger. Re-insert the heatsinks as illustrated in the picture printed on them.

Reinstall the cover, plug it in and power up your system. If all goes well it should boot up as it always had, but this time with a more capable CPU! There is no need to change any settings in the CMOS. I just let me system boot up to VMWare ESXi 6.5.

Upgrade ESXi to 6.7

Now for the moment of truth, can we upgrade? Of course we can. There are many ways to upgrade ESXi depending on your use case and setup. I can count 4 ways:

New Install
Upgrade from Bootable ISO
Upgrade from vCenter or Update Manager
Upgrade from the CLI using Zip file

Since I'm already on 6.5 I originally chose to go with option 3 and try to use the automatic unattended method with vCenter. However was getting an error message when trying to import the ISO image.

The error I was getting was The uploaded upgrade package cannot be used with VMware. Usually this works flawlessly, I had already done a 6.5 to 6.7 upgrade using the same approach.

Turns out I had recently upgraded to vCenter 7.0 and VMWare requires that the ESXi major and minor version match with the vCenter version. I had the same problem when I was on vCenter 6.7 and tried to upgrade a 6.0 system to 6.5. It's a terrible design choice from VMware.

So I go with the next option, the tried and true method of booting of the ISO using the IMM's virtual media manager. I've done it this way countless times. It's usually flawless, except for this time.

The upgrade process errored at 78% with a cryptic message about not being able to read a file on a datastore, It was "Errorno 51" to be exact.

I would like to point out that none of these problems are in any way related to the CPU upgrade. I use a SanDisk Cruzer Fit for the VMWare install. These fit perfectly inside the IBM x3550/3650's internal USB port, and it's likely that there's some bitrot occurring. I ran into the same situation when I tried to upgrade a system running 6.0 to 6.5. Interestingly enough this is not the first or second time I've encountered issues on my flash drives. Maybe something going on with increased solar flare activity in recent months (as of this writing) that may have knocked a few electrons out of place. Or perhaps these servers have just been sitting on for months with very little read/writes to the flash storage, and that's what causes bitrot. It's all very fascinating.

Re-attempting the upgrade is futile, it won't provide you with that option anymore. Again, VMWare... design choices. Even though the upgrade said it was 78% complete, it really didn't "do" anything to the system. I simply rebooted back to 6.5 and will attempt an upgrade from the CLI using the ZIP file.

ESXi CLI Upgrade

To upgrade using the ZIP file, enable the SSH service on your ESXi server and log into the CLI. Place the ESXi 6.7 on a datastore that the system has access to and issue the following comamnd:

esxcli software vib update -d /path/to/datastore/esxi-zip-filename.zip

Since I use a Lenovo image there are some extra parameters that need to be applied. Without them I was getting the following error:

[root@oberon:~] esxcli software vib update -d "/vmfs/volumes/9c0c4e20-86a94ae4/Other Software/VMWare/VMware-ESXi-6.7.0.update03-19195723-LNV-202204121.zip"
 [DependencyError]
 VIB VMware_bootbank_esx-base_6.7.0-3.163.19195723 requires esx-update >= 6.7.0-3.163, but the requirement cannot be satisfied within the ImageProfile.
 VIB VMware_bootbank_esx-base_6.7.0-3.163.19195723 requires esx-update << 6.7.0-3.164, but the requirement cannot be satisfied within the ImageProfile.
 Please refer to the log file for more details.

After some research I learned that I needed to include a "profile" name in my upgrade command. I can determine the profile name to use with the esxcli software sources profile list command, using the ZIP file as the value for the "-d" parameter.

esxcli software sources profile list -d "/vmfs/volumes/9c0c4e20-86a94ae4/Other Software/VMWare/VMware-ESXi-6.7.0.update03-19195723-LNV-202204121.zip"
Name                                Vendor  Acceptance Level
----------------------------------  ------  ----------------
Lenovo_ESXi6.7u3-19195723_20220412  Lenovo  PartnerSupported

After that, I can retry the upgrade with an extra "-p" option where I can specify the profile name.

esxcli software profile update -p Lenovo_ESXi6.7u3-19195723_20220412 -d "/vmfs/volumes/9c0c4e20-86a94ae4/Other Software/VMWare/VMware-ESXi-6.7.0.update03-19195723-LNV-202204121.zip"

If all goes well, you'll get a message after a few minutes telling you the upgrade was successful and that you need to reboot. Yet, in my case all did NOT go well.

Broken Pipe Error

It's almost as if this server is fighting against me and does not want to be upgraded. My upgrade failed with the following error. I think it might be the same issue as when I attempted an ISO upgrade.

[root@oberon:~] esxcli software profile update -p Lenovo_ESXi6.7u3-19195723_20220412 -d "/vmfs/volumes/9c0c4e20-86a94ae4/Other Software/VMWare/VMware-ESXi-6.7.0.update03-19195723-LNV-202204121.zip"
 [InstallationError]
 [Errno 32] Broken pipe
       vibs = VMware_locker_tools-light_11.3.5.18557794-18812553
 Please refer to the log file for more details.

Indeed, some how something somewhere went wrong. I figured, why not retry. However it's not so easy as just running the command again. There is some clean up to do.

If you do not clean up first, you'll see an error like this one.

[root@oberon:~] esxcli software profile update -p Lenovo_ESXi6.7u3-19195723_20220412 -d "/vmfs/volumes/9c0c4e20-86a94ae4/Other Software/VMWare/VMware-ESXi-6.7.0.update03-19195723-LNV-202204121.zip"
 [StatelessError]
 The transaction is not supported: VIB EMU_bootbank_elxiscsi_12.0.1188.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_hid-hid_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_nvmxnet3_2.0.0.29-1vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.3.104.16075168 cannot be live installed. VIB VMW_bootbank_sata-sata-svw_2.3-3vmw.670.0.0.8169922 cannot be live installed. VIB ELX_bootbank_elx-esx-libelxima-8169922.so_12.0.1188.0-03 cannot be live installed. VIB VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-bnx2x_1.78.80.v60.12-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-vmklinux-9-2-2-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922 cannot be live installed. VIB QLC_bootbank_qfle3f_1.0.68.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ata-pata-via_0.3.3-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_nenic_1.0.29.0-1vmw.670.3.73.14320388 cannot be live installed. VIB MEL_bootbank_nmlx5-rdma_4.17.71.1-1OEM.670.0.0.8169922 cannot be live installed. VIB MEL_bootbank_nmlx5-core_4.17.71.1-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-ips_7.12.05-4vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.670.0.0.8169922 cannot be live installed. VIB QLC_bootbank_qedi_2.19.59.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_cpu-microcode_6.7.0-3.155.18812553 cannot be live installed. VIB VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_nfnic_4.0.0.44-0vmw.670.3.104.16075168 cannot be live installed. VIB VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_ntg3_4.1.5.0-0vmw.670.3.116.16713306 cannot be live installed. VIB EMU_bootbank_elxnet_12.0.1216.4-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-libfc-9-2-2-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-libfcoe-9-2-1-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-libfc-9-2-1-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-libfcoe-9-2-2-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_esx-update_6.7.0-3.163.19195723 cannot be live installed. VIB VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-igb_5.0.5.1.1-5vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-libata-9-2-1-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-libata-9-2-2-0_6.7.0-0.0.8169922 cannot be live installed. VIB MSCC_bootbank_smartpqi_67.4252.0.103-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_sata-ahci_3.0-26vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_block-cciss_3.6.14-10vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-aic79xx_3.1-6vmw.670.0.0.8169922 cannot be live installed. VIB QLC_bootbank_qfle3_1.0.84.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-enic_2.1.2.38-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.3.112.16701467 cannot be live installed. VIB VMW_bootbank_net-forcedeth_0.61-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_vmw-ahci_2.0.7-2vmw.670.3.143.17700523 cannot be live installed. VIB VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB INT_bootbank_ixgben_1.12.2.0-1OEM.670.0.0.8169922 cannot be live installed. VIB INT_bootbank_igbn_1.8.2.0-1OEM.670.0.0.8169922 cannot be live installed. VIB INT_bootbank_icen_1.7.3.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922 cannot be live installed. VIB BCM_bootbank_lsi-msgpt35_20.00.00.00-1OEM.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.670.2.48.13006603 cannot be live installed. VIB VMW_bootbank_sata-sata-nv_3.5-4vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_pvscsi_0.1-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_sfvmk_1.0.0.1003-7vmw.670.3.104.16075168 cannot be live installed. VIB VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.1.28.10302608 cannot be live installed. VIB QLC_bootbank_qcnic_1.0.26.0-1OEM.670.0.0.8169922 cannot be live installed. VIB QLC_bootbank_qedentv_3.11.41.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-cdc-ether_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_char-random_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB EMU_bootbank_lpfc_14.0.326.12-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.670.3.73.14320388 cannot be live installed. VIB VMW_bootbank_net-usbnet_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB MEL_bootbank_nmlx4-en_3.17.70.1-1OEM.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_vsan_6.7.0-3.163.19184739 cannot be live installed. VIB VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_nhpsa_2.0.22-3vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.670.0.0.8169922 cannot be live installed. VIB MEL_bootbank_nmlx4-rdma_3.17.70.1-1OEM.670.0.0.8169922 cannot be live installed. VIB MEL_bootbank_nmlx4-core_3.17.70.1-1OEM.670.0.0.8169922 cannot be live installed. VIB EMU_bootbank_brcmfcoe_12.0.1278.0-1OEM.670.0.0.8169922 cannot be live installed. VIB QLC_bootbank_qedentv-ens_3.11.42.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_mtip32xx-native_3.9.8-1vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_sata-sata-sil24_1.1-1vmw.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.2.48.13006603 cannot be live installed. VIB VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_nvmxnet3-ens_2.0.0.21-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_sata-sata-promise_2.12-3vmw.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_lsu-intel-vmd-plugin_1.0.0-2vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794 cannot be live installed. VIB LVO_bootbank_lnvcustomization_6.73-10.1 cannot be live installed. VIB VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_ne1000_0.8.4-2vmw.670.2.48.13006603 cannot be live installed. VIB VMW_bootbank_scsi-hpsa_6.0.0.84-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794 cannot be live installed. VIB VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_misc-drivers_6.7.0-2.48.13006603 cannot be live installed. VIB VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-e1000e_3.2.2.1-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_vmkfcoe_1.0.0.1-1vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_ata-pata-amd_0.3.10-3vmw.670.0.0.8169922 cannot be live installed. VIB INT_bootbank_i40en_1.16.3.0-1OEM.670.0.0.8169922 cannot be live installed. VIB QLC_bootbank_qfle3i_1.0.23.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-nx-nic_5.0.621-5vmw.670.0.0.8169922 cannot be live installed. VIB QLC_bootbank_qedf_2.2.62.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_sata-ata-piix_2.12-10vmw.670.0.0.8169922 cannot be live installed. VIB BCM_bootbank_lsi-mr3_7.720.04.00-1OEM.670.0.0.8169922 cannot be live installed. VIB BCM_bootbank_bnxtnet_219.0.29.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_esx-base_6.7.0-3.163.19195723 cannot be live installed. VIB VMW_bootbank_qflge_1.1.0.11-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_lpnic_11.4.59.0-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608 cannot be live installed. VIB VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794 cannot be live installed. VIB VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608 cannot be live installed. VIB QLC_bootbank_qedrntv_3.11.40.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_ohci-usb-ohci_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB Avago_bootbank_lsi-msgpt3_17.00.02.00-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.670.0.0.8169922 cannot be live installed. VIB BCM_bootbank_bnxtroce_219.0.13.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-mpt2sas_19.00.00.00-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_sata-sata-sil_2.3-4vmw.670.0.0.8169922 cannot be live installed. VIB Marvell_bootbank_qlnativefc_3.1.87.0-1OEM.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922 cannot be live installed. VIB VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.670.0.0.8169922 cannot be live installed. VIB VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.670.1.28.10302608 cannot be live installed. VIB VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ne1000_0.8.3-8vmw.650.2.75.10884925 cannot be removed live. VIB VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106 cannot be removed live. VIB VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607 cannot be removed live. VIB QLogic_bootbank_qlnativefc_2.1.81.0-1OEM.600.0.0.2768847 cannot be removed live. VIB VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106 cannot be removed live. VIB QLC_bootbank_qfle3f_1.0.68.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_nhpsa_2.0.22-3vmw.650.2.50.8294253 cannot be removed live. VIB EMU_bootbank_brcmfcoe_12.0.1115.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106 cannot be removed live. VIB MEL_bootbank_nmlx4-rdma_3.16.11.10-1OEM.650.0.0.4598673 cannot be removed live. VIB MEL_bootbank_nmlx4-core_3.16.11.10-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_vmw-ahci_1.1.6-1vmw.650.3.96.13932383 cannot be removed live. VIB VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ntg3_4.1.3.2-1vmw.650.2.75.10884925 cannot be removed live. VIB VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106 cannot be removed live. VIB QLC_bootbank_scsi-qedil_1.15.14.0-1OEM.600.0.0.2494585 cannot be removed live. VIB VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_nvme_1.2.2.28-1vmw.650.3.96.13932383 cannot be removed live. VIB VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106 cannot be removed live. VIB Broadcom_bootbank_lsi-mr3_7.710.07.00-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303 cannot be removed live. VIB QLC_bootbank_qedf_1.3.41.0-1OEM.600.0.0.2768847 cannot be removed live. VIB VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.650.2.79.11925212 cannot be removed live. VIB VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.650.2.79.11925212 cannot be removed live. VIB Broadcom_bootbank_lsi-msgpt35_11.00.00.00-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106 cannot be removed live. VIB QLC_bootbank_qfle3_1.0.84.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846 cannot be removed live. VIB VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106 cannot be removed live. VIB INT_bootbank_ixgben_1.7.20-1OEM.600.0.0.2768847 cannot be removed live. VIB VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106 cannot be removed live. VIB VMware_bootbank_vsan_6.5.0-3.111.15079806 cannot be removed live. VIB MEL_bootbank_nmlx5-core_4.16.14.2-1OEM.650.0.0.4598673 cannot be removed live. VIB MEL_bootbank_nmlx5-rdma_4.16.14.2-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_usbcore-usb_1.0-3vmw.650.2.50.8294253 cannot be removed live. VIB VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106 cannot be removed live. VIB LVO_bootbank_lnvcustomization_6.53-10.1 cannot be removed live. VIB VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607 cannot be removed live. VIB VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_nenic_1.0.29.0-1vmw.650.3.96.13932383 cannot be removed live. VIB INT_bootbank_i40en_1.8.6-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_misc-drivers_6.5.0-3.96.13932383 cannot be removed live. VIB VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106 cannot be removed live. VIB BCM_bootbank_bnxtnet_214.0.230.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106 cannot be removed live. VIB VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.650.3.96.13932383 cannot be removed live. VIB VMW_bootbank_smartpqi_1.0.1.553-28vmw.650.3.96.13932383 cannot be removed live. VIB Avago_bootbank_lsi-msgpt3_17.00.02.00-1OEM.650.0.0.4598673 cannot be removed live. VIB MEL_bootbank_nmlx4-en_3.16.11.10-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB INT_bootbank_igbn_1.4.10-1OEM.600.0.0.2768847 cannot be removed live. VIB VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106 cannot be removed live. VIB EMU_bootbank_elxiscsi_12.0.1108.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.650.3.96.13932383 cannot be removed live. VIB VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106 cannot be removed live. VIB QLC_bootbank_qfle3i_1.0.23.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMware_bootbank_esx-base_6.5.0-3.111.15177306 cannot be removed live. VIB VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.650.2.50.8294253 cannot be removed live. VIB VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106 cannot be removed live. VIB VMware_bootbank_cpu-microcode_6.5.0-3.108.14990892 cannot be removed live. VIB QLC_bootbank_qedentv_3.11.16.0-1OEM.650.0.0.4598673 cannot be removed live. VIB EMU_bootbank_elxnet_12.0.1115.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303 cannot be removed live. VIB VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106 cannot be removed live. VIB EMU_bootbank_lpfc_12.4.242.0-1OEM.650.0.0.4598673 cannot be removed live. VIB VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106 cannot be removed live. VIB VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106 cannot be removed live. VIB VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.650.2.50.8294253 cannot be removed live. VIB VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106 cannot be removed live. VIB VMware_bootbank_esx-tboot_6.5.0-3.111.15177306 cannot be removed live. VIB VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106 cannot be removed live.
 Please refer to the log file for more details.

To clean up and allow the upgrade to restart I had to delete or (more safley) rename the /store/packages directory on the system.

mv /store/packages /store/packages.bad

Now I can re-run the upgrade and see that it was sucesfull.

[root@oberon:~] esxcli software profile update -p Lenovo_ESXi6.7u3-19195723_20220412 -d "/vmfs/volumes/9c0c4e20-86a94ae4/Other Software/VMWare/VMware-ESXi-6.7.0.update03-19195723-LNV-202204121.zip"
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: Avago_bootbank_lsi-msgpt3_17.00.02.00-1OEM.670.0.0.8169922, BCM_bootbank_bnxtnet_219.0.29.0-1OEM.670.0.0.8169922, BCM_bootbank_bnxtroce_219.0.13.0-1OEM.670.0.0.8169922, BCM_bootbank_lsi-mr3_7.720.04.00-1OEM.670.0.0.8169922, BCM_bootbank_lsi-msgpt35_20.00.00.00-1OEM.670.0.0.8169922, ELX_bootbank_elx-esx-libelxima-8169922.so_12.0.1188.0-03, EMU_bootbank_brcmfcoe_12.0.1278.0-1OEM.670.0.0.8169922, EMU_bootbank_elxiscsi_12.0.1188.0-1OEM.670.0.0.8169922, EMU_bootbank_elxnet_12.0.1216.4-1OEM.670.0.0.8169922, EMU_bootbank_lpfc_14.0.326.12-1OEM.670.0.0.8169922, INT_bootbank_i40en_1.16.3.0-1OEM.670.0.0.8169922, INT_bootbank_icen_1.7.3.0-1OEM.670.0.0.8169922, INT_bootbank_igbn_1.8.2.0-1OEM.670.0.0.8169922, INT_bootbank_ixgben_1.12.2.0-1OEM.670.0.0.8169922, LVO_bootbank_lnvcustomization_6.73-10.1, MEL_bootbank_nmlx4-core_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-en_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-rdma_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx5-core_4.17.71.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx5-rdma_4.17.71.1-1OEM.670.0.0.8169922, MSCC_bootbank_smartpqi_67.4252.0.103-1OEM.670.0.0.8169922, Marvell_bootbank_qlnativefc_3.1.87.0-1OEM.670.0.0.8169922, QLC_bootbank_qcnic_1.0.26.0-1OEM.670.0.0.8169922, QLC_bootbank_qedentv-ens_3.11.42.0-1OEM.670.0.0.8169922, QLC_bootbank_qedentv_3.11.41.0-1OEM.670.0.0.8169922, QLC_bootbank_qedf_2.2.62.0-1OEM.670.0.0.8169922, QLC_bootbank_qedi_2.19.59.0-1OEM.670.0.0.8169922, QLC_bootbank_qedrntv_3.11.40.0-1OEM.670.0.0.8169922, QLC_bootbank_qfle3_1.0.84.0-1OEM.670.0.0.8169922, QLC_bootbank_qfle3f_1.0.68.0-1OEM.670.0.0.8169922, QLC_bootbank_qfle3i_1.0.23.0-1OEM.670.0.0.8169922, VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.670.0.0.8169922, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.0.0.8169922, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-via_0.3.3-2vmw.670.0.0.8169922, VMW_bootbank_block-cciss_3.6.14-10vmw.670.0.0.8169922, VMW_bootbank_char-random_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.670.0.0.8169922, VMW_bootbank_hid-hid_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMW_bootbank_lpnic_11.4.59.0-1vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.670.3.73.14320388, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.670.0.0.8169922, VMW_bootbank_misc-drivers_6.7.0-2.48.13006603, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.670.1.28.10302608, VMW_bootbank_ne1000_0.8.4-2vmw.670.2.48.13006603, VMW_bootbank_nenic_1.0.29.0-1vmw.670.3.73.14320388, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.670.0.0.8169922, VMW_bootbank_net-bnx2x_1.78.80.v60.12-2vmw.670.0.0.8169922, VMW_bootbank_net-cdc-ether_1.0-3vmw.670.0.0.8169922, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.670.0.0.8169922, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.3.112.16701467, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.670.0.0.8169922, VMW_bootbank_net-enic_2.1.2.38-2vmw.670.0.0.8169922, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.670.0.0.8169922, VMW_bootbank_net-forcedeth_0.61-2vmw.670.0.0.8169922, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.670.0.0.8169922, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.670.0.0.8169922, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-nx-nic_5.0.621-5vmw.670.0.0.8169922, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.670.0.0.8169922, VMW_bootbank_net-usbnet_1.0-3vmw.670.0.0.8169922, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.3.104.16075168, VMW_bootbank_nfnic_4.0.0.44-0vmw.670.3.104.16075168, VMW_bootbank_nhpsa_2.0.22-3vmw.670.1.28.10302608, VMW_bootbank_ntg3_4.1.5.0-0vmw.670.3.116.16713306, VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794, VMW_bootbank_nvmxnet3-ens_2.0.0.21-1vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3_2.0.0.29-1vmw.670.1.28.10302608, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_pvscsi_0.1-2vmw.670.0.0.8169922, VMW_bootbank_qflge_1.1.0.11-1vmw.670.0.0.8169922, VMW_bootbank_sata-ahci_3.0-26vmw.670.0.0.8169922, VMW_bootbank_sata-ata-piix_2.12-10vmw.670.0.0.8169922, VMW_bootbank_sata-sata-nv_3.5-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-promise_2.12-3vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil24_1.1-1vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil_2.3-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-svw_2.3-3vmw.670.0.0.8169922, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.670.0.0.8169922, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.670.0.0.8169922, VMW_bootbank_scsi-aic79xx_3.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.670.0.0.8169922, VMW_bootbank_scsi-hpsa_6.0.0.84-3vmw.670.0.0.8169922, VMW_bootbank_scsi-ips_7.12.05-4vmw.670.0.0.8169922, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.670.0.0.8169922, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.670.0.0.8169922, VMW_bootbank_scsi-mpt2sas_19.00.00.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.670.0.0.8169922, VMW_bootbank_sfvmk_1.0.0.1003-7vmw.670.3.104.16075168, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmkfcoe_1.0.0.1-1vmw.670.1.28.10302608, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794, VMW_bootbank_vmw-ahci_2.0.7-2vmw.670.3.143.17700523, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMware_bootbank_cpu-microcode_6.7.0-3.155.18812553, VMware_bootbank_esx-base_6.7.0-3.163.19195723, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, VMware_bootbank_esx-ui_1.33.7-15803439, VMware_bootbank_esx-update_6.7.0-3.163.19195723, VMware_bootbank_esx-xserver_6.7.0-3.73.14320388, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.670.1.28.10302608, VMware_bootbank_lsu-intel-vmd-plugin_1.0.0-2vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.670.2.48.13006603, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.2.48.13006603, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794, VMware_bootbank_native-misc-drivers_6.7.0-3.89.15160138, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922, VMware_bootbank_vsan_6.7.0-3.163.19184739, VMware_bootbank_vsanhealth_6.7.0-3.163.19184740, VMware_locker_tools-light_11.3.5.18557794-18812553
   VIBs Removed: Avago_bootbank_lsi-msgpt3_17.00.02.00-1OEM.650.0.0.4598673, BCM_bootbank_bnxtnet_214.0.230.0-1OEM.650.0.0.4598673, Broadcom_bootbank_lsi-mr3_7.710.07.00-1OEM.650.0.0.4598673, Broadcom_bootbank_lsi-msgpt35_11.00.00.00-1OEM.650.0.0.4598673, EMU_bootbank_brcmfcoe_12.0.1115.0-1OEM.650.0.0.4598673, EMU_bootbank_elxiscsi_12.0.1108.0-1OEM.650.0.0.4598673, EMU_bootbank_elxnet_12.0.1115.0-1OEM.650.0.0.4598673, EMU_bootbank_lpfc_12.4.242.0-1OEM.650.0.0.4598673, INT_bootbank_i40en_1.8.6-1OEM.650.0.0.4598673, INT_bootbank_igbn_1.4.10-1OEM.600.0.0.2768847, INT_bootbank_ixgben_1.7.20-1OEM.600.0.0.2768847, LVO_bootbank_lnvcustomization_6.53-10.1, MEL_bootbank_nmlx4-core_3.16.11.10-1OEM.650.0.0.4598673, MEL_bootbank_nmlx4-en_3.16.11.10-1OEM.650.0.0.4598673, MEL_bootbank_nmlx4-rdma_3.16.11.10-1OEM.650.0.0.4598673, MEL_bootbank_nmlx5-core_4.16.14.2-1OEM.650.0.0.4598673, MEL_bootbank_nmlx5-rdma_4.16.14.2-1OEM.650.0.0.4598673, QLC_bootbank_qcnic_1.0.26.0-1OEM.650.0.0.4598673, QLC_bootbank_qedentv_3.11.16.0-1OEM.650.0.0.4598673, QLC_bootbank_qedf_1.3.41.0-1OEM.600.0.0.2768847, QLC_bootbank_qedrntv_3.11.16.0-1OEM.650.0.0.4598673, QLC_bootbank_qfle3_1.0.84.0-1OEM.650.0.0.4598673, QLC_bootbank_qfle3f_1.0.68.0-1OEM.650.0.0.4598673, QLC_bootbank_qfle3i_1.0.23.0-1OEM.650.0.0.4598673, QLC_bootbank_scsi-qedil_1.15.14.0-1OEM.600.0.0.2494585, QLogic_bootbank_qlnativefc_2.1.81.0-1OEM.600.0.0.2768847, VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.650.2.50.8294253, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.650.2.50.8294253, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.650.3.96.13932383, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-3.96.13932383, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.3-8vmw.650.2.75.10884925, VMW_bootbank_nenic_1.0.29.0-1vmw.650.3.96.13932383, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.22-3vmw.650.2.50.8294253, VMW_bootbank_ntg3_4.1.3.2-1vmw.650.2.75.10884925, VMW_bootbank_nvme_1.2.2.28-1vmw.650.3.96.13932383, VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_smartpqi_1.0.1.553-28vmw.650.3.96.13932383, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.2.50.8294253, VMW_bootbank_vmkata_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.3.96.13932383, VMW_bootbank_vmw-ahci_1.1.6-1vmw.650.3.96.13932383, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_cpu-microcode_6.5.0-3.108.14990892, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-base_6.5.0-3.111.15177306, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607, VMware_bootbank_esx-tboot_6.5.0-3.111.15177306, VMware_bootbank_esx-ui_1.33.4-13786312, VMware_bootbank_esx-xserver_6.5.0-2.50.8294253, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.650.3.96.13932383, VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.650.2.79.11925212, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-11vmw.650.2.75.10884925, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.650.2.79.11925212, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vsan_6.5.0-3.111.15079806, VMware_bootbank_vsanhealth_6.5.0-3.111.15079807
   VIBs Skipped: INT_bootbank_iavmd_2.7.2.1001-1OEM.670.0.0.8169922, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.2-3.89.15160138, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2vmw.670.3.116.16713306

My theory of bitrot probably stands as the root cause of all this, but it was transient because I was eventually able to get the upgrade to work when I re-attempted.

Boot into ESXi 6.7

I reboot my server and I see that wonderfull screen that tells me I am running VMware ESXi 6.7 on a x3550 M2

IBM System x3550 M2 (7946) running VMWare ESXi 6.7

So even as late as 2023 you can still run a supported version of ESXi server on the IBM x3550/3650 M2's. This is a system IBM announced all the way back in March 2009. By running ESXi 6.7 you can unlock the ability to use the latest and greatest version of vCenter Server 8.0 with this hardware.

Do expect that VMware will drop support for 6.7 with vCenter 9.0 when that comes out. By that time the M4's and M5's will have taken over the second hand market. They are already starting to get retired and I am seeing some M5's show up for some reasonable prices. In any case, you would have gotten up to 20 years of service from the M2/M3 series, and when the versions of VMware ESXi/vCenter finally get too old to be useful, you can always throw the latest copy of FreeBSD on those guys and use Bhyve or Jails for your workloads,

Not bad, not bad at all

Systems & Networking