Summary: A possible cause for a failed ESX 4 to ESX 4.1 upgrade.
Date: Around 2011
Refactor: 2 January 2025: Checked links and formatting.

During the upgrade from one of our hosts from 4.0 to 4.1 we experienced that the host would hang during the upgrade process, at exactly 33%. The first time I was impatient and restarted vcenter, the host and tried again. That time I decided to wait. After more than an hour it failed with this error:

VMware vCenter Update Manager had an unknown error. Check the Tasks and Events tab and log files for details.

Also, when looking at the host in the esxupdate log (/var/log/vmware/esxupdate.log) I saw these errors:

[2011-03-03 22:56:14]   DEBUG:    cos.rpm: error: rpmdbNextIterator: skipping h#       2 Header SHA1 digest: BAD Expected(ec2f873ad51b3275ad5656a8a85cd0e2f7cff681) != (f777d38c915c35e45de79023745b6757c9b61038)
[2011-03-03 22:56:14]   DEBUG:    cos.rpm: error: rpmdbNextIterator: skipping h#       2 Header SHA1 digest: BAD Expected(ec2f873ad51b3275ad5656a8a85cd0e2f7cff681) != (f777d38c915c35e45de79023745b6757c9b61038)
[2011-03-03 22:56:14]   DEBUG:    cos.rpm: error: rpmdbNextIterator: skipping h#       2 Header SHA1 digest: BAD Expected(ec2f873ad51b3275ad5656a8a85cd0e2f7cff681) != (f777d38c915c35e45de79023745b6757c9b61038)
[2011-03-03 22:56:14]   DEBUG:    cos.rpm: error: rpmdbNextIterator: skipping h#       2 Header SHA1 digest: BAD Expected(ec2f873ad51b3275ad5656a8a85cd0e2f7cff681) != (f777d38c915c35e45de79023745b6757c9b61038)
[2011-03-03 22:56:14]   DEBUG:    cos.rpm: error: rpmdbNextIterator: skipping h#       2 Header SHA1 digest: BAD Expected(ec2f873ad51b3275ad5656a8a85cd0e2f7cff681) != (f777d38c915c35e45de79023745b6757c9b61038)

The problem is a corrupted rpm database. But when I tried to fix it I ran into a locking problem, probably because VMware Update Manager is still using the database:

[root@ESX-02 vmware]# rpmdb --rebuilddb
warning: waiting for transaction lock on /var/lib/rpm/__db.000

So I had to reboot the host (again), and after that I could fix the database:

[root@ESX-02 ~]# rpmdb --rebuilddb
error: rpmdbNextIterator: skipping h#       2 Header V3 DSA signature: BAD, key ID 37017186
[root@ESX-02 ~]# rpmdb --rebuilddb

Note: I ran the rebuilddb option twice to make sure there were no errors left.