Friday, September 4, 2009

SQL Failure After Upgrade from VMWare 3.5 to VSphere 4 Fix


I ran into a problem with our Microsoft ISA Server after moving to VSPhere 4 today. After searching the 'net for a solution I found a lot of painful solutions, most of them had no effect or made matters worse.

I have finally tracked the issue down and corrected it myself. It's a 5 min. fix. If you want to skip all my jabbering and just get to the solution, check the section marked Solution below. Fair warning that the Problem and History section contains information you need to know to confirm that this is indeed the correct solution for you.

In this case the victim was an ISA server, but any SQL server or VM client that uses Msvcr71.dll could be affected.

Problem and History

After upgrading my ESXi server from version 3.5 to 4.X to allow for management via VSphere 4 I noticed external users were unable to connect to my reverse proxy server.

I logged into the ISA server via the VSphere manager as I was unable to connect via RDP (If you are running a SQL only server you will not have this problem). When I opened the ISA manager (Start-->All Programs --> Microsoft ISA Server --> ISA Server Management) I noticed that Monitoring showed both the Firewall and MSDE services were down (The Monitor can be accessed by Clicking Monitor on the left hand menu, and selecting the Dashboard tab). I then attempted to restart the services by clicking the Services tab, right-clicking on the Microsoft Data Engine and selecting Start. I repeated this for the Microsoft Firewall. Both times ISA reported that it was unable to start the service.

The Database instance for the firewall (not to be mistaken for the Microsoft Firewall Service) is MSSQL$MSFW. It can be manually started in the Services Administrative Tool (Start -->All Programs -->Administrative Tools --> Services) by right clicking on the MSSQL$MSFW service and selecting Start. In this case I got error events 7009 (Timeout waiting for the MSSQL$MSFW service to connect) and 7000 (The service did not respond to the start or control request in a timely fashion). OK, great.

I needed to know If SQL was actually still viable, so I ran sqlservr.exe (C:\Program Files\Microsoft SQL Server\MSSQL$MSFW\Binn\sqlserver.exe)* and braced for the excitement. I finally got something useful: The message "This application has failed to start because MSVCP71.dll was not found. Re-installing the application may fix this problem". I did not really feel much like taking half a day to reinstall and configure so I searched the server for msvcp71.dll. There were a couple of them with the added extension .F26FFD4A_Yadayada which probably would have worked if I copied them to %windir%\system32 and renamed them, but nothing named msvcp71.dll. I powered up the old ESXi 3.5 VM (I ALWAYS backup before an upgrade, and you should too) and sure enough msvcp71.dll existed in both the %windir%\system32 and more tellingly C:\Program Files\VMWare\VMWare Tools. Apparently the 3.5 and older tools use msvcp71.dll, and 4.0 does not. When I upgraded the VMWare tools the 3.5 tools are uninstalled along with msvcp71.dll from both the VMWare Tools directory and System32. VMWare tools 4.0 does not require msvcp71.dll, so the .dll is never returned.. Very very naughty.

Solution

Easy peasy mac and cheesy: Download msvcp71.dll from dll files at http://www.dll-files.com/dllindex/dll-files.shtml?msvcp71 or copy it from your (better have) backed up VM. Copy msvcp71.dll from wherever you unzipped the file to %windir%\system32. Reboot the server. Prayer may help here. Confirm the correct operation of SQL. Go get a beer.

*Yes bin is actually "binn" in the path

Post a Comment