Exchange 2003 Fail Server

Holy hell, it’s hard to recover mail from an Exchange Server if you’ve lost your domain controller. But I’ve done it. Once again, I’m a legend.

On Monday, a customer’s Windows 2003 Small Business Server went down due to a planned power outage. When it came back up again, the drives were corrupt. Their most recent backup was over a year old, and the RAID mirror had replicated the corruption.

I spent the better part of Monday and Tuesday fighting with the machine. Before I did anything, though, I imaged the data. I then replaced the existing drives with new 1TB drives to eliminate hard drive failure, and then began the arduous process of rebuilding data.

Active Directory was gone. I was able to log into the server using cached credentials. All attempts at either backing up or repairing the NTDS folder failed miserably. I reimaged the new drives probably four times.

Then on Wednesday morning, the server stopped booting. My boss and I decided to replace the hardware and begin rebuilding the server from scratch. This is an extremely painful process, because it means literally recreating years of configuration, certificates, licencing, e-mail and user data in a few days.

I was able to repair the Exchange database, using the built-in command-line tools. I made a backup of the EDB and STM files on a separate partition, along with the user shares and the Intranet website files (and SQL database), to prepare for the server rebuild.

Yesterday, Thursday, it became clear that Windows 2003 was not going to run on the new hardware we specified. What ever I tried, Active Directory kept failing. Whether at the initial AD setup phase, or during the Exchange 2003 installation, we got error after error.

So it was with a bold and insecure step that we decided to put our faith back in the original server. As the hard drives had already been replaced, I decided to replace the power supply as well.

As of right now, the server is still going strong. I believe that the power outage caused damage to the power supply in the server, which in turn caused corruption on the drives. With a new supply, the server has been performing admirably (despite being a Dell).

Tonight, I decided that I would spend some time this long weekend to recover the Exchange mailboxes. If you know anything about Exchange, you’ll know that it is deeply tied to Active Directory. Since I’ve had to rebuild the server (and change the domain name to avoid conflict when we reconnect the server to the network), all associations between the old Exchange database and the new domain are severed.

There are tools that Microsoft provides, to allow reconnection of the Exchange store. However, there are some steps one must follow:

1. Ensure that you’re logged in as a Domain Admin. It’s the right thing to do in this case.
2. Shut down the MSExchangeIS service and swap the EDB and STM files with the ones you’ve recovered from the old server.
3. Restart the Information Store service, and mount the store you’ve switched in.
4. Make sure the user has full rights on the Mail store, including Receive As and Send As permissions.
5. Now the fun part: if you’ve changed permissions, you should restart Windows. It’s quicker than waiting for the AD to replicate permissions (usually 15 minutes, but can take longer). I was fortunate in that I set up a LAN consisting of only one machine: the server.
6. Once in Exchange System Manager again, hook up the Administrative Group display. It gives you more options. You’ll have to go out and come back in again.
7. Now create some AD users. Make sure you do not give them email addresses, or associate them with Exchange. The reason is simple: you want to assign the old mailboxes with the new users.
8. In the mail store, set the Limits to 9999 days for keeping items. This will bring across every item associated with the old mailbox for the user.
9. In the Mailbox Recovery Center, hook up your mail store. You may need to refresh something, or run the Cleanup Agent or something. I forget now.
10. Right click on an account, and Find Match. If the AD user you’ve created has the same name, you’re in luck. You can then Reconnect it and it’ll be happy.
11. Once you’ve reconnected all of the mailboxes, start up Exmerge. This tool allows you to export and import mailboxes in PST format. Very nice. Very useful.
12. Using the two-step process, export all the mailboxes from the mail store to a folder on your hard drive. Make sure the errors are managed accordingly.
13. Shut down the Information Store again, and switch back the new files.
14. Restart the Information Store and mount the store if necessary.
15. Using Outlook from each users’ machines, import each PST file back into Exchange. Personally, I’d do this route instead of maintaining the old mailboxes, because you can take this opportunity to clean up each mailbox.