How I saved my butt by restoring my moodle to pre-upgrade state

Today, I had the bright idea (bad idea) to upgrade one of my moodle instances from 2.3 to 2.6.  A little background first, I have already upgraded a few other moodle instances to 2.6 over the past week without any trouble.  I simply waited til the end of the day (lower traffic on site), put the site in maintenance mode, and did the upgrade. 

A quick recap of the upgrade steps taken on previous sites:

1. Download and unzip the updated moodle code
2. Create a temp folder
3. Copy the config.php file from the moodle instance you are about to upgrade into the temp folder
4. Copy any custom images you are using, mine were located in the local folder, into the temp folder
5. Delete all the moodle code in the site folder
6. Copy all the upgraded moodle code into the currently empty site folder
7. Copy the config.php file back into the site folder (with all the upgraded moodle files) 
8. Copy the image files back into the new local folder off the site root
9. Browse to the site URL, and watch the moodle upgrade do its stuff.

Prior to starting the upgrade on my latest site,  I checked the Live Logs report to ensure no one had been on the site this am.  The coast looked clear, so i put the site in maintenance mode and executed the steps above.  The entire process took maybe 10 minutes and everything looked good.  10 minutes later, my customer sends me an email with something like this

"I tried accessing my course this am and got a php error saying contact a programmer."  The blood pressure starts to rise....I thought, she must have tried hitting the course as it was in the upgrade process, although that was faulty thinking since i had the sight in maintenance mode.

2 minutes later another email "why does the site look different?"

2 minutes later another email "all the courses are now set to self enrol, whats going on?"

Blood pressure rising.....I'm thinking, "you are a dummy"...

I accessed the course that was giving the php error message and could access it fine.  I was logged in as the administrator and could see the course, no error.  Then I logged in as one of the students in the course, accessed the course and saw the error, in big redish letters

Coding error detected, it must be fixed by a programmer: PHP catchable fatal error 

Now my blood pressure was really starting to rise.  Even though this error makes no sense and should not have happened, it was caused by the upgrade.  This error is a problem in the DB.  I googled the error, hoping there might be something I could do to resolve the issue, quickly, but no dice.

To recap, I had gone against good process and upgraded a site without any communication to my customer and owner of the site.  I had become lulled by the ease of which the other sites upgrades and from what I thought was a very inactive moodle instance.

My customer, rightly, is trying to understand why I would do this without contacting her but is mostly concerned about the participants in one of the site courses being able to continue to access the course and the content that they had already contributed.  I cant just restore the course and re-enroll the accounts, they would lose all their progress and I would be in big trouble and have a very pissed client.

Next, I tried restoring a backup of the course that my automated backup script created two days earlier using moodle version 2.3.  The course restore process made it most of the way and actually reported that it failed at the 99.6 % point of the restore, nice huh! Even if this would have worked, I would have lost a couple days of participant progress.  But, that would have been something.

Next, I logged back into the site in the admin role, entered the course in and make a new fresh 2.6 version course backup.  I restored the fresh backup without error, but when I logged in as one of the students, clicked the newly restored course, BOOM, same error message (see above). 

My blood pressure was still going up....

I was running out of option in my new instance (moodle 2.6).  I mean, we could deal with the site looking different, and even the courses having self enrollment options on them, since we could remove them, but we COULD NOT lose the course with the current batch of enrolled users and their contributions/progress in the course, we call that a show stopper.  NO, that would not do...my customer was loosing patience and faith in me and moodle.

In the pressure cooker of the moment, my brain said "James, put the entire site back to the way it was by putting back the moodle 2.3.1 code base and restore the DB backup that you have from last night!".  What a saving grace, that I had a copy of the DB from 2:30 this am.  This is why you backup the DB every night, or at least on some regular interval, in case you ever need it!.

Really, the important takeaway for me is prior to performing ANY MOODLE UPGRADE, create a fresh backup of the current DB.  Remember the upgrade will upgrade the DB and not allow you to role that back to the previous version.  This is the FIRST TIME, after about 30 moodle upgrades, where I needed to restore the DB to its pre-upgrade state!

1. found the moodle 2.3.1 code, that I had used in a folder on the site called server downloads, that I had saved from the previous install
2. copied the config.php file from the root of the upgraded moodle 2.6 code base, into a temp folder
3. copied the custom theme from the theme folder into a temp folder
4. deleted all the moodle 2.6 code from the site root
5. copied the moodle 2.3 code back into the site root
6. copied the config.php file back into the site root and put the theme folder back
7. opened a DOS command window and changed to the MySQL folder
1. cd wamp\bin\mysql\mysql5.5.24\bin
2. logged into MySQL using
3.  mysql -u username -p dbnameToRestore
4.   provided password for the username (this account exists in the users table in the mysql DB on the moodle site) - I use phpMyAdmin to access the DB and table
5.   After authenticating, I was greeted by a mysql prompt like; 'mysql>'
8. I then issued the this command to restore the DB
1.  source: drive:\path\backupfilename.sql
9.  which restored the 2.3 version of the DB over the upgraded 2.6 version.

I browsed back to the root of the site and everything was back to the way it was prior to my upgrade earlier this am.  I accessed the course and logged in as one of the students and accessed the course, and there was no error, thank you Jesus. 

This is the first time I have seen any problem with a course, its enrollments or content after an upgrade.  The first time I have had a real problem with upgrading. I looked in the php error log for the instance and got a little more detail about the error, here it is:

"Default exception handler: Coding error detected, it must be fixed by a programmer: PHP catchable fatal error Debug: Argument 1 passed to role_get_name() must be an instance of stdClass, boolean given"

There are a few morals coming from this experience.

1. Plan your upgrades, do not assume they will work flawlessly.
2. Test your upgrades, especially if some of the content is using a lot of course type activities.
3. Back up your DB, regularly.  This saved my butt today!
4. Know how to restore a DB backup, practice disaster recovery!

Since I had practiced disaster recovery and had restored DBs before, I knew what to do and did not let the STRESS OF THE MOMENT  cause any more problem than necessary - I cool under fire.