Batching in lots of users in moodle

I have been helping people the past couple days with batching and enrolling of bulk groups of students.  I used to say, stop batching, create the accounts via the native interface and then enroll via the native interface.  That is ok when you are dealing with a few students.  When dealing with a lot of students, this process is impractical.  For example, one of the districts we support are bulk loading every student in their middle and high schools.  That was about 1000 students per building.  They have a data system that clearly generates the student name, email, username and password, all the required moodle fields.  They use the moodle interface and choose bulk user uploads.  It got interesting when they sent me a copy of the two csv files generated by their system and said "james, this one works fine but the other does not, we cannot find any difference".

How to find a problem in a big honkin csv file?

I glanced at it and it did look like the other one that worked.  Past experience has taught me that the problem is often times in a trailing , "comma" at the end of a line or more evilly, buried deep inside a line.  I started looking line by line and counting commas, but that was too straining on my eyes.  Instead, I grabbed the first 100 lines of the file, including the all important header line with the field names, and saved to a temp file and then imported the temp file.  I did this about 6 times in a row.  On the seventh try (records 701 - 800) the upload failed, it did not parse the file properly.  With a more narrow record set, I began checking commas again and found the culprit.  There was a comma in a last name.  Pure evil.

Other things I learned were that moodle did not care if the file was .xls, .csv or .txt.  But a couple times, I have had BIG PROBLEMS when trying to import an excel file or .xls into moodle.  Excel includes something in the file format that I cannot see that is an illegal character and causes the whole operation to fail.

I have copied the contents from the .xls file and pasted into a simple editor like NotePad ++ and then imported that file into moodle WITHOUT problem.  Same data, at least the same data that I can see.

A final note, when I paste the contents into the editor and save as .txt or .csv, there are no commas as delimeters, only tabs.  When importing into moodle, I can choose a different delimeter, one that recognizes the tabs and parses the file correctly.  Yea.

Moral of the story?

Have your system create the csv file as a .txt or .csv file.  Do not use an excel spreadsheet when you have a choice.  Second, if you have a problem, break the large file down into smaller pieces until you find the culprit. Third, use batching to automate account creation and enrollment, for large #'s.  Fourth, promote self registration on your moodle site.  Set it up to allow students to create their own account and self enroll.

If you have a database with all the user credentials already in place, try an enrollment plugin that allows you to synch up with it, rather than having to deal with a bulky problematic .csv file.

I'm just saying....

Also - when enrolling, the column for the course short name is course1, not course or short course or Short course....sometimes I am a dummy, only in hindsight though.