Officials discuss causes of problems with online evaluation
A series of factors centered around a higher-than-normal workload on the CTools system combined to cause the April 20 shutdown of the new online teaching evaluation process.
Information technology officials told the Senate Advisory Committee on University Affairs on May 11 they have identified the problems that contributed to the shutdown and have taken steps to fix them.
Additionally, the ongoing consolidation of U-M's IT functions should help avoid such problems in the future, officials say.
"A number things that came up this term are things we now have the opportunity to address with the consolidation of the IT systems," said John Merlin Williams, executive producer of Digital Media Commons.
The primary contributor to the evaluation shutdown was a much higher level of intense use by the system in very short processing cycles. Although robust pretesting of the system occurred, that level of short-cycle processing was unprecedented and severely strained resources, Williams said.
Other factors exacerbated the problem, causing a sort of "perfect storm" that necessitated closing the course evaluation process six days into the eight-day schedule for the Winter Semester evaluation process to enable the CTools system to remain in use.
In general, those problems included a level of concurrent users that was the highest of the term, and an e-mail notification process that inadvertently triggered volumes of students logging in to submit evaluations that the system could not manage.
It was the second semester that the online questionnaire had been used. Evaluations were accepted without incident during the first semester.
Also contributing to the problem is a broader issue: The number of functions being carried out through CTools has grown significantly since it was created as a research project in 2003. A system of 10 basic tools six years ago now contains 35.
"We've put mission-critical systems in it," Laura Patterson, associate vice president for administration information systems, told SACUA.
A long-range solution, Williams said, involves making the CTools infrastructure more robust so it is able to handle the increased processing loads being demanded of it.
For the short term, several steps are being taken to address the problem:
• Parts of the software will be replaced with more efficient code.
• Some lower-priority functions that put stress on the system will be turned off during peak usage periods like the end of a term.
• The e-mail notification process will be conducted in waves so responses will be spread more evenly through the day.
• A higher-capacity database environment is scheduled for installation in July.
• External reviews of the CTools software, configurations and production support practices will be conducted over the summer.
At the time of the evaluation shutdown 52 percent of students had responded to one or more evaluations, comprising 39 percent of all possible evaluation responses. This data has been collected and is being processed for reporting to the academic units.