Outcome Analysis of Distance Learning:

A Comparison Between Conventional and Independent Study Instruction

Abstract
In this report we describe two methods used to present an introductory computing class. One version used conventional scheduled lectures and laboratories; the second version was based on independent study (IS), with students working on their own. Two sections of the IS course have been offered to date. The first section compared a control group of comparable students from the lecture mode class with IS students, in terms of performance outcome. Results based on pre-testing and post-testing of both groups show that the IS students performed as well as those in the lecture course. Two quarters later, the same instructor offered a second IS course at the same time that another instructor offered a lecture section. These courses showed similar results. Analysis of the use of multimedia resources suggests a need to adapt linear lecture material to a more dynamic, user-controlled format for multimedia presentation in IS courses.

Independent study courses offer a potentially attractive alternative to scheduled lecture courses for many students. Enhanced by interactive electronic communication, this approach allows instructors and teaching assistants to use their time in responding to student questions, whether electronically or by other means. Furthermore, IS courses can reach audiences not otherwise able to take such courses, owing to scheduling, location, financial or other factors.

An earlier paper (Walters, 1994) describes an introductory computing course, "General Education" (ECS 15), for non-computer majors. Several hundred students enroll each year; it's always filled to capacity, with a waiting list. In the Fall quarter of 1995, in addition to the regular ECS 15 class, there was a first offering of ECS 15AT (Autotutorial), an IS ECS 15 class. There was an additional section of ECS 15 AT (the IS course) in the spring quarter 1996. The first author of this report taught all three sections.

We present a comparison of the two versions (ECS 15 and ECS 15AT) offered in fall quarter 1995 and the additional section of the IS version (ECS 15AT) offered in Spring 1996. We describe the courses, their resources, the delivery of both lecture and IS sections; we also present the outcome of each section. The paper concludes with recommendations for improving the IS offering.

Introduction to Computers

ECS 15 is a general education course using conventional lectures and scheduled laboratories. The same course, taught in the IS mode, ECS 15AT, uses a student-instructor interaction named Remote Technical Assistance (RTA), videotaped lectures, a laboratory manual with detailed exercises for each session, and other materials for use in self-study mode. ECS15 and ECS15IS include nine laboratory exercises covering the use of MSDOS, a word processor, use of the Internet and the Web, a spreadsheet, and four exercises in a high-level computer programming language (Mumps or Scheme). For three hours per week, groups of approximately 25 students meet in the lab. The lab contains 30 fast PCs with Internet connections and a variety of pre-loaded software. At least two instructors/assistants are available. In addition to the laboratory exercises, there is one midterm, a final, and a term paper written on the use of computers in some field of interest to the student. The course is offered every quarter to approximately 100 students in lecture/scheduled laboratory mode. The enrollment is limited by hours available for computer laboratory use and funding for instructors/assistants/graders. The demand for the class always exceeds the available space.

Administration of the Scheduled Versus the IS Offerings

We decided to test the IS version with a small group of students--25 or fewer--and to offer two sections simultaneously, one lecture mode, the other IS (we use the term IS to refer to the latter group).

The lecture mode section was open to 100 students using lectures, scheduled laboratories, and strict deadlines for all assignments. The second section was open to juniors or seniors; they were not permitted to attend lectures or scheduled laboratory sessions, relying instead on the printed and electronic resources made available to them, but including access to the instructor and TA during office hours and the learning materials described earlier. Both groups had the same deadlines, and possibilities of different exposure to instructional resources were minimized.

Twenty-five students signed up for the IS version, with 20 completing the course in Fall 1995. Forty-six of the students in the lecture section were at the equivalent academic level and served as the control group for the study. In Spring 1996, the IS section had an enrollment of 25, with 21 students completing the course.

Recognizing that this paired offering represented an excellent opportunity to study the effects of distance learning, we performed an analysis of the relative merits of the two courses. The support paid for a research assistant, for data collection and analysis, and for additional student instructional staff.

Our pre-test to give us baseline information on the students taking both versions of the course included several types of questions: term identification, programming concepts, computer components, and the relative costs of hardware and software. We also surveyed both groups to determine general background, including educational level of parents, high school and college GPA, and the degree to which they made use of computers for word processing or other applications. The registrar provided cumulative college gradepoint average (GPA) for students in both groups.

Learning resources. Learning resources included the instructional staff, who were available during scheduled laboratories (for lecture mode students) and during office hours; handouts included the laboratory manual, lecture notes, term paper guidelines, a list of references used by former students, and auxiliary material on the World Wide Web (WWW). Included on-line were: guides to the use of software; lecture notes in HTML format; previous examinations; solutions posted following each week's laboratory exercises; sample term papers; a keyword-indexed list of over 1,000 references used in previous classes; searchable on-line by keyword; course syllabus and all handouts; a list of videotapes including those described below and ones purchased commercially for the class; and a glossary of computer terms.

We videotaped the majority of the 16 lectures given in the conventional class as well as 9 laboratory introductions, recording them and using the instructional media facilities available at Davis. Tapes, which ranged in length from 9 minutes to 45 minutes, included extensive use of special graphics, live shots of such scenes as computerized traffic signals, barcode scanning in grocery stores, and computer workstations of several different types. We used some of these tapes in the first part of scheduled lecture hours, supplementing them with a discussion period at their conclusion. The tapes were also available for playback in a campus facility. Arrangements for video check-out are being explored. We have not yet concluded arrangements to broadcast the tapes overnight via the local cable service, but negotiations are underway.

A concept under development at Davis, the software package called Remote Technical Assistance (RTA), provides students with three methods of communicating with instructional staff and access to auxiliary resources. The first, an advanced form of messaging significantly improved over e-mail, includes the ability to attach screen snapshots, files (e.g., programs, term paper drafts, spreadsheets), and even audio clips in store-and-forward mode. The second mode permits live interaction with a member of the instructional staff, including shared screen annotation of snapshots and other images, and multimedia file transfer to enhance the interactive dialog. The third component is an "expert system," described in Fonseca and Reed (1996). The expert TA (ETA) is based on resources prepared for the course and augmented by responses to frequently asked questions as they occur in other modes of RTA use.

Deadlines. Both IS and lecture mode students had strict deadlines for submission of laboratory exercises and term paper-related assignments. However, students taking the IS course were allowed to complete assignments early (including taking examinations when they felt prepared).

Student performance measures. To provide a consistent base, the same individuals did the grading in all three sections, using absolute measures; there was no grading "on a curve." Laboratory exercises turned in on time received full credit.

As a further incentive to in-depth learning, students earned a maximum 7 points extra credit by completing advanced portions of the laboratory exercises and by turning in newspaper clippings related to computer uses in today's society. Letter grades assigned were 88, 78, and 70 as lower limits for grades of A-, B-, C-, respectively. The weighting of points assigned to different portions of the class were: laboratory exercises: 40%; Term Paper: 25%; Midterm 15%; and Final: 20%.

Analysis for Comparison of Student Performance

Table 1 below shows two performance outcomes along with the differences from the baseline measure. Because the final exam was not cumulative, the average of exams (midterm and final weighted in the same ratio as for the course) became the exam measure (expressed as a percentage). The pre-test most closely matched the exam questions in type of material. The total course score became the second measure. Each outcome is compared to the pretest scores for the group.

The pretest scores were not significantly different, although the average for the Spring IS section was more than 10 points higher than the averages for the other sections. One explanation is that by Spring quarter, all students have had at least two quarters to familiarize themselves with the computers on campus at Davis. During Fall quarter, freshman and transfer students are completely new to campus.

The outcome results of the three groups were nearly identical in exam totals and course totals as shown by the group averages in Table 1. The percentage of improvement from baseline (difference from pre-test) measures were smaller for the spring section (although not significantly), which appears to be due to the higher pre-test average in that section.

The difference from baseline was 45% or more in all sections. That is, students achieved almost exactly the same final scores in all groups, and almost identical improvements from the pretest to the exams. One student in the Spring section did well on both exams, but did not complete a term paper, therefore receiving a score of 0 for that portion of the course (25%). Therefore the student's total score was 25 points lower (of 100), which could explain the smaller difference in course total compared to exam total.

We next analyzed the basic profiles of students in the three groups. A commonly used criterion of achievement is overall GPA. The GPA at UC Davis is on a scale of 0 to 4.0 with A receiving a 4.0, B receiving a 3.0, etc. A plus or minus on a grade adds or subtracts 0.3 of a grade point, respectively, with the exception that an A+ also receives a 4.0.

The two groups had no significant differences in this measure (see Table 2). Female to male students were approximately 2:1 and 3:1 in the Fall lecture and IS sections, respectively, and were approximately 1:1 in the Spring IS section.

A greater standard deviation in the IS students for both quarters reflected a proportionately greater number of students at the top and bottom GPAs in the IS groups, whereas lecture students were more tightly grouped near the average. A comparison of the histograms of GPA scores for each section further verified this. The histogram for the lecture section shows a much larger group of students in the center than the ones for the IS sections, although there are approximately the same number of students spaced at both "ends" of all the distributions. This finding lends support to the hypothesis that both the fast learners and the slow learners are more likely to self-select IS instruction than those in the middle of a class (in order to move faster or receive more individualized attention, respectively).

Student Evaluations

Almost all students (including those in both IS groups) completed final course evaluations, which sought (a) an overall evaluation of the course effectiveness; (b) estimates of the use of different learning resources and opinion of their value; and (c) reasons for selection of this format and degree of satisfaction with the approach and the various auxiliary learning resources. These results, summarized below, do not appear in table form for reasons described next.

Overall course satisfaction was high, averaging 8.7 in Fall 1995 (81 responses of 135 enrolled students) and 8.1 in Spring 1996 (8 responses of 26 enrolled) on a rating scale of 1-10. Unfortunately, the numerical evaluations for Fall 1995 were not separated by group (lecture vs. IS), so it was not possible to get direct comparisons, but the positive response from the additional questions asked of IS students suggests that their satisfaction equaled that of lecture students. Eighteen of 20 IS students in Fall 1995 and 12 of 15 IS students in Spring 1996 responded that the course met their expectations (one of the negative responses in Fall 1995 was tempered by the comment, "but the instructor made up for it"). Fifteen of 20 IS students in Fall 1995 and 13 of 15 in Spring 1996 expressed satisfaction with the IS format; one commented, "It is empowering to realize I can teach myself." The greatest frustration with this mode of instruction was lack of immediate answers to questions. As noted in the following sections, we have addressed some of the problems mentioned and are working on others.

To the question, "Would you take another IS course?": 11 students in Fall 1995 and 13 students in Spring 1996 said yes, 4 said no in Fall 1995; there were no negative responses in Spring 1996. The remainder said it would depend on the course content.

Use of learning resources. Because this class has made extensive use of electronic and personal support for several years, both lecture and IS students had equal access to e-mail, videotape libraries, keyword-indexed term paper reference listings (both electronic and hard copy), electronic lecture notes and other WWW resources, and electronic solutions to laboratory pre- and post-tests, practice and actual midterm, practice final, and a variety of other items.

As our survey showed, however, there were significant differences in the use made of these resources, based on self-reporting and instructor data. The results are summarized in the Table 3.

E-mail use decreased with both groups during Fall 1995 in the final weeks of the term (based on separate statistics collected by the instructor). Office visits were higher for both IS sections compared to the lecture section. The visits tended to increase at the end due in large part to an especially difficult laboratory exercise at the end of the term. IS students reported spending over twice as much time on this last laboratory exercise on average (11.4 hrs for IS students in Fall 1995 and 4.9 hours in Spring 1996, vs. 5.5 for the control group).

Hardly any students used the videotapes. It may be significant, however, that the one IS student who saw almost all tapes also received the top grade in the final and was in the top three students in overall performance for the course. Students seemed to feel that the lecture notes were sufficient, and that going to the playback center to view the tapes was too inconvenient. A tape check-out procedure, or broadcasting over the local cable TV channel would facilitate the viewing of tapes by distance learners and on-campus students with more time or location restrictions. Also, the linear format (30 minutes or so per tape) does not seem to appeal to students outside the lecture hall.

Reasons for selecting the IS format. When the IS students were asked what their reasons were for selecting this format (multiple reasons were encouraged), eight Fall 1995 and five Spring 1996 students reported schedule conflicts with job or other preferred activities. Six fall and five Spring students wished to move at a faster pace. Three students in each fall and spring section enrolled because the lecture mode was closed out. Three in each IS section stated they felt they had the self-discipline to undertake a course in the IS format. Two in each Fall and Spring section honestly reported that they wanted to avoid lectures. One student in the spring noted that they were experimenting to see if they had the required motivation.

These answers suggest that the IS mode is not for all students. Many prefer the tried and true methods of instruction. Some reported that competition with other scheduled classes led to difficult choices in the use of time. Several lecture-mode students attended relatively few lectures, evidently content with just a passing grade and exhibiting minimal interest in the course. These students might have done even worse in an IS mode course. One IS student in the spring reported attendance at over 2/3 of the lectures given by another instructor that quarter. (That student finished with an excellent course total, approximately 95%).

Costs

Resources generated for ECS 15 include a series of lecture notes, a laboratory manual, video tapes, and electronically stored resources. In addition, the instructor has compiled both a hard copy and an electronic bibliography, indexed by keywords, of references used by previous students taking the course.

The total cost of producing the videotapes, exclusive of the instructor's time scripting and taping, was just under $50,000, or an average of slightly less than $2,000 per tape. Videotaping of lectures and laboratory introductions was the most expensive item prepared for the IS version. These tapes required careful scripting, animation sequences, field trips for shots of special sites (e.g., bar-code scanning in a grocery store), taping with director, technician and recorder plus instructor at each session, editing/revision, and some update or replacement.

Students in the lecture mode class used all of the resources. Whereas only selected videotapes are shown each quarter, they are also used in lectures as a better means of bringing exhibits to the classroom and of using animation and other techniques. Because the instructor makes a practice of preparing lecture notes and laboratory exercises for other courses not at present taught in the IS mode, it is difficult to assign a cost specific to IS instruction in this case.

Cost associated with delivery of the course. It is not clear what costs can be specifically ascribed to the IS class. To obtain more precise information, in a second IS section during Spring 1996 only the instructor and one reader responded to e-mail, holding office hours, and having one assistant available during one three hour lab section per week. The second author of this report taught a concurrent lecture section of ECS 15 in Spring 1996, with slightly different course content. The IS students did take advantage of office hours with their instructor and assistant, as well as the scheduled lab time. In addition to student contact, the same amount of time is necessary to grade the assignments, exams, and term papers of IS students.

Conclusions and Recommendations

Based on student comments from Fall 1995 IS section, we scheduled laboratories during Spring 1996 each week (three hours) with one assistant attending to answer questions. This was greatly appreciated by the students; they could get immediate help from the lab assistants, resulting in greatly reduced time spent by students.

As discussed in some detail by Laurillard (1993), the use of technology in instruction requires training and careful attention to the course objectives during design and construction of these resources. Electronic resources and other instructional support seem to have been adequate for this course. Videotapes, although acceptable replacements for some lectures in lecture mode, do not fit the IS learning mode. We need to revise the content, convert to a non-linear, hypertext form of presentation of short clips of those sequences (animation, live action) where motion is important, and convert other material to fixed images with audio or written narration.

Undoubtedly the greatest disappointment to the instructor, TAs, and students, was the fact that RTA did not get to operational status in time for it to be used in this class. It is clear that the interactive dialog mode would have received significant use had it been available in a form convenient to students.

One conclusion that seems inescapable: resources for courses of this type will require constant updating and revision. This is not different from the needs of conventional courses, but it may require a little more planning ahead.

These results strongly support the idea that IS courses can succeed, given the right motivation, instructional materials, and support.

Although students taking each IS course were on campus during some portion of each week, the results could be extrapolated to distance learning, where student-instructor interaction would have to be via electronic means. The RTA concept would be a major benefit for that type of independent instruction, also.

Additional investigations should produce more information about alternative forms of instruction, including Distance Learning. Although many questions remain unanswered at this time, we hope that this study will provide a stimulus for future work of a similar nature.

References

Fonseca, S. P. & Reed N. E. (1996, August). Integration of an expert teaching assistant with distance learning software. Proceedings of the Thirteenth National Conference on Artificial Intelligence 2, 1388.

Laurillard, D. (1993). Rethinking university teaching: A framework for the effective use of educational technology. London: Routledge Press.

Walters, R.F. (1994). An introductory course on M worth exporting. M Computing 2(1), 13-19.