A pilot study led by the Bok Center for Teaching and Learning placed a premium on person-to-person interaction. “The beauty of new digital teaching tools is that they provide us with the type of rich data that we haven’t been able to readily study before. This first iteration was a learning-by-doing adventure,” said lead investigator Jenny Bergeron, director of educational research and assessment at the Bok Center.

B. D. Colen/Harvard Staff

Nation & World

A virtual analysis

7 min read

Pilot study put blended-format courses to the test

A new analysis of four blended-format courses taught last fall offers practical guidance for faculty members interested in fresh pedagogical approaches.

The pilot study led by the Bok Center for Teaching and Learning and released today after months of checks and balances showed that students responded most to lesson structure and execution, placed a premium on person-to-person interaction, and found redundancies between in-class and online instruction.

A related report from the Harvard Graduate School of Education (GSE) that focused on a single undergraduate history course also covered in the Bok study had similar findings.

The Bok study analyzed student responses to four undergraduate General Education courses taught in the fall of 2013 that integrated online components into the classroom: “The Einstein Revolution,” “Concepts of the Hero in Classical Greek Civilization,” “China,” and “Science and Cooking.” Each course had run previously in a residential format and some had been offered fully online via edX, and the courses differed in size, classroom type, and pedagogical practice.

The blended versions of the classes used a variety of online learning experiences at different stages of development. The online content was developed through HarvardX to assess how each class integrated the online elements, evaluate its strengths and weaknesses, and gauge student reactions to the format.

“Conducting extensive research on the use of digital tools to improve teaching and learning on campus, in fact, was one of the chief goals for creating HarvardX,” said Rob Lue, faculty director of HarvardX and director of the Bok Center.

“What we really ended up doing was coming up with some strategic pointers on how to thoughtfully implement blended courses,” said lead investigator Jenny Bergeron, director of educational research and assessment at the Bok Center. “The beauty of new digital teaching tools is that they provide us with the type of rich data that we haven’t been able to readily study before. This first iteration was a learning-by-doing adventure.”

‘Conducting extensive research on the use of digital tools to improve teaching and learning on campus, in fact, was one of the chief goals for creating HarvardX.’ — Rob Lue

The study used data to identify effective techniques that could be adopted widely by faculty. Bergeron, GSE graduate students Michelle Bellino and Lisa Shen, and Bok staff members conducted student surveys and focus groups, ran in-class observations, and compared existing assessments such as the Harvard Q guide (a compendium of student feedback on courses and faculty long used by the College as a benchmark).

“Our goal was to look at the implementation,” explained Bergeron. “We knew that we didn’t want to conflate implementation challenges with impact. So we aimed to examine what was working and not working as a way to help faculty improve future iterations of their courses.”

While the variability among the four College courses made general interpretations a challenge, the student assessments did reveal some commonalities that were not necessarily course- or instructor-specific. Among the key findings:

  • Students tended to conflate the teaching approach with the blended format, responding more to the teaching itself than to how specific online or blended elements worked.
  • Students appreciated the quality of the HarvardX materials, and most found them interesting and engaging.
  • For the most part, students spent roughly the same amount of time on homework and preparation for the blended class as they did for a traditional Harvard course.
  • Students valued the increased flexibility and ability to learn at their own pace, but still wanted in-person interactions with faculty and among themselves. They said that sections — small-group discussions outside the class ― were especially vital, enabling feedback, time for Q&A, meaningful collaborations, and a deeper sense of intellectual community.
  • The most common student complaint was that online learning opportunities were often redundant with in-class components, as faculty experimented with how to best use class time and encourage participation. In-class activities worked best when they were well-structured, such as when students were given discussion questions, problem sets, or worksheets in advance.
  • In any setting, students cut corners to save time, earn participation points, or get through required assignments or assessments. Many adopted efficiency strategies while watching the online lessons, causing some to integrate the materials in less-than-meaningful ways.

With the findings in mind, Bergeron’s research team developed a set of recommendations for faculty who are planning to blend existing courses, designing new ones, or are interested in critically assessing new teaching approaches.

Foremost is planning, which should address questions such as how time is spent in class or sections; the instructor’s role as facilitator; what form of content — media, lecture, or text — works best for what purpose; what physical environment best promotes learning objectives; how to ensure in-class engagement.

The study found that expectations about participation and time commitments should be well-defined across all components, especially those that are online. Grading systems for online components should also be transparent. Likewise, instructors must be clear about the nature of the course and why and how it is blended, as many of the students regarded the digital component as an attempt to supplant rather than supplement the in-class experience.

Lastly, digital materials and in-class activities must be evenly balanced. Finding the right blend of online elements, active learning experiences, and in-person interaction is an iterative process, and it’s not a matter of flipping a switch. Integrating new or additional learning activities without dropping activities from a previously designed course can be counterproductive.

The Bok Center has developed a blended learning support team and, in collaboration with HarvardX, will participate in the Faculty Academy, an intensive workshop sponsored by the Office of the Vice Provost for Advances in Learning that provides faculty with hands-on experimentation and expert guidance in emerging pedagogies.

Lue, a professor of the practice of molecular and cellular Biology who himself is developing an online learning module in biology, said he expects that faculty will use the study as a starting point as they think about how to integrate innovation, technology-driven or otherwise, into their own current and future courses.

“As we have seen with faculty who embrace new modes of teaching, the process of engagement is often the most important outcome, as it inspires faculty to rethink how they teach as well as consider integrating research and other assessment components they may not have done before,” he said.

His sentiments were echoed by research assistant Yiran Zhao and professor of education Andrew Ho, both at GSE, who ran a parallel study focused only on the Societies of the World course “China,” one of the first courses HarvardX created for the edX platform, and one of the first blended humanities courses offered at the College.

“Our study provided quasi-experimental estimates of the impact on learning of the transition from standard residential to blended classroom using test scores,” said Ho, the co-chair of the HarvardX research committee.

Like Bergeron, Ho and Zhao found that student preferences for the blended or traditional class format varied, and that the format had no significant impact on test scores. However, students did report that elements that enabled higher levels of engagement — whether online or in-class ― were of great benefit.

“We are at the first stages of understanding impact of blended classrooms and other new learning technologies, and there are no definitive takeaways other than faculty must be committed to ongoing and rigorous assessments over time, and that like online lectures, any new assessment tools must be given the same care in development,” said Ho.

For descriptions of some of the courses studied. Outside media can contact michael_rutter@harvard.edu for the full Bok Center report.