The Capstone Seminars in the Department of Humanities:
A Report on Present Directions and Future Opportunities

by Norbert Elliot

Background

Originating in 1996 as part of the new General University Requirements in the humanities, the capstone seminars were first required for undergraduate students who enrolled in NJIT in the fall of 1997. Since that time, the seminars have become a central feature of the undergraduate educational experience at NJIT. In the fall of 2004, the Department of Humanities hosted eighteen capstone seminars with an enrollment of 296 students. With classes capped at seventeen students, the seminars witnessed only a 7.4% withdrawal rate. In the spring of 2005, the Department will host twenty seminars.

In their orientation, the Capstone Seminars were modeled after the Humanities, Arts, and Social Science general university requirements in category D at Massachusetts Institute of Technology. The HASS-D categories, designed to offer a structured and intellectually coherent learning experience, are grouped into five areas:

• Literary and Textual Studies (subjects devoted to the interpretation of texts, to literary traditions, and to genres);
• Language, Thought, and Value Subjects (courses that focus on the study of concepts, principles, and modes of expression basic to efforts at understanding individuals and their place in the universe);
• Visual and Performing Arts (courses drawn from music, the visual arts, drama, dance, and film, some historical and analytical, others more directly concerned with the creation of art);
• Cultural and Social Studies (courses that approach human societies by examining forms of social, cultural, economic, political, and religious organization and behavior;
• Historical Studies (courses that investigate the development of people, institutions, or countries over a considerable period of time).

Indeed, the titles of the capstone offerings at NJIT reveal that these categories continue to be retained:

• Literary and Textual Studies: Multiculturalism and Cinema; The Bible as Literature; Tales of King Arthur; The Literature of War and Peace; Romance in the Middle Ages
• Language, Thought, and Values: Religion and Culture; Mystery Writing; Literature and Film; Literary Trials and Courtroom Drama; Literature and Medicine; Shaw on Film; Literature and Opera
• Visual and Performing Arts: Theatre and Technology; American Musical Theater; Literature, Film, and the Performing Arts; Creative Production
• Cultural and Social Studies: Documentary Studies; Leadership and Technology; Technological Fixes and Social Problems; Evaluation of Social Programs
• Historical Studies: Nazi Germany; War and Misperception; International Relations

Unlike the MIT program in which students are admitted by lottery, however, the NJIT capstone program holds no barriers to timely graduation. The small average class size and 93% retention rate are fundamental evidence of the commitment of the Department and of the College of Science and Liberal Arts to provide a stimulating undergraduate experience for all undergraduate students, an experience that expedites graduation rates.

The Fall 2004 Meetings

The Department of Humanities continues a tradition that, in management terms, would be identified as continuous quality improvement. Within the Department, this approach includes the appointment of a course coordinator who, with colleagues, convenes meetings, identifies and reinforces curricular goals, assesses the achievement of those goals, and sets future directions. In the Fall of 2004, the instructors who have written this report participated in three goal-oriented meetings:

• Meeting 1 (Reflections):
  • To understand and promote the significance of the capstone seminar experience to the NJIT community
  • To review the value of the content of the capstone seminars
  • To describe the value added to undergraduate education through the capstone seminars
  • To identify ways for students to make the seminar a significant experience—a culminating event—in their undergraduate education
• Meeting 2 (Assessment):
  • To identify meaningful assessment activities for our students and ourselves
  • To discover ways to share the assessment results within the NJIT community
• Meeting 3 (Advertisement):
  • To identify strategies to advertise the significance, content, and assessment of the seminars in a variety of ways

In the meetings and in continuing office visits, hall chats, and extended e-mail discussions, we addressed the curricular goals of the seminars, studied the ways the seminars are assessed, and established future goals for our students and ourselves.

Curricular Goals

During our first meeting, we turned to the stated and actual goals of the capstone seminars. We recalled, generally, the overall goals of our model, the MIT curriculum and its devotion to addressing the following:

• skills in communication, both oral and written;
• knowledge of human cultures, past and present, and of the ways in which cultures have influenced one another;
• awareness of concepts, ideas, and systems of thought that underlie human activities;
• understanding of the social, political, and economic framework of different societies;
• sensitivity to modes of communication and self-expression in the arts.

During the meetings—as well as in subsequent meetings during the semester—we recalled and reinforced the goals of our seminars:

• To treat each student as a unique individual capable of humanistic appreciation of cultures and their diverse complexities, capable of classroom leadership through strong interpersonal skills, capable of adaptability to emerging ideas, and capable of an interest in continual learning
• To engage that student in the course content through seminar techniques (e.g., oral and written explication and application of ideas; individual and group review of those ideas; and reflection of instructor's own experience with the course content)
• To improve the communications skills of each by means of writing-as-process techniques—drafting, peer and instructor review, revising, and submission—techniques which reinforce engagement with the course content; and
• To improve the communication skills of each student by means of oral presentation techniques—student-led discussion topics, informal presentations, and formal presentations— techniques which reinforce engagement with the course content

As is the case with our MIT HASS-D model, we recognized that the seminars we taught were appropriate for students who may never take another subject in that area of study and, thus, served as a means of broadening student experience. However, we found that we at NJIT were working toward a collaborative model in which the students were often responsible for in-depth presentations of material. Indeed, the instructors, rather than taking a didactic role, worked to establish a setting in which they served in a mentor role, often reflecting, for example, on their roles as researchers, writers, and performers. Clearly, the eighteen NJIT seminars were more student-centered in their pedagogical approach than the endless roster of MIT courses—offered within countless departments by innumerable faculty—to fulfill the HASS-D requirement.

Such careful attention to the communication skills of students—along with attention to the development of collaborative and leadership skills—would serve to augment any of the majors offered by NJIT. If the subject matter of the seminars was novel, the emphasis on communication was traditional, aligned with the results of employer surveys telling us that excellent communication skills are essential to successful career development. Thus, the capstone seminars provided skills transferable to all endeavors requiring analysis and critical thinking.

Assessment

In our second meeting, we turned to the assessment of our seminars. It is, we realized, one thing to craft eloquent language about learning outcomes and quite another to demonstrate these outcomes in an empirically valid and reliable fashion.

To begin, we summarized the results of our student surveys. As is the case of each of the GUR cohorts hosted by the Department of Humanities—Freshman Composition, Cultural History, Technical Writing, and the 300-Level Electives (in Literature, History, Philosophy, Theater, and Writing)—the Capstone Seminars have always received high student evaluations. Below, for example, are summary statistics for nineteen sections of the spring 2004 student evaluations:

Table 1. Student Evaluations, Spring 2004

The correlation (r=8.31) between the range and the median suggests that the two variables of interest—the content of the course and the achievement of the instructor—are closely associated. We understood this finding to be evidence of student satisfaction with the seminars.

Such self-reporting measures have, however, been questioned by the agencies that accredit programs offered by NJIT: The Middle States Commission on Higher Education, The Accreditation Board for Engineering and Technology, The National Architectural Accrediting Board, the American Assembly of Collegiate School of Business, and the American Public Health Association. Since the capstone seminars were launched in 1996, outcomes assessment of student performance—again, ours is a total quality management approach—has been maintained.

In the fall of 2003, for example, the students presented what they determined was their best work to their instructors. A sampling plan was then devised to yield a 95% confidence interval, and instructors met to read the papers of those identified students. A score of 12 (scores of 6 by two independent readers working with an established scoring rubric) was determined to be the highest possible score; a score of 2 (scores of 1 by two independent readers) was determined to be the lowest possible score. The results of that reading are given below:

Figure 1. Capstone Best Paper Reading, Fall 2003

Table 2. Descriptive Statistics from Capstone Best Paper Reading, Fall 2003

As Figure 1 reveals, the student scores were well distributed. And, as Table 2 demonstrates, the mean score was well above the 7.0 traditionally associated with the cut score—the lowest acceptable score of a 12 point scale—associated with such evaluation methods. A comparison of the fall 2003 scores with the scores of students from the previous semester's reading (in the spring of 2003 when a mean score was reported as 7.413) revealed that there was no significant difference in a comparison across two semesters (t=.9431; p=.1737). This stability in mean scores—combined with the high correlation of adjudicated inter-reader reliability (r=7.917 in a discipline that establishes .6 as an acceptable measure of inter-reader reliability)—suggests that there is a great deal of stability in the curriculum within the capstone seminars.

Beyond best paper readings, portfolios of student work are also evaluated in the GUR offered by the Department of Humanities. Portfolios of student work, instructors have found, enhance their understanding of the overall achievement of students as expressed in work completed throughout the semester; that is, in measurement terms, portfolios increase the construct validity of the assessment by allowing instructors to view students longitudinally across fifteen weeks of the semester. Table 3 below provides summary data from the fall 2003 portfolio reading:

Table 3. Portfolio Reading, Fall 2003

As was the case with the best paper readings in the capstone seminars, the inter-reader agreement in each GUR cohort (i.e., Freshman Composition, Cultural History, Technical Writing, and the Capstone Seminars) exceeded the .6 correlation coefficient for each of the variables assessed (i.e., critical thinking, global perspective, drafting, textual citation, oral presentation, collaboration, and overall score.) Thus, again, we note great stability in the way instructors evaluate student work across a diverse range of seminars.

The overall portfolio score for the capstone seminars (8.93) was, as Table 3 documents, the highest among all of the GUR offerings—each of which exceeded the cut score of 7. In addition, the instructors were pleased to note the high mean score recorded for critical thinking (9.22) and drafting (8.79). Here, the instructors found outcomes-based evidence that the goals of the capstone seminars were being met. Through their demonstrated critical thinking skills, students were proving that they were indeed capable agents (for example, of humanistic appreciation of cultures and their diverse complexities) by writing papers that extended beyond mere summary into in-depth analysis. As well, the seminar techniques appeared to be working: students were clearly drafting and revising their work in a writing-as-process fashion.

Yet, we realized that our assessment system was inadequate when it came to evaluating the oral performance of our students. In the portfolios, the evaluators had looked for evidence of oral presentations—student notes, for example, or Power Point slides. We soon realized that the low mean score on the variable (6.39) was due to the inadequacy of the way that variable was being assessed. Yet, we also realized that the score could just as easily have been interpreted as an inadequacy in the curriculum. We knew, therefore, that we should address the complexity of accounting for our desire to improve the communication skills of each student by means of oral presentation techniques. We also knew, however, that empirical validation of this variable was beyond our means. Performing portfolio evaluations of student writing was time-consuming, and planning a reliable evaluation of student oral presentations—having each student in some nineteen sections of seminars present in front of two instructors during, say, the final exam week—was beyond our ability. We agreed, then, to maintain student-led discussion of topics, informal presentations, and formal presentations as a goal of our seminars, but we admitted that demonstration of student achievement of this goal in a valid and reliable fashion was simply beyond our resources.

Addressing the marginal score of 7.76 on the citation variable, however, was within our resources. Although within an acceptable range, the score suggested that students were not performing text-based research by using the resources of the Van Houten Library and Dana Libraries. (Indeed, it could be argued that sophomores in Cultural History appeared to have better research and documentation skills.) We were concerned that if students did use documentation formats such as those offered by the Modern Language Association or the American Psychological Association, then perhaps the students were not writing papers that were sufficiently researched. We agreed that researched assignments employing either MLA or APA documentation formats would be a requirement of each seminar.

Future Directions

In our third meeting of the semester, we set directions for the spring 2005 offerings. First, we focused on the need to have a common set of goals for the seminars while maintaining the uniqueness of each seminar. We came to realize that the general portfolio we used in the Department—a collection of all the work the student did within a given course—could be narrowed in scope in the Capstone Seminars. The new system, termed a tailored portfolio, would be designed by each student to include the following:

• A reflective statement preceding each assignment included in the portfolio. Here, the students would provide details of the assignment, reflecting on the origin of the assignment, the process by which it was completed, and its value. Such a statement, we believed, would allow us to understand more about our ability to help students achieve leadership, prove capable of adaptability to emerging ideas, and demonstrate an interest in continual learning. As well, the reflective statement would allow us to determine the extent to which we were truly engaging student in the course content through seminar techniques.
• A descriptive statement capturing the seminar. This assignment would be a final, extended essay written at the end of the course. In it, the students would provide a kind of meta-narrative about their experiences in the course. Again, this assignment would allow us to understand more about the ways the seminars were serving our students.

If these two tasks—the reflective statement and the descriptive essay—were to be common, then the instructors would work with each student to produce a portfolio that was representative of the work in the course. For example, in theatre seminars that required an extensive journal to be kept, that journal would become part of the portfolio. In history seminars that required an extensively researched essay, an annotated bibliography would become part of the portfolios. In seminars such as documentary studies, students would post their work in a web site using, for example, streamed media to showcase their documentaries. Thus, the instructors felt, the tailored portfolio would enable students to assemble their work in a fashion that was reflective of the content and design of the course.

In the spring 2005 semesters, the following instructors volunteered to field-test the two new assignments and the new portfolio techniques: Charles Brooks, Dennis Donahue, Norbert Elliot, Doris Fleischer, James Lipuma, Michele Rittenhouse, and Karl Schweizer. At the end of the semester, these instructors will report on their experiences and offer their newly designed portfolios, including the new tasks, for assessment.

Conclusions

As we write, we are beginning to prepare for the next portfolio assessment on January 14. Our Chair has invited the university president, deans, and provost to the reading. "Portfolio assessment is a time- and energy-consuming process," Bob Lynch wrote in his invitation, "which we perform at the end of every term. Whenever ABET and Middle-States teams visit, we summarize what we do and show them sample portfolios. They are invariably impressed. We want you to know it too, not by our testimony only, but by witnessing, at least for a short time, the effort that goes into our GUR program assessment." The effort that goes into the design and execution of the capstone seminars is, indeed, extraordinary. It is not immodest to say that our efforts demonstrate that our curriculum meets and exceeds that of our MIT model.

In the future, we plan to retain the self-governing nature of these seminars. Those who do not wish to participate in our collegial efforts to design and assess the curriculum will not be invited to teach the seminars; those who do will find a community of teacher-researchers dedicated to serving a diverse group of NJIT students.

NJIT has a very complex upper-division student body. Previous formal studies of those enrolled in the Technical Writing GUR cohort revealed that junior level transfer students often possess writing skills comparable to freshmen at our university. In addition, a substantial number of first-time, full-time NJIT students tell us that English is not their most proficient language. And, articulation agreements with Rutgers University-Newark allow students to enroll in courses that are not known to be committed to the NJIT writing-as-process pedagogy. In light of these three complexities, the capstone seminars serve as a certifying agent for our university. The students who succeed in these seminars have demonstrated that they have met the demanding goals of a well-designed curriculum.

Really, of course, such goals also reflect the qualities of an educated citizenry. We stand ready to ensure that these worthy seminars, so critical to each NJIT undergraduate student, are offered and assessed within an environment of excellence.