Executive Summary

This document summarizes the findings of the LEAD Center report entitled, "Evaluation Report #1: The Computer Research Association - Distributed Mentor Project (CRA-DMP)." This summary is primarily intended for those who have not read the entire report, and presents no new analysis.

1. Background

1.1 Evaluation Methods

We conducted structured, open-ended interviews with the mentor and student participants who participated in the 1994 and/or 1995 program years.

We interviewed ten out of the twenty-eight total 1995 student participants in the summer of 1995. Each student participated in two interviews: one at the beginning of her program and another upon the completion of her program. We conducted one interview with ten out of the twenty-five total 1994 student participants. (One of these students participated in 1994 and 1995.)

We conducted a single interview with nine out of the twenty-five 1995 mentors in the fall of 1995. Four of these mentors participated in the DMP in both 1994 and 1995. We interviewed ten out of the twenty-four 1994 mentors once in the fall of 1995.

The reader should note that qualitative and quantitative research methods differ not only with respect to data collection but with respect to analysis. Individual interviews allow the researchers to "get inside of" the experiences of these diverse participants. Data collection methods are as open-ended and subject-responsive as feasible to ensure that the experiences of the study participants, not the researchers, are reported. Likewise, analysis processes are fundamentally inductive to ensure that the participants' experiences shape the findings. In practice, this means that the researchers make every effort to at least temporarily suspend the ideas that structured their interview protocols. The analysis of interview transcripts is focused on determining what is most important to the participants. The primary analytical categories that emerge as the researchers process the transcripts are apparent in the table of contents. In contrast to survey methods, these methods do not yield precise, quantitative assessments of the proportion of participants holding pre-specified opinions. However, these methods provide extraordinarily rich information expressing the complexity of the lived experiences of the study participants.

1.2 The CRA-DMP

The National Science Foundation (NSF) awarded the Computer Research Association (CRA) funds with which to support the Distributed Mentor Project (DMP) for undergraduate females in computer science and engineering (CS&E). The intent of the award was to increase the number of women entering graduate school in CS&E by involving them in research. Students from around the country who participated in the program spend approximately ten weeks involved in a research project at their mentor's university. Twenty-five students and twenty-four mentors were matched in 1994 and twenty-eight students and twenty-five mentors were matched in 1995.

2. Evaluation Findings

2.1 Research Question 1

The primary questions addressed in this evaluation are:

Are there measurable effects, positive and/or negative, resulting from the Distributed Mentor Project? If so, What kinds of qualitative effects are experienced by the DMP students and can patterns in mentee/mentor interactions be ascertained and associated with the measurable effects of the program?
We have addressed these research questions by summarizing the student section of the report. The italicized text reflects the evaluator's point of view.

Motivations for participating in the program

Almost all of the student participants entered the DMP with the expectation that their experience would give them the knowledge with which to make decisions about their future career paths. Many mentioned that they were particularly interested in finding out if computer science and engineering (CS&E) was "right" for them by working in the field. In addition, many hoped to learn about graduate school, research and academic positions through the DMP and planned to make decisions about whether these were paths they wanted to pursue in the future. This indicates that the students selected to participate in the DMP are utilizing the program in accordance with its goal of encouraging undergraduate women to consider and pursue graduate studies in CS&E.

The impact of immersing students in an academic environment

In the DMP, students were immersed in an academic environment through the following processes:

Most students entered in the program with little understanding of graduate school and no experience with research. These students expressed that "living the life" of a graduate student helped them to become more sophisticated about CS&E because they developed a more complete understanding of graduate school, the research process and the faculty role within the university. Many students stated that by observing and interacting with graduate students and faculty and doing a "hands-on" research project, they:

Defining interests in graduate school and an academic career in CS&E

We came to realize that by immersing students in a graduate and research environment, the DMP is preparing these students to make knowledgeable career decisions. By giving students an opportunity to experience aspects of graduate school life prior to making their actual career decisions, the DMP is enabling students to make informed decisions about a career and about graduate school:

Many students identified with and enjoyed the people and experiences they encountered and came to realize that graduate school was "right" for them. With their more sophisticated understanding and confidence about their abilities these students may be more likely to succeed in graduate school because they are better prepared for an academic environment.
Some students realized that they were either not interested or not prepared to do research and decided not to attend graduate school following their undergraduate education. The students who felt unprepared for graduate school stated that they lacked a well-developed interest in some area of CS&E and did not feel that it was appropriate for them to pursue an advanced degree at this time. We posit that this is not a negative outcome of the program in that it prevented students from making a potentially inappropriate career choice.
Most students felt that their observations and interactions with faculty and their experience with research were accurate indicators of a career in academia. As a result, through their experience in the DMP, many students came to understand whether they would like to work in academia:

Some students commented that the role of the faculty member appealed to them and as a result, they were planning to attend graduate school in order to become a professor.

Some students enjoyed their work in an academic environment, but were still ambivalent about choosing academic CS&E as a career. Many of these students stated that they had not worked in industry and had no basis for comparison and so were hesitant to decide upon a career in academia. Some of these students commented that although they enjoyed the freedom and flexibility of an academic schedule, they also had reservations about working in such an unstructured environment.

Some students came to realize that an academic career did not appeal to them. Many of these students expressed that they needed more external direction and structure than a career in research would offer them.

Evaluator Question

If students encounter serious problems in the DMP, do they allow this negative experience to be the defining factor in whether they attend graduate school? We are defining "serious problems" as students feeling that their mentor excluded them from professional and personal interactions with her and as a result, encountered problems with the program itself.

Due to the small number of students who had negative experiences in the program, we cannot draw definitive conclusions about this question. Despite this small sample, their experiences raise important questions and issues about the implementation of the DMP and its effects on the students. In subsequent years of this evaluation as more data is gathered, we will be able to more completely explore the relationship between program implementation and its effects.

Students as "part of a larger whole:" Including students in the research process and graduate school environment creates a more successful DMP experience

Our interviews indicated that involving students in a collaborative research process was critical to their having a successful experience in the program and also achieving the outcomes described in the earlier section. Virtually all students emphasized the importance of doing a project that was related to, and had an impact on, their mentor's research group. These students expressed that their project felt like "real" research precisely because it contributed to the research of someone professionally involved in CS&E and for many, this experience helped them to realize that they were capable of contributing valuable information to the field.

Moreover, students were more motivated and inspired to do research when their mentor treated them as valuable members of the research group, capable of providing input about their research. A few students said that they felt excluded from the collaborative research process as a result of being assigned to a project that was irrelevant to their mentor's research or not working with members of the research team. These students stated that they felt isolated from the research process and became less motivated to work on their project.

The multi-dimensional role of the mentor

The mentor played a critical role in facilitating a positive experience for the students during their participation in the program as well as through acting as a role model and a professional resource.

Defining the working relationship

Most students expressed that they knew what their mentor expected of them and were able to function within that framework. However, a few students felt unclear about the protocol for interacting with their mentor or didn't feel that they had an understanding of their mentor's expectations.

We came to understand that students viewed their mentor as an authority figure and were uncomfortable, at least initially, with approaching her with questions and concerns. A number of these students lacked confidence about their skills and abilities, and were reluctant to initiate interactions and discussions with the mentor. As a result, the students placed the responsibility on the mentor for defining the nature and frequency of interactions during the program. Thus, the mentor can reduce her student's anxiety about their interactions, both professional and social, by defining when and how often they would meet, what was expected of them in these meetings, and the times at which the mentor would be available for questions (outside of the regular meetings). If the student worked with graduate students, the mentor also needed to provide a clear delineation of their working relationship.

"Making time" for the students

Most students stated that their mentor was available throughout the program to meet with them and address their questions. A few students expressed that their mentor either was not available or was not a good resource. When students felt that they didn't have much contact with their mentor, they were disappointed with the program and questioned the value of the mentoring aspect of the program. In fact, when students perceived their mentor as too busy to meet with them, many felt as if their presence, rather than being helpful, was more of a distraction.

Designing and implementing a project: Giving students freedom within limits

Many students entered the DMP with no experience in how to conduct research and lacked enough knowledge and experience in CS&E to define a project independently. As a result they needed the mentor to frame a project or series of projects for them that provided an overall structure. Yet within this framework students wanted to have some input in helping to define their project as well as a choice in how they would proceed in conducting their research. However, it is important to note that students who did not receive enough structure and guidance, felt lost and without direction.

Orienting students about the field of CS&E

Many students commented that their mentor introduced them to the culture of academia by giving them professional advice and direction. They looked to their mentor as a reliable resource to address their questions about graduate school, research and academic life.

"Modeling" life in the field

Although the students were aware that women could succeed in CS&E, they had little or no contact with female faculty members. By observing and working closely with their mentor, many students said that they were able to identify with and relate to her experiences as a female in CS&E and viewed her as an accurate role model for what they could accomplish in their professional life. Many students commented that prior to participating in the DMP, they feared that an academic career in CS&E would be too time intensive to be able to focus on other interests and said that their mentor showed them that it was possible to be successful in CS&E and have outside interests. These students stated that this interaction with their mentor provided encouragement that they too could succeed in the field.

Logistical problems within the DMP

Many students expressed frustration about organizing various aspects of their living arrangements at the mentor's university. Because the participants were not "official" students at the host university, they were often denied housing and access to other components of university life unless their mentor intervened. Another issue that arose in our discussions with students was that some had difficulties receiving their checks from the CRA on time.

2.2 Research Question 2

A second question in this evaluation is:

What, if any, special problems and/or satisfactions do faculty members experience as mentors in this program?
We have addressed this question by summarizing the mentor section of the report. The italicized text reflects the evaluator's point of view.

The Goal and Structure of the DMP

Virtually all mentors stated that the focus of the DMP was to introduce undergraduate females to research and a graduate school environment in CS&E with the intention of encouraging these students to consider going to graduate school. Many mentors expressed that although they encouraged students to pursue graduate studies in CS&E, they did not pressure these students into going to graduate school. Rather, they attempted to provide an introduction to graduate school and research as well as reinforcing confidence about their capabilities of succeeding in graduate school in CS&E. With this information and encouragement, the mentors stated that the DMP students would be able to make a more informed choice about attending graduate school.

All of the mentors that we interviewed commented that the DMP provided an excellent opportunity to both encourage women to consider graduate school in CS&E and prepare them for a career in CS&E because it immersed them in a research environment and "plugged" them into a network of CS&E professionals:

The mentors said that by immersing students in a research environment in which they interacted with faculty and graduate students and did "hands-on" research, students developed a better understanding of graduate school and research. They also stated that students developed confidence about their abilities to succeed in graduate school and research through "living the life" of a graduate student and doing "hands-on" research.

Most mentors that we interviewed commented that a positive outcome of the DMP was that it matched students and mentors from different universities. Given the low numbers of women in many areas of CS&E, many mentors viewed the matching as a good way to create connections between female CS&E professionals and to provide role models for undergraduates that they may not have access to at their own institution.

While the goal of the DMP is to increase the number of women entering into graduate school and research careers in CS&E, we came to realize that the program may also function to increase students' success in graduate school. This is because students became more sophisticated about the field through: developing their networking skills, making connections with professionals in CS&E, realizing possible areas of specialization, and increasing confidence about their abilities.

The Mentors' View of the DMP Students

Many mentors stated that they were impressed with their students' abilities and initiative but cautioned that they needed to spend time introducing their DMP student to the research topic due to the fact that undergraduates have limited background and research experience in CS&E. In fact, when mentors were asked to give advice to faculty considering participating in the DMP, many commented that the faculty member should be prepared to invest time in the program.

Lack of background may affect program success

Some mentors stated that their student lacked the background necessary to do a research project that would be interesting and productive to either the student or the mentor. In fact, some mentors commented that sophomore-level students may not have the necessary background and sophistication to do research in the ten-week time frame of the program. These mentors expressed that only junior level students should be admitted into the program, because they would have taken the upper-division classes which would give them the background and sophistication to do projects that could interest both the mentor and the student.

Evaluator Question

Should there be a minimal background requirement of students?

Unmotivated students hinder success of program

A few mentors stated that their student was unmotivated and applied minimal effort toward her research project. When mentors perceived their student to be unmotivated, they viewed their experience in the DMP as a "waste of time" and as not being beneficial to either the student or the mentor.

Inevitably there are students who experience difficulties in any program. In some cases, these problems result from personals issues and little can be done to alter the student's experience in the program. However, in some situations, the problems may result from circumstances involving program implementation. As remarked above, a few mentors described their students as unmotivated. Our interviews indicated that there was a disjuncture between the student and mentor perceptions of the student's intentions. While mentors characterized their student as unmotivated, the students expressed that they wanted to contribute, but felt lost and without direction. These students lacked the confidence to assert themselves and felt shy about asking "basic" or "obvious" questions of the mentor. The students thus retreated from interaction with the mentor and this was misinterpreted as a lack of motivation and interest in working on the research project. We came to realize that initially these students may need more structure and guidance from the mentors in order to proceed in the research and develop confidence. Through gaining knowledge and experience in the programs such as the DMP, these students can develop independence and goal orientation.

Receiving assistance in their research: Two different mentor expectations of the DMP

There were two differing opinions among the mentors when discussing the importance of receiving some research benefit from the student while participating in the DMP. Many mentors participated in the DMP with the intention of volunteering their time to help an undergraduate and viewed any contribution to their research as a bonus. Some mentors viewed the experience in the DMP as an investment of time and felt that some assistance from the student on their research program was a critical part of their participation in the program. These mentors expressed that they would like to have some input in the process of selecting students for the program in order to ensure that the students had the skills and background to complete their project.

Through trying to ensure that the students have the proper background needed to conduct research in the mentor's area of expertise, the mentors are attempting to create an experience that may be beneficial for both mentor and student. However, this could be problematic. As noted earlier, students entered into the DMP with the intention of making decisions about future career paths. Thus, many viewed the DMP as more than a research experience; they viewed it as a way to learn about graduate school and academic life. In fact, through their experience in the DMP, many students were able to decide if a career in CS&E was "right" for them. If the selection process is defined solely by student background, this may prevent the program from reaching students who are at a critical juncture in a career decision and would benefit from a valuable experience such as the DMP.

Evaluator Questions

How important is the student background in CS&E to the success of the DMP for the student? for the mentor?

Would allowing mentors to participate in the selection process fit with the goal of the program?

The Role of the Mentor in the DMP

In our interviews, we asked the mentors about their definitions of "mentoring." Many responded by describing "mentoring" as a process of assessing the individual needs of the mentee and then attempting to provide for those needs. As a result, we came to understand that the role of the mentor was multifaceted and dependent upon the individual student. The undergraduates entered into the DMP with little understanding of graduate school, the nature of research or how to conduct it, the academic community in CS&E and the life of a female faculty in CS&E. Our interviews with the mentors indicated that the main roles they discussed paralleled the students' needs: being a role model, providing professional guidance, and guiding students in their research project.

Being a role model: Providing an example of a successful woman in CS&E

Most mentors expressed that part of their role in the DMP was to provide a model of a "real life" example of a CS&E female faculty member. Many commented that this modelling took two different forms: students could observe the mentor in her daily activities, and the mentor could explicitly discuss and provide guidance about the types of experiences she encountered as a female in CS&E. Many mentors mentioned that, although a discussion of their experiences was beneficial to the student, the largest impact on the student occurred by observing the mentor on a day-to-day basis.

Guiding students in their professional development

Virtually all mentors stated that an important aspect of their role was to guide their student in her professional development. This included:

Guiding their student in her career choices. For many mentors, guidance involved introducing the student to new opportunities and encouraging her to take advantage of them. Many mentors commented that students were not always aware of their options or, if they were aware of them, did not have the confidence to take advantage of them.

Giving students an idea of the research process. Many mentors realized that their students had little or no conception of what research was about and, as part of their role, involved and discussed with their student the nature of the research process. As these mentors explained, this immersion involved introducing students to the day-to-day process of "doing" research, providing them with the context of how their area of interest fits in to the larger body of current research, introducing students to other people who can assist them, and giving them names of journals to which they can submit their research.

Orienting students to the professional culture of academic CS&E. Some mentors took their students to conferences and remarked that this was a way for the students to interact on a professional and social level with other CS&E professionals.

Setting up a framework for the student's research

Many mentors prepared for their student's arrival by setting up accounts for her on the local system, getting an office for her, and helping her to find housing. A few mentors suggested that the DMP provide a list of instructions of what should be done to prepare for a student's arrival.

All of the mentors we interviewed placed their student in an office in their building so the mentors could easily stop by the student's office. Many mentors commented that they did not have a set meeting time with their student, but made sure that they were in contact with their student at least a few times a week. These mentors felt that the almost day-to-day interaction precluded the need for weekly meetings because they could follow the student's progress on her project more closely. Other mentors set up weekly meetings for their student. Many of these mentors commented that the structure of weekly meetings provided a framework and guidance for the student learning and motivated the students to work on the project. Some mentors commented that weekly meetings helped to ensure that their student were enjoying their project and was also working hard on it.

Guiding the student in her research

Almost all mentors stated that undergraduates did not have enough experience with research to be able to independently choose a research topic and direct their own research. Thus, all mentors provided the direction for their student's research in the form of defining a problem for them to solve, or giving them ideas for a future topic to explore. While defining the structure of the project and providing overall guidance, most mentors expressed that they encouraged their students work out the details of the project. These mentors expressed that it was important for the students to have freedom to do explore solutions within the framework of the project in order to get some experience with research. Many mentors struggled to find the balance between guiding the student and letting the student struggle on her own and emphasized that they wanted the student to be challenged by the project, but not get so frustrated that she would give up.

The role of the graduate student in assisting the students in their research

The mentors who had graduate students working with the DMP students mentioned that their roles differed from the graduate student. While the graduate student answered the day-to-day technical details of the project, the mentor provided the global picture of where the project was headed and the context of how the project fit with other research either in the department or in the research community at large.

Strategies in creating a successful DMP experience

Creating a well-defined and doable project that challenged the student and was part of the mentor's research program

Most of the mentors we interviewed commented that having a well-defined and doable project that challenged the student and was part of the mentor's research program was essential to having a successful DMP experience for both the student and mentor. These mentors stated that by successfully completing a difficult research project, students would develop more confidence about their ability to do research and realize that they could contribute something of interest to the research community. These mentors framed the project around their students' background and abilities and attempted to create a project that was challenging enough for the student to be interested, yet not too difficult that she would get overly frustrated.

Many mentors stated that, given the short time frame of the program, it was important to be able to assess their student's background accurately so that the student can work on a project without a large degree of initial preparation time. Some mentors commented that their student was unable to finish her project because they had overestimated her background knowledge and needed to spend time at the beginning of the program providing information. These mentors expressed that they should have contacted their student before she arrived to determine her background in order to frame a project more suited to her needs.

In addition, many mentors suggested having multiple projects prepared at the beginning of the program. These mentors expressed that the student would be more motivated when she was able to choose a project that interested her and having multiple projects would prevent the student from getting too frustrated or bored by one project and allow her to learn different material.

Immersing students in a research community

Another strategy for a creating a successful DMP experience was to immerse students in a research community that included the mentor, graduate students, undergraduate students, and other faculty. Many mentors stated that involving a student in a research community would benefit the student by providing multiple resources and social interaction while also benefiting the mentor by decreasing the time demands of mentoring an undergraduate.

Evaluator Question

Should the CRA-DMP require that the mentor have a community of graduate students with which the students can work over the summer?

Receiving recognition for their participation in the DMP

In the 1994 and 1995 program years, the mentors received neither monetary nor professional recognition for their participation. Many mentors commented that they would have preferred to receive some sort of recognition for their participation in the DMP in the form of:

A letter to the department chair or the academic Dean. Many mentors commented that the CRA should send their department chair or an academic Dean a letter about their participation in the DMP. These mentors felt that such a letter would serve a dual purpose: it would inform their department of their participation in the DMP and also may assist the student in her interactions with the host university.

Publicity for their participation in a journal. A few mentors mentioned that they would have liked to have been honored for their participation in the DMP by having their name listed in some publication, like the Computing Research News. These mentors felt that this publicity would benefit the mentor, her institution, and the DMP.

Funding which may influence more faculty to participate. Many mentors commented that if the DMP provided some sort of funding, more faculty may be inclined to participate. Many of these mentors stated that the funding would not have to be directed toward the mentor. The funding could be applied to cover expenses of having the student, or to the department, or to be a stipend for both the student and mentor to attend a conference.

Table of ContentsNext SectionBack to the CRA-DMP Home Page