Standard 2. Assessment System and Unit Evaluation

The unit has an assessment system that collects and analyzes data on applicant qualifications, candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.

Significant Evidence: Performance-based Assessment Transition Plan, Performance Assessment Plan for Administrative and Supervisory Preparation Programs, Reading Specialist Transition Plan, Counselor Education Performance-based Assessment Plan, Instructional Technology Performance Assessment Plan

Element 1: Assessment System

During the 2000-2001 academic year, program faculty did an NCATE 2000 scan to begin studying the standards, determining what we were doing that fit the standards, and what needed to be done (see NCATE 2000 Scan). These program discussions were brought to the Center Advisory/NCATE Steering Committee for determining next steps (see minutes of Steering Committee, February 2001).

Initial Programs: Admission, Continuation, and Completion

Following the NCATE 2000 Scan, an analysis of syllabi and professional and content standards began. Syllabi for core professional courses were submitted to programs; core professional studies faculty analyzed their courses in relation to INTASC, ISTE, and state competencies. Program faculty did the same for the professional studies courses in their specific content areas. Then program faculty compiled program specific matrices that identified the courses in which INTASC standards were being addressed in some way and the courses and degree to which ISTE standards were being addressed. (Each program has both matrices in the “electronic exhibit room” under Programs. Click here for an example, the English Education INTASC matrix. Click here for an example, Fifth-year Elementary Education ISTE matrix.)

Faculty filled out a form for each course they taught, detailing the kinds of performance tasks they expected of candidates in that course and which INTASC standards the tasks addressed. They also indicated whether the tasks were assessed and, if assessed, whether or not a rubric was in the syllabus. These materials were valuable to faculty who had to prepare Program Review documents for their professional organization standards. These materials also formed the basis for program discussions with faculty in core courses as to which tasks fit program standards best and which tasks would need to be required in particular courses regardless of who taught the course and the need for rubrics for assessment consistency.

The next step was for each program to develop a Program Performance Assessment Matrix that indicated all the different types of performance assessments occurring throughout the courses and experiences in each program. They also decided which tasks across each program that would serve as “data points,” i.e. these “D” tasks scores would be collected to demonstrate candidate progress and level of performance. These tasks were selected to meet specific professional standards as well as national, state, and unit standards and reflect the conceptual framework. They also provide multiple assessments of knowledge, skills, and dispositions across each program. (Click here to see an example, History and Social Science Performance Assessment Matrix with linked “D” data points to tasks in the specific syllabi and a code for how the tasks are tied to the conceptual framework.)

From the Program Performance Assessment Matrix, an Excel database was designed for each program (Program Performance Assessment Database – PPAD). Because most programs operate as cohorts, candidates are entered together in a database targeting their completion date. Program advisors, with staff support, manage the collection of data points.

An online Data Points Assessing Candidate Performance database was set up specifically for collecting scores on tasks in core professional studies courses (password protected because it includes candidate information). These courses are taught by a number of faculty, but certain tasks and procedures have been regularized across the courses. The Center manages the entry of candidates into the online Data Points system. Support staff access tasks scores and fill in Program databases (PPADs). Program advisors use the database for monitoring candidate progress. (Programs will have the option of using the Data Points online “collector” for their program tasks, but all data will eventually be entered into an Excel database for analysis.) When candidates complete the program, the database is sent to the Center for Teacher Education for program analysis and for aggregation of core tasks across programs.

Another part of the assessment system is an online Professional Education Data Entry System (password protected) to manage tasks that have multiple parts to the assessment, for example student teaching evaluation and portfolio evaluation. The data from this system are downloaded in the Center for analysis. (The annual Faculty Scholarly Productivity Data Entry is also part of this online data collection system.)

Portfolio development began in 1999 as part of a PT3 Capacity Building grant. The process involved meeting with Arts & Sciences faculty to look at content standards (both state and professional organizations), a review of course syllabi, discussions of tasks and evaluation, format considerations – as the portfolio process evolved from a few all electronic portfolios in 2000 to a requirement for all secondary completers in 2002 and all elementary completers in 2003. The portfolio tasks and evaluation are linked to national, state, professional, and institutional standards. Program faculty review portfolios as they are in progress, and they review them prior to presentation. Secondary candidates present their portfolios to a three-person team, often consisting of at least one classroom teacher. Elementary faculty also have more than one reviewer for each portfolio.

As our assessment plan developed, program faculty were also reviewing the Transition Points across programs. Programs now are more consistent in having both a written application and an interview. They have continued to maintain rigorous review of transcripts to meet content specific requirements. This spring they have implemented an additional writing task for the application, specifically targeted toward a preliminary assessment of dispositions. After some discussion of early field experience evaluations, faculty found that their evaluation forms were similar and targeted primarily dispositions. A common early field experience evaluation was designed and will be implemented in fall 2003. Likewise, a common student teacher evaluation was adopted for all programs (except special education). However, while electronic portfolios are now required for all program completers, the evaluation instruments differ because portfolios are a major data source for program review documentation. Secondary portfolios have a common set of criteria as well as program specific criteria for address professional standards. Elementary portfolios address ACEI standards and are aligned with state, national, and unit standards as well.

All programs ask candidates to video tape their teaching and do a reflective analysis. (There are cases where school policy will not allow video taping; in those cases audio taping is done.) Portions of these tapes are used as streaming video in portfolios to demonstrate certain features of the candidate’s performance. Other candidates have used portions of their video tapes at state conferences, e.g. the science education candidates regularly present with their program faculty at the state science conference. These video or audio tapes are used for self-assessment, goal setting, an analysis of classroom interactions, and specific aspects of teaching, e.g. questioning techniques, giving directions, non-verbal communication. Because the camera is normally trained only on the teacher (school policy), the tapes rarely can be used to study student interactions. Instead, candidates use the observational techniques they learn in teacher inquiry to study individual and group behaviors.

During early field experiences candidates engage in conversations with students about their work: tutoring students, diagnosing problems, studying how students think about assignments. They also have opportunities to respond to student work. During student teaching, the emphasis on student work goes beyond the pre-test post-test assessment of impact on student learning. Because methods classes accompany both field experiences, candidates are able to study how students engage with assigned tasks, the range of engagement, and interpretations of tasks that they can bring back to their methods classes. The attitude we hope to engender about student work is that it is a window into a child’s process of thinking, not just a product of his/her thinking. These inquiry projects and reflections on student work appear in candidate portfolios.

An exit survey (Educational Benchmark, Inc - EBI) provides good data for overall program assessment, and this spring we will be able to modify the survey so that the data will be reported by program as well as overall. Follow up of completers is three-pronged: an initial survey in early fall to determine job placement, contact information, demographics of the placement, and feedback on program strengths and suggestions for improvement, followed by a survey sent to their principals at the end of one year of teaching with a candidate self-assessment on the same criteria. This survey was developed with input from the Professional Education Committee, which recommended a short, carefully targeted list. We decided to use the candidate outcomes from our Conceptual Framework.

Next Steps for Full Implementation of Assessment Plan for Initial Programs

Rubrics for portfolio assessment have evolved as faculty reviewed and refined the process each spring following portfolio evaluations. No reliability data have been collected thus far although discussions of criteria and application of criteria have taken place within program team evaluators. However, we need to establish reliability across programs using the same evaluative criteria, e.g. secondary programs, because those common data will be aggregated as well as used for program review documentation. That process will begin with the spring 2003 portfolio evaluations.

Next steps also include revision and/or modification of rubrics for all tasks but especially in core tasks that cut across all programs. As the rubrics are refined, then reliability of scoring needs to be measured and evaluated. Again, this is an important consideration when a number of faculty are teaching a given course that has core tasks assigned for collection. Professional education core studies faculty will continue to meet with program faculty to redesign tasks and rubrics that better address standards.

Evaluation of where and how dispositions are assessed needs continued discussion. We now have two specific points, the entry and during and after the first field experience, when dispositions are assessed. There are other points within specific classes where reflections and collaborative tasks provide windows into candidates’ dispositions. However, those opportunities are not systematically monitored for all candidates. Faculty need to review what is being done and decide on what will strengthen assessment of dispositions and ethics, e.g. using a normative assessment or a self-assessment protocol.

Additional questions will be developed for the EBI survey that initial preparation candidates complete as an exit requirement. Some categories of interest are the candidates’ assessment of their content preparedness regardless of where they received their preparation, perception of education faculty’s integration of technology for instruction, perception of candidates’ impact on student learning, and the perception of the usefulness of the electronic portfolio to document candidates’ meeting professional and content standards, among others.

This spring program faculty and clinical faculty will assemble in a workshop to evaluate the ways we are assisting candidates in documenting their impact on student learning and to develop multiple strategies that we can offer candidates. This workshop will be critical to developing a more systematic approach to documenting candidates’ impact on student learning while they are in the program. However, we need to develop a long-range follow up that will help us assess how well our candidates impact student learning in their own classrooms over time. (We find that even in the second year out, we are losing contact with many of our completers.) Our hope is that the new data base the Virginia Department of Education is developing to track teachers and student achievement, among other things, will assist professional education units with getting these data.

Entrance interviews need to be more systematized with a rubric for all interviews (some are currently being used) to score particular aspects of a candidate’s interview.

Advanced Programs

The Reading Specialist Program set up a Program Performance Assessment Matrix with an Excel data base similar to those in the initial preparation programs. There is also a transition timeline accompanying the plan.

Administration and Supervision Assessment Plan describes in detail the admission, continuation, and exit criteria and data collection for program assessment. Special Education Administration and Supervision candidates complete the program with other administration and supervision candidates and in addition meet doctoral requirements of the university.

The Counselor Education Assessment Plan for initial preparation of school counselors describes the admission, continuation, and exit criteria. There is also a Program Performance Matrix with an Excel data base for collecting performance data across the courses and experiences in the program. The Counselor Education uses video tapes extensively for candidate reflection, self-assessment, and tracking competencies.

The Instructional Technology Program Assessment Plan details admission, continuation, and exit performances and a system for collecting the data.

Element 2: Data Collection, Analysis, and Evaluation

The Center for Teacher Education is responsible for almost all data collection associated with initial preparation programs and candidate performance as well as the data generated for AACTE, NCATE, Title II, and state reports. Advanced programs are managed by program faculty, department support staff, and the Center staff when licensure and endorsement is involved.

The Center maintains databases of all candidates by program, with data including contact information, Praxis I and II scores, field placements, and location of first teaching position, among other things. Hard copies of files that include transcripts and licensure application are also kept for five years. The Center developed the Program Performance Assessment Databases (PPADs) and will manage the analysis of that data. (Click here for Fifth-year Elementary Program Performance Assessment analysis.) The Center will upload candidates into the data base and also assist support staff as they implement the online database, Data Points for Assessing Candidate Performance.

The Center also administers the EBI survey to program completers. EBI will begin with this spring’s data to analyze responses by program as well as across all programs. This analysis will assist us in responding to the state reporting protocol, which requests that data be presented programmatically. (See 2002 EBI Survey results.) EBI will present 2003 data as well as a cumulative report.

The Center developed and manages the online Professional Education Data Entry System for collecting multiple response data. This system includes a Faculty Scholarly Productivity database, in which faculty enter the number of specific types of scholarly activities they have engaged in that year. Faculty enter their responses in conjunction with submitting a complete Faculty Activities Report each spring to their department head. The data entry system also manages the student teacher evaluation and portfolio evaluation data.

A survey is sent each fall from the Center to all completers, asking about the demographics of their teaching, such as rural, suburban, or urban, the special needs population they instruct, and the SES of their students (click here for Follow-up of Initial Preparation Graduates). Each fall a survey is sent to principals of the previous year’s completers, asking them to assess candidates on the expectations detailed in the Conceptual Framework (click here for results of principals’ survey). A similar survey is sent to those same completers for a self-assessment on the same expectations (click here for results of candidate self-assessment). We are finding that many candidates are not in the same school where they taught the first year; therefore, we have been unable to match candidate self-assessment with employer assessment on the same criteria. The data are aggregated across programs as well as broken out by program. This spring we will mail the survey to both the principal and candidate close to the end of the school year, i.e. the assessment will take place after one year of teaching, rather than waiting until the beginning of the following year. The Center also collects survey data each spring from cooperating teachers concerning their satisfaction with university supervision (click here for cooperating teacher survey data). These survey data will be analyzed yearly as well as across years.

The Professional Education Committee is comprised of program leaders, representative clinical faculty, school personnel directors, and representatives from Arts & Sciences. During each fall and spring meeting, a major topic of discussion is program effectiveness and the quality of our candidates after they enter teaching. School representatives give us good feedback. But collecting data from a broad range of our candidates after three years is problematic because during those first few years they move more frequently than later in their careers. Graduates who win teaching awards or achieve National Board Certification frequently contact their program advisors, thanking them for their preparation. However, those are not the majority of our graduates. Also collecting data on their impact on student learning is problematic unless candidates would volunteer their SOL pass rates, and only in some cases can those results be attributable to a given teacher. Now with yearly testing as part of No Child Left Behind and the Virginia database that is being developed for tracking teachers and their effectiveness, Virginia Tech and teacher preparation programs will have access to data that reflects our graduates’ effectiveness in helping their students learn. This database will yield much needed feedback on our graduates, feedback that can assist with program improvement.

The Center uses institutional data when possible, but for the most part, because degrees cut across programs and degrees are a major way of university tracking on specific categories, we have had to set up a collection system with departmental input to identify ethnicity, full-time or part-time enrollment and such for both initial preparation programs as well as advanced programs. For example, all candidates in graduate initial preparation programs receive a master’s degree in curriculum and instruction, yet that one degree has completers from multiple programs.

Advanced programs in reading and counseling will use the PPAD system that the Center will manage and analyze for program review documentation. Administration and Supervision and Instructional Technology have developed individual systems suited to their program needs. Those programs will collect data and analyze it for program review documentation. Those data will be accessible by the Center as well.

Element 3: Use of Data for Program Improvement

Employer survey results with written comments are shared with individual program advisors, especially where data indicate a need for some change. The data are also provided for discussion at program meetings. The cooperating teacher survey of university supervision, with all written feedback of strengths and weaknesses, is shared with program faculty and used in the training of field supervisors. The first follow-up of graduates is an excellent source of feedback because we specifically ask for written comments on “What one course, person, or experience was most valuable in your teacher preparation program?” and “What would you suggest as an improvement to the teacher education program at Virginia Tech?” We compile these comments and follow up with programs for discussion. (The data are available in the Center for review.)

EBI data are shared with programs and in the Center Advisory meetings as well as the Professional Education Committee. The 2002 results showed students thought we did not do enough with job searching. As a result, we coordinated with Career Services to share information about job fairs and notify all completers by e-mail of all fairs and openings. The data also showed that students also recognized the need for more diversity. All syllabi were reviewed at the program level to focus, reshape, and foreground various aspects of diversity in class interactions and assignments. Faculty also integrated university multicultural activities into their curricula. For example, TESH programs ask candidates to participate in at least one event offered by the Office of Multicultural Affairs and to reflect on the contribution of that event to their knowledge about the topic of the event, how it affected their attitude toward the topic, and whether how what they experienced could be used in their teaching. Toward the end of the semester candidates share in their experiences in afternoon content seminars and, in that way, learn about events they did not attend as well.

When 2003 EBI data come back broken down by program, we will be able to do a lot more with the results at the program level. When data are aggregated, it is difficult to see particular patterns within the larger unit. Also by adding up to 10 questions that target specific areas of interest for us, we will generate information on which to make programmatic decisions. For example, perception of education faculty’s integration of technology for instruction is extremely important because this has been a focus of faculty development for some time. Also, candidates’ perception of the usefulness of the electronic portfolio to document their meeting professional and content standards needs to be evaluated. We have continued to develop the portfolio process with the assumption that candidates view portfolios as a good way to demonstrate what they know and are able to do. Being able to look at candidate response programmatically will be beneficial to not only evaluating the whole process but also looking at the ways that particular programs carry out portfolios and the differences in candidates’ responses. Classroom management was another area where overall the programs rated lower than in other areas. We discussed this result in the Center Advisory Committee because generally faculty knew that classroom management was embedded throughout all programs. However, we decided to foreground classroom management, to make some activities more visible. Elementary education developed a list of web resources on classroom management that is accessible to all candidates. Secondary education arranged a one-half day workshop with a consultant prior to student teaching.

Many of our candidates sign a Portfolio Consent Form, which allows faculty to use examples from portfolios for instruction, serve as models for other candidates, and provides a source for research as well as reflection on program improvement. Until now the emphases have been on improving the portfolio process and evaluation. We will soon have a body of portfolios that we can analyze to see what experiences seem to contribute most to candidate growth as reflected in the portfolios, how to assist candidates in reflecting upon their practice, how candidates are documenting student learning, and the ways that inquiry can be threaded throughout the portfolio rather than presented as one required element.

Data from Praxis II this year will be used to make immediate decisions about new directions for placement of the test within programs. Those data will be brought to the Center Advisory Committee, which will make a recommendation to the Professional Education Committee for its consideration.

Advanced programs will use their program data and advisory committees for program improvement as well as the processes described in their assessment plans. For example, the administration and supervision preparation program has designed a systematic process for collecting data with monitoring and scanning activities, analyzing and synthesizing the data, translating the findings into program components, integrating components into the program, and assessing and evaluating outcomes. The Reading Specialist and Counselor Education programs will use input from school partners as well as program data to guide improvements. Instructional Technology program will monitor standards and program requirements as such programs must always keep on the cutting edge to be effective.