General Overview

of the

Unit Assessment System (UAS)

 

The School of Education at Arkansas Tech University, in response to Arkansas Department of Education and University mandates, and with the involvement of our professional community, has developed and is implementing the Unit Assessment System (UAS) to develop, gather, and employ evaluative information about candidates’ performance in all undergraduate and graduate programs.  The UAS provides a manageable structure for how the School of Education secures information on internal and external standards that describe candidates’ ability to have positive effects on student learning.  This plan describes procedures for performance assessments and evaluations.  This plan also describes the decision making process used by the unit to judge candidates’ acceptable attainment of the standards using multiple measures, at multiple and appropriate points, to program completion at the undergraduate and graduate levels.  Finally, the plan includes the summarizing and sampling of performance evidence, ensuring that it is representative of the performance of standards peculiar to each program.  Assessments used to determine admission, continuation in, and completion of programs are reviewed for predictability of candidate success.  The UAS enables the unit to take effective steps in eliminating sources of bias in performance assessments and in establishing fairness, accuracy and consistency in its assessment procedures.  The UAS is designed to be a comprehensive and integrated set of evaluation measures that are used to monitor candidate performance and manage and improve operations and programs.

 

The UAS encompasses assessment of candidate’s performance at the program level.  This is highly appropriate for the unit, as it consists of departments that offer a variety of preparations.  The UAS is an overall management system that provides support to, and coordination of efforts of, each program in order to insure that all programs have designed and implemented comprehensive assessment practices that are of high quality.  Programs in the unit are not all at the same stage of development, and they have not achieved the same quality in their assessment efforts, however, each program has developed assessment plans based on the incorporated Program Assessment Form of ATU.   These program assessment plans form the core of the UAS.   As these plans are implemented, the UAS intends that the following features will be evident:

  • An emphasis on candidates’ learning at the program level.
  • A grounding in curriculum maps aligned with the Conceptual Framework and state and national standards.
  • An emphasis on on-going and continuous assessment of candidates’ learning related to the Conceptual Framework and state and national standards; regularly gathered and summarized program evaluation data from multiple points and from the multitude of stakeholders.
  • Incorporation of a means for continuous improvement of the quality of the program.
  • Incorporation of a variety of fair, valid and trustworthy assessment strategies that insure the credibility of data on candidates’ learning from admission through induction.
  • Information on assessment requirements and the results of such assessment is accessible to candidates throughout the program.
  • Current information on results of assessments is available to stakeholders.
  • Stakeholders are involved in assessments and in the decision-making processes used to improve programs based on assessment results.
  • A developing ability to predict success of candidates in programs and upon graduation.
  • An ability to monitor quality of instruction provided by faculty.
  • An ability to enhance the overall operation of the unit through decision-making based on data on candidates’ performance, faculty performance, and resources.
  • Use of technology to support data gathering and aggregation.
  • Distribution of responsibility for implementation and coordinated management of the UAS.
  • Regular internal and external assessments of the assessment system.

 

Management Structure for the School of Education UAS

 

      Based on existing academic regulations, responsibility for oversight and management of professional preparation programs at ATU flows from the Dean of the School of Education in coordination with all deans from all units at all levels, the Office of Academic Affairs, and the Vice President for Institutional Research; through department heads within and across units; through the faculty in and across units, as represented by the various committees established in the unit and standing in the university.  Responsibility for program assessment in the unit is shared among department faculty in departmental committees, the department head, the Office of Director Teacher Education Student Services, and the Dean of the unit.  Responsibility for unit assessment and its integration with university assessment is also, shared among departmental committees, the department head, the Office of Director Teacher Education Student Services, and the Dean of the unit.

 

      Assessment activities are balanced between those conducted at the university level, the unit level, and those conducted by the faculty in specific professional preparation programs (Exhibits).  Some assessment activities utilize unit-wide data that are maintained in the office of the Dean of the unit or the Director Teacher Education Student Services.  These data, such as national assessment data, reflect common elements across the various preparation programs.  Other assessment activities, such as the results of performance in the field, are program specific and obtained by individual program faculty under the guidance and supervision of department heads and Director Teacher Education Student Services.  Stakeholders provide guidance to the assessment process at the university, unit and program levels.  The UAS reflects a distribution of responsibility for source, management, and operation.

 

            The unit will ensure that its assessment system is continuously managed.  Coordination of the offices of the Dean and the Director of Teacher Education Student Services and their respective staff in the fall of 2001 has designated a full-time Document Examiner, one 75% time secretary, and one graduate assistant to conduct and maintain the UAS.  A networked, PC-based File Pro database system has been developed for collection, recording, storage, and retrieval of data.  Queries are generated in EXCEL spreadsheets.  The system became operational in the fall of 2000. The secretary and the Document Examiner monitor the database and ensure that appropriate data are entered or received from designated sources for recording.  Data are secured and confidentially maintained. The Dean, the Director of Teacher Education Student Services, the secretary and the Document Examiner have full access by password.  Faculty have access in read-only status by password.  The secretary and the Document Examiner provide data to the Dean, the Director of Teacher Education Student Services, and the unit’s department heads all of whom, in turn, distribute information to appropriate stakeholders throughout the year and annually in summary.  These data summaries, then, provide the basis for reviewing and revising the conceptual framework, program curricula, and the assessment system.  The unit’s technology staff also assist with specific tasks, as needed.

The Dean is a member of the University Assessment Committee as well as coordinator for the unit.  Responsibilities of the Dean as coordinator of the UAS include:

  • Oversight of the implementation of the UAS
  • Leadership of the Unit Assessment Committee and revisions to the UAS as a plan for assessment
  • Organization of assessment-focused professional development activities for faculty and staff in the unit
  • Coordination with university-wide assessment activities
  • Surveying of graduates and their employers
  • Facilitation of stakeholders’ involvement in implementing and refining the UAS based on results of assessment activities
  • Provision of technical assistance to faculty in support of refining and implementing programmatic assessment plans
  • Assuring routine entry of performance data on candidates
  • Collaborating with faculty and staff to aggregate and disaggregate data on candidates’ performance on a regularly scheduled basis
  • Assisting faculty in analyzing and interpreting data
  • Refinement of the database
  • Maintaining mechanisms for effective communication with programs and departments within the unit and across the university on assessment issues
  • Coordination of periodic evaluations of the UAS

 

The UAS will become an element of the institutional assessment plan and be linked to all other Units.  For example, relevant data received from the institutional assessment include admissions’ demographics.  Relevant data provided by the Unit to general education include Praxis I and II assessments.  Data is made available to departments of Arts & Sciences and to the Associate Vice President of Academic Affairs.  Appropriate interfaces with these offices are near completion.

 

The UAS, in part or as a whole, has been on the agenda of each meeting within the unit and will continue to be germane to all unit business.  Each meeting has provided some small step forward in the desired dialogue, exchange, input, revision, and refinement of the UAS.  Stakeholder representation varied within these meetings, but across all meetings all intended stakeholders were reached.  There is now a network for collaborative development and a large pool of committed participants in the system.

 

Evaluation Plan for the UAS

 

Evaluation and assessment within the unit will be a process for determining the extent to which the goals, and outcomes as conceptually developed and organized, are actually producing the desired results.  The UAS will enable the unit to identify the strengths and weaknesses of its conceptual plans, and to determine the areas in which the curriculum is effective and in need of improvement.  The unit’s system will enable the unit to determine whether the curriculum as designed, developed, and implemented is producing the desired results.

           

The validity of the assessment is maximized by being sure about the purpose, defining the teacher candidate characteristics to be evaluated, specifying levels of performance along appropriate continuums and through articulate rubrics, using exercises that sample the range of performance contexts, and comparing ratings with other achievement data when possible.  The reliability of the assessment is maximized by using clear criteria, continuously training raters, carefully attending to planning, and implementing appropriate scoring procedures.  These activities will demand increased stakeholder collaboration, especially faculty collaboration across content area departments.

 

Program assessment will be a formative process and an ongoing part of unit activity.  The assessment process is the systematic determination of what knowledge, performances, and dispositions are best nurtured by which experiences provided, or what experiences may be added to ensure the accomplishment of programmatic benchmarks.  Data will be collected and analyzed at summative decision points and between in all programs.  Licensure tests, technology assessments, portfolio evaluations, field evaluations, and the state report card will, particularly, document the preparation of candidates such that individual diagnosis and group needs assessment are available to improve the opportunities for candidates’ development.  Identifying the strengths and weaknesses in the performance of individuals, and diagnosing the strengths and weaknesses of groups of candidates such as by tasks completed, by courses completed, or by program completed, can be used to provide for educational needs of the group, to improve the opportunity for learning by task or course, or to refine the program.  These data will assist programs in identifying those candidates ahead of, or behind, the expected level of performance for decisions about continuing or providing remedial opportunities.

 

The unit and stakeholders will review program assessment results annually in preschool in-service.  Aggregated assessments from individual candidates and other sources will be used to refine and revise the conceptual framework and program opportunities. The unit will use the collective presentation of candidate assessments and related data to document and define the quality of programs preparing candidates.  Quality will be determined on the basis of whether candidates are acquiring learning outcomes and whether the programs and unit are achieving their respective goals.  That is, whether the knowledge, performances and dispositions are best nurtured by the experiences that lie ahead or by seeking out additional experiences to ensure the accomplishment of benchmarks and achievement of readiness to teach.   Questions leading examination of the data will include:

·        Do we have sufficient and revealing evidence of knowledge, performances, and dispositions?

·        How can we build more effective tasks around and from the need for evidence of critical knowledge, performances, and dispositions?

·        Are performance tasks effectively anchoring the courses and focusing the instructional work?

·        Are we able to distinguish between those who really have a grasp of the standards and those who don’t?

·        What misunderstandings and misconceptions are occurring?  How do we check for those?  How do we assist the candidate in resolving them?

Answers to these questions will be considered through the Peer Review Process and in preschool in-service to analyze the sources of performance evidence and provide for on-going development.

 

The unit conducts an annual Peer Review Process.  Each fall semester candidates are asked to rate faculty on their effectiveness through the use of a university-designed survey.  Candidates also rate how well they have mastered each of the course objectives.  The faculty reviews these ratings and prepare teaching portfolios which include overviews of courses taught, examples of performance tasks and instructional strategies, sample(s) of teacher candidate work, and evaluation rubrics.  Through the Peer Review Process faculty discuss the results of the evaluations and any changes they wish to make to better support teacher candidate’s achievement of the standards.  Recommendations for change are then submitted to the department head and Dean.  Proposals for change are presented to the Teacher Education Council, which includes a variety of stakeholders in its membership.

 

Yearly review of all data collected will follow the Peer Review Process, submission of Exit Portfolios, and Exit Evaluation in the spring semester.  Results will be provided to appropriate stakeholders for review and input which is expected to lead to continuous programmatic and unit improvement.  Reports on all assessments of the system will be shared with stakeholders through posting on the unit’s web page and at the annual meeting of the Assessment Task Group held in the spring of each year.  Assessment and evaluation information will be compared to available external sources, i.e., scores and sub-scale results on state licensure tests.  The collective data on candidates’ and graduates’ performance will be summarized in the Dean’s Report to the unit and stakeholders, the State Report Card, the NCATE review for accreditation, and learned society program reviews.  This review will prompt proposals for change in an appropriate sequence for stakeholder input prior to the Unit and University governance structure for revision of programs and/or assessment system.  The Peer Review Process, the Exit Portfolio, Exit Evaluations, and induction follow-up studies will be key components in documenting the shift from program processes, what candidates do in their preparation programs, to the competencies of candidates, what candidates can do with and for P-12 learners.  The Peer Review Process and resulting preschool in-service will provide clear evidence that data collected on candidate performances and evaluative assessments gathered from candidates are used to make decisions on curriculum and program practices.  Examples of changes in the program, including general education, professional education, and field work, which result directly from analyses of candidate performances and evaluative assessments, became available in the spring of 2003 following the full implementation of the UAS.

 

The UAS provides for annual review and revision of the assessment system itself.  Implementation of program assessment plans and the overall operation of the UAS will be evaluated annually by the Unit Assessment Committee.  Data collected, and the implementation of the UAS, will be reviewed annually, at the close of the spring semester.  The results of these evaluations will be presented in annual reports to the Vice President of Institutional Research, chair of the University Assessment Committee.  Obtaining valid and reliable results will be vital to the success of the assessment plan.  It is the unit’s goal to make the judgment-based evaluation process determining professionals for the future, systematic and objective as it can be while focusing on the most important attributes of performance relevant to the program standards.  Summaries will be prepared and distributed to appropriate stakeholders.  Stakeholders will be gathered in the pre-school in-service meeting in August.  The assessment system will become a standing agenda item.  Central questions to be asked are:

·        How adequately is performance described at benchmarks and between?

·        How adequately is data recorded/reported?

·        How adequately, and to whom, are results communicated, i.e., teacher candidates, Director Teacher Education Student Services, Faculty, Committees, Dean, institution and departments, and other appropriate stakeholders?

·        What are the most critical pieces of performance data for candidates in each program?

·        What is the relationship of the data to the Unit’s conceptual framework?

·        Does the assessment system assess the alignment between standards-based performances and assessment?

·        Is the assessment system clear and understandable?

·        Are assessment results consistent across sources and across time?

·        Is the assessment system feasible from the perspective of all stakeholders?

·        Does the assessment system use multiple and diverse assessment strategies to fairly assess teacher candidates’ performance?

·        Does the assessment system tell us what we want and need to know when we want and need to know it?

·        To what extent does the assessment evidence provide: 1) a valid and reliable measure of the standards? 2) sufficient information to support inferences about each teacher candidate’s readiness to teach? 3) meaningful opportunities for teacher candidates to exhibit the knowledge, performances, and dispositions through authentic performance tasks?

In addition, beginning in the Spring of 2005 and every five years thereafter, the UAS will be evaluated by the Board of Examiners (BOE) team from NCATE.  Results will be submitted in BOE reports to NCATE.  This review will, also, be instrumental in the refinement of the UAS.

 

The unit, in response to the annual review, resets goals and focuses faculty development in support of their accomplishment.   Support includes, but is not limited to, unit in-service and work days and professional development opportunities in the process of aligning standards, curriculum, instruction, and assessment.  The goal is that, as a unit, there is clear purpose of assessment, and that the assessment is effectively communicated to candidates with performance criteria conveyed in understandable ways, such as standards-based rubrics, prior to the assessment and throughout challenging performance exercises.