Reliability-Validity Study for Secondary Education

 

October 21, 2004       

 

Present in Cra211 for 9: 30 meeting:  Pat Roach, Tim Carter, Sid Womack

 

Data from the 2003-2004 year of  Secondary Program completers were used to examine the reliability of instrument used for assessing candidates.  Split-half analyses were used for students for whom we had complete data.  Our findings are:

 

1.         For the SEED2002 Field Experience Evaluation Form for Secondary program completers in 2003-04, the split-half reliability with 45 pairs of data was 0.917,  p<.001.  Items on that instrument were matched with Arkansas State Department of Education and INTASC standards.  Some of the items were dispositional and one matched at least two ISTE technology standards.

 

2.         For SEED3554, the Research Awareness Report rubric had a split-half reliability of 0.735 (p< .001) with 36 pairs of data.  Data on some program completers were not available because they evidently took this course elsewhere.  This rubric was matched with Arkansas Department of Education Standards for Beginning Teachers and Local Diversity Standards.  Some ADE standards mapped to were dispositional.

 

3.         For SEED3554, the Field Experience Evaluation Form had a split-half reliability of 0.9974 (p<.0001), n=36.  These items too had been matched to Arkansas ADE Standards, which had been mapped to INTASC standards. 

 

4.         For the SEED4556 Field Experience Evaluation Form, the split half reliability was 0.9975, p<.0001, wiht 40 pairs of usable data from 2003-2004 program completers.  Items on that instrument were matched with Arkansas State Department of Education and INTASC standards.  Some of the items were dispositional.

 

5.         For the SEED4556 Unit rubric, the split half reliability was .9846 with 40 pairs of data from 2004-2004 program completers, p<.001.  Rubric items were matched to state and therefore ISLIC standards as well as a number of those items being matched to local diversity standards.  Some items were dispositional in nature.

 

6.         The SEED4556 Second Teach rubric had a problem that likely lies in the coding of data.  The decision was made to have Sid deal with the coding problem and run it again. 

 

7.         The SEED4503 rubrics for the School Law report and for the History/Philosophy report did not show an acceptable reliability due to the way data are now being coded.  Decision was made that since the SEED4503 data were not needed to address the standards that are already being assessed at other points and in other classes, to remove the SEED4503 data from the master matrix.