Title page for ETD etd-072399-154928

Type of Document Master's Thesis
Author Sienknecht, Jr., Ronald Theodore
Author's Email Address tsien@vt.edu
URN etd-072399-154928
Title An Empirical Analysis of Rating Effectiveness for a State Quality Award
Degree Master of Science
Department Industrial and Systems Engineering
Advisory Committee
Advisor Name Title
Van Aken, Eileen M. Committee Chair
Coleman, Garry D. Committee Member
Koelling, Charles Patrick Committee Member
  • Halo Error
  • Organizational Assessment
  • Quality Award
  • Rating Effectiveness
  • Interrater Reliability
  • Leniency and Severity
  • Range Restriction
Date of Defense 1999-06-28
Availability unrestricted
This research clarified existing inconsistencies in self-assessment literature, and added to the body of knowledge for rating effectiveness of organizational assessments by defining relationships among rating effectiveness criteria (ratee, rater, rating scale, rating process) and measures (interrater reliability, halo error, leniency and severity, range restriction) based on extensive literature review. A research framework was developed from this review, and was employed in computing rating effectiveness measures at the individual (i.e., examiner or eight rating scale dimensions) and sector (e.g., Private Manufacturing Sector, Private Service Sector, Public Local Sector, Public State & Federal Sector) levels for a State Quality Award (SQA) using data from the 1998 applications.

Interrater reliability (measured by intraclass correlations for each rating scale dimension) was low to moderate, and differed by dimension. Halo error (measured by the determinant of the dimension intercorrelation matrices for each examiner) was present for all examiners. Leniency and severity (measured by presence of statistically significant Rater main effect for each dimension) was present for 11 of 32 cases, and differed by dimension. Range restriction (measured by variance analysis for each dimension) was present for 22 of 32 cases, and differed by dimension. A post-hoc principle component analysis indicated poor internal reliability for the rating scale. To improve, the SQA should replace the existing rating scale and provide in-depth training on all elements of the rating process. The importance of the SQA using boxplots, histograms, and rating effectiveness measures to make fully informed decisions was discussed.

  Filename       Size       Approximate Download Time (Hours:Minutes:Seconds) 
 28.8 Modem   56K Modem   ISDN (64 Kb)   ISDN (128 Kb)   Higher-speed Access 
  thesis.pdf 1.83 Mb 00:08:27 00:04:20 00:03:48 00:01:54 00:00:09

Browse All Available ETDs by ( Author | Department )

dla home
etds imagebase journals news ereserve special collections
virgnia tech home contact dla university libraries

If you have questions or technical problems, please Contact DLA.