Title page for ETD etd-10252003-132129

Type of Document Dissertation
Author Burnkrant, Steven Richard
Author's Email Address srburnkr@opm.gov
URN etd-10252003-132129
Title Interrater Agreement of Incumbent Job Specification Importance Ratings: Rater, Occupation, and Item Effects
Degree PhD
Department Psychology
Advisory Committee
Advisor Name Title
Hauenstein, Neil M. A. Committee Chair
Carlson, Kevin D. Committee Member
Foti, Roseanne J. Committee Member
Harvey, Robert J. Committee Member
Stephens, Robert S. Committee Member
  • Job specification
  • Competency
  • Validity
  • KSAO
  • Job analysis
  • Interrater agreement
Date of Defense 2003-10-23
Availability unrestricted
Despite the importance of job specifications to much of industrial and organizational psychology, little is known of their reliability or validity. Because job specifications are developed based on input from subject matter experts, interrater agreement is a necessary condition for their validity. The purpose of the present research is to examine the validity of job specifications by assessing the level of agreement in ratings and the effects of occupational tenure, occupational complexity, and the abstractness of rated worker requirements. Based on the existing literature, it was hypothesized that (1) agreement will be worse than acceptable levels, (2) agreement will be higher among those with longer tenure, (3) agreement will be lower in more complex occupations, (4) the effect of occupational tenure will be more pronounced in complex than simple occupations, (5) agreement will be higher on more abstract items, and (6) agreement will be lowest for concrete KSAOs in complex occupations. These hypotheses were tested using ratings from 38,041 incumbents in 61 diverse occupations in the Federal government. Consistent with Hypothesis 1, agreement failed to reach acceptable levels in nearly every case, whether measured with the awg or various forms of the rwg agreement indices. However, tenure, occupational complexity, and item abstractness had little effect on ratings, whether agreement was measured with rwg or awg. The most likely explanation for these null findings is that the disagreement reflected a coarse classification system that overshadowed the effects of tenure, complexity, and abstractness. The existence of meaningful subgroups within a single title threatens the content validity of job specifications: the extent to which they include all relevant and predictive KSAOs. Future research must focus on the existence of such subgroups, their consequences, and ways of identifying them.
  Filename       Size       Approximate Download Time (Hours:Minutes:Seconds) 
 28.8 Modem   56K Modem   ISDN (64 Kb)   ISDN (128 Kb)   Higher-speed Access 
  Burnkrant.pdf 670.93 Kb 00:03:06 00:01:35 00:01:23 00:00:41 00:00:03

Browse All Available ETDs by ( Author | Department )

dla home
etds imagebase journals news ereserve special collections
virgnia tech home contact dla university libraries

If you have questions or technical problems, please Contact DLA.