NOTICE Equalisation of scores in SSC Examinations

NOTICE Equalisation of scores in SSC Examinations Staff Selection Commission has been conducting various examinations in multiple batches because of large number of candidates and difficulties in getting adequate educational institutions for holding the examinations in a single batch. For perhaps the first time in its history, the number of applicants in a single examination exceeded one million when the Combined Higher Secondary Level Examination, 2010 for the recruitment of Lower Division Clerks and Data Entry Operators, elicited response from over 16 lakh candidates, with app. 21% of them applying online. This would require the Examination, rescheduled on 27 & 28.11.2010 (in view of Common Wealth Games), to be held in at least three batches. The Commission, with the help of experts, has striven to construct question papers of comparable difficulty level. While such an exercise is theoretically possible, in practice it is impossible to have two or more question papers of identical difficulty levels. Even if the difficulty levels of question papers vary slightly, candidates taking more difficult papers may be at a disadvantage viz-a-vis others. Therefore, there is a need for equating of the marks in examinations involving multiple batches and question papers. 2. The Commission had examined the views of an Expert Group, constituted by it with the approval of Government of India in 2009, on this issue. The Commission had placed before the Expert Group that the technique to be followed for equating should be transparent, easily comprehensible to the candidates, acceptable to experts and prove itself in Courts of Law if and
when challenged. This was accepted by the Expert Group which further advised the Commission to place a paper on the technique on its website for adequate time, give publicity to such placement through the media, invite comments, observations and suggestions and decide on adopting the technique thereafter. 3. Accordingly, the Commission places this paper inviting views, comments and suggestions from academicians, parents, candidates and also stakeholders on or before 31.10.2010. Such views, comments and suggestions, if any, may be addressed to Smt. Vandana Sethi, Deputy Secretary, Staff Selection Commission, Block-12, CGO Complex, Lodhi Road, New Delhi-110 003 (e-mail: vsethi.ssc@gmail.com). The Commission proposes to adopt the Equipercentile Equating Technique for examinations involving multiple batches from Combined Higher Secondary Level Examination, 2010 onwards.
PAPER ON EQUATING TECHNIQUES 1. Introduction: Building Question Papers based on the same test specification either on blueprint or content template will not ensure that these tests are equivalent, specifically based on the difficulty level, content and ability tested. To ensure fairness in the scoring system when candidates are administered different Question Papers of the test, an appropriate statistical procedure is needed to equate the test scores based on the different Question Papers. In other words, Equating is needed. 2. Large-scale testing programs often require multiple Question Papers to maintain test security over time and to enable equivalent tests to be administered without repeating identical items. The outcomes after equating are we’ll get the same scores in tests in which different question papers were administered. 3. Equating is a statistical process that is used to adjust scores on multiple question papers so that scores on the forms can be used interchangeably. It adjusts for differences in difficulty among Question Papers that are built to be similar in difficulty and content. 4. Example: Here is an example of Test of English language in 2 Question Papers (VA1 and VA2) which consists of 50 items each that would assess the usage of articles amongst other things. VA1 1. ___ book that I bought yesterday is interesting. 2. I saw ___ man wearing a red suit.
VA2 1. ___ goat was crossing ___ river. 2. I have been waiting for ___ auto. After evaluating the Answer Sheets the overall performance of the students in two different Question Papers from similar background students is found to be varying with the following mean scores. Mean scores are: VA1 14.5 VA2 12.5 The number of right answers in VA1 is found to be 2 units more than those of VA2. Similar variation in mean scores may be found in the case of all the questions included in the test. Therefore, there could be a need for adjustment of scores of each test so that scores from different question papers can be used interchangeably 5. Process followed in Equating • The purpose of equating should be clear. • The Examination should have multiple question papers on the same subject. • Alternate question papers are set in accordance with items of the same content, difficulty level and ability tested. • Data of responses to items/questions should be collected. • The responses should be in the form of (A, B, ……….., X) or (Yes/No) etc. for objective type tests. • Equating requires that a choice be made about what type of relationships between different question papers is to be estimated. For example, this
may involve decision on whether to use linear/non linear methods of equating. • Descriptive statistics to obtain mean, standard deviation and/or the percentile rank of scores may need to be generated after evaluation of the papers. • A statistical equating method should be carefully selected for adoption. • The results of equating should be evaluated carefully. . 6. Different Methods of Equating a) Median/Mean Equating b) Linear Equating (Based on mean and S.D.) c) Equipercentile Equating d) Equating using Item Response Theory a) Median/Mean Equating Procedure: • In the given example, two question papers of Verbal Ability Test are administered on two similar groups of student(VA1 and VA2) • Mean score of VA1 = 14.5 & Mean score of VA2 = 12.5 • Difference between mean scores = 14.5 – 12.5 = 2 • Mean equating involves the addition of a constant (2 = difference in the mean scores) to all the raw scores on VA2 to find equated scores on VA1 b) Linear equating (Based on Mean and S.D.) This involves equating the two tests using the mean and standard deviation of each question paper. If x represents a score on the VA2 (in the given example) and y represents a score on the reference form (VA1), then x and y are equivalent in a group of examinees if:
Procedure: For adjusting the new scores according to the reference form the formula is: X(VA2) Y(VA1) Mean 12.5 14.5 S.D. 4.5 4 Y adjusted to X: c) Equipercentile Equating The method of equi-percentile equating involves fixing up a common percentile for all forms (it is observed that same test score in each of the question papers gives rise to different percentiles) and then finding the different scores in these forms in terms of number right scores. Thus a score of 17.2 in series -1 score is equivalent to 14.5 in series-2.This in essence is equipercentile equating. Procedure: Given below are two sets of scores, series 1 and series 2, which are multiple question papers of the same test. As far as possible, the no. of examiners in either series should be equal. Their percentile ranks are calculated from the grouped data using SPSS or Excel. ymean(y)xmean(x)SD(y)SD(x)(xmean(x))ySD(y)mean(y)SD(x)1012.5414.512.284.5
series 2 score percentile rank 1 0.00% 2 5.50% 3 11.10% 4 16.60% 5 22.20% 6 27.70% 7 33.30% 8 38.80% 9 44.40% 10 50.00% 11 55.50% 12 61.10% 13 66.60% 14 72.20% 15 77.70% 16 83.30% 17 88.80% 18 94.40% 19 100.00% series 1 score percentile rank 3 0.00% 4 5.20% 5 10.50% 6 15.70% 7 21.00% 8 26.30% 9 31.50% 10 36.80% 11 42.10% 12 47.30% 13 52.60% 14 57.80% 15 63.10% 16 68.40% 17 73.60% 18 78.90% 19 84.20% 20 89.40% 21 94.70% 22 100.00%
The above graph shows both percentile ranks of series 1 and 2 are put together in order to enable the percentile score of one raw mark of series-1 to be slightly different from the raw mark score of series-2. The percentile rank of 17 in series -1 is 88.8% and in series -2 the same score of 17 has a percentile of 93.6%. In the method of equi-percentile equating, a cut-off of a percentile rank to qualify for selection is to be fixed, say 75 percentile and this common 75 percentile has a raw score of 14.5 in series-1 and 15.26 in series-2. Therefore, anyone who gets more than 14.5 in series -1 and 15.26 in series 2 will qualify and the equi-percentile is 75. SSC PROPOSES TO USE THE EQUIPERCENTILE METHOD IN VIEW OF ITS SIMPLICITY
d) Equating using Item Response Theory (IRT) Basic Concepts • Ability (θ): Ability is measured on the scale (-3 to +3) • Item difficulty (b): The item difficulty of an item and the ability of the test taker are on the same scale. it is invariably taken as the middle point of the item characteristic curve where the curve shows a tendency of contra flexure that is bending in the opposite directions (-3 to +3) • Both of these are on the same scale and along the x-axis • Probability of getting the correct answer (0 to 1) on the y-axis • This is a fraction like 0.75which means if a person of particular ability say θ ( any ability between -3 and +3) attempts 100 times the same item 75 times he will get it right and 25 times he will get it wrong. Fred Lord’s Model - ∞ + ∞ -3 + 3 - 4 + 4 α tan α = a Item discrimination Item Difficulty ‘b’ Guessing c Pi (θ) (probability of getting answer right on any item I with ability θ) 0.5 +1.0 1-c
Types of equating in IRT i) Horizontal equating ii) Vertical equating Horizontal equating is equating scales of about the same ability Vertical equating is equating scales across completely different levels of ability THE COMMISSION DOES NOT USE IRT IN ITS EXAMINATIONS.