Category: Problem Solving

Question Difficulty in GMAT Official Guides

GMAT Official Guide Question DifficultyStarting with the 11th Edition of the Official Guides, the Graduate Management Admission Council placed questions in order of difficulty. This is a welcome change from prior editions, because students can now choose practice questions at the appropriate difficulty level. Beginning students need not tackle difficult questions that may cause them to feel overwhelmed and frustrated, nor do advanced students need to waste time on questions that are well below their capabilities.

Mysterious Methodology

The methodology that the GMAC uses to assign difficulty ratings to questions remains a mystery, however. Although there is a clear overall correlation between question placement and GMAT Genius’ assessment of difficulty, the correlation is far from perfect and there are many outliers, particularly with math. Let’s consider the Problem Solving section of the GMAT Official Guide (13th Ed.).

Problem Solving Analysis

There are 230 questions in this section, excluding questions in the Diagnostic Test chapter of the book. GMAT Genius assigns difficulty at five different levels: Super Easy, Easy, Moderate, Hard, and Very Hard. To translate this into numbers, we can assign points of 1, 2, 3, 4, and 5 to each of these levels, respectively. With perfect correlation and distribution, we should see the first 46 questions (230 divided by 5) all at a Super Easy level (average difficulty 1.0), the next 46 questions all at an Easy level (average difficulty 2.0), and so on. Instead, here is how we would rank question difficulty:

OG 13E PS Super
Easy
Easy Moderate Hard Very
Hard
Average
Difficulty
1 – 46 11 29 6 0 0 1.9
47 – 92 3 18 23 2 0 2.5
93 – 138 1 10 21 13 1 3.1
139 – 184 0 12 26 4 4 3.0
185 – 230 0 5 25 11 5 3.5
Total 15 74 101 30 10 2.8

As shown in the table, average question difficulty (based on our assessment) does not increase much in the last three quintiles. In fact, the middle quintile is actually slightly harder than the fourth quintile. Surprisingly, we consider five of the top quintile questions “Easy,” including #220, which is supposed to be one of the hardest questions in the entire set. We also consider question #117 in the third quintile “Very Hard,” even though it barely passes the halfway mark. Yet the GMAC considers this question easier than #130, which we rate as “Super Easy.”

Disparity Among Similar Questions

We see this disparity even among similar questions. Questions #178 and #186 both entail overlapping sets, and based on the ordering, the GMAC considers #186 slightly harder than #178. Yet we rate #178 as “Very Hard” and #186 as “Moderate” because the former is a complex application with three groups whereas the latter is a straightforward application with two groups. As another example, consider #137 and #203 that we described in our prior post about repetitive math questions. These two questions are very similar, yet the GMAC considers the latter much harder than the former. By contrast, we rate both these questions as “Hard” and actually consider the latter slightly easier because the numbers are much easier.

Our assessment of difficulty is admittedly somewhat subjective, but probably more realistic than how the GMAC assigns difficulty. Our difficulty ratings take into account our observations of how students tend to find the questions in term of difficulty, the ease of the calculations involved, and the length of the Official Guide explanation, except when the explanation is inefficient or misses a shortcut. For verbal questions, we also take into account the difficulty of the incorrect answers, since process of elimination can be particularly helpful on verbal. Note that our assessment of difficulty skews towards the center; a question would need to be incredibly easy or incredibly difficult to qualify as “Super Easy” or “Very Hard”, respectively. The GMAC most likely has a much greater distribution in its difficulty assignments.

Conclusions

A key takeaway from this analysis is that you should not assume that the Official Guides are an objective measure of question difficulty. Although we don’t know how the GMAC assigns difficulty and would expect a few differences of opinion, we can confidently claim that quite a few questions are ordered incorrectly. So if you encounter a few questions in the Official Guides that are hard for you, it does not necessarily mean that all subsequent questions are beyond your reach. Another implication of this analysis is that the Official Guides do not contain enough difficult practice questions, a complaint we often hear from our advanced students. The GMAC has released two relatively new study products (Exam Pack 1 and Question Pack 1), so let’s hope that Difficult Pack 1 is in the works.

Repetitive Math Questions in GMAT Official Guides

GMAT Official GuidesIn your preparations for the GMAT math section, it is essential that you practice with retired real GMAT questions using the GMAT Official Guide (13th Edition) and the GMAT Quantitative Official Guide (2nd Edition). These two books provide you with a combined total of 430 Problem Solving and 322 Data Sufficiency practice questions.

Repetitive Problem Solving Questions

One interesting observation from our analysis of these questions is the frequency with which some questions closely mirror another question. Due to copyright issues, we cannot reprint these repetitive math questions from the Official Guides. But if you have your copies handy, compare the following sets of Problem Solving questions:

  • 13E #128 (page 170) vs. 2E #132 (page 79): These two questions involving combinatorics are nearly identical, even using the same numbers in the problem and in the first three answer choices (one of which is the correct answer). Only the context changes: one question involves finding the minimum number of colors for a distribution center’s coding, and the other involves finding the minimum number of letters for an experiment’s coding.
  • 13E #194 (page 179) vs. 2E #165 (page 84): Both questions provide three fractions of a total amount and the remaining balance that is unaccounted for, and ask us to calculate the total amount. One question relates to the amount of a trust fund whereas the other involves the number of students in a class, but the numbers within the problems are identical (except that one total is in thousands and the other is not).
  • 13E #124 (page 169) vs. 13E #133 (page 171) vs. 2E #56 (page 69): These three questions all involve calculating the number of pairs that are possible out of a larger group. Two of these questions ask about the number of table entries needed to show the mileage between any two cities, and the third asks about number of games needed so that every team in a league plays each other once. For these types of questions, we teach our students a tabular approach, a combinatorics approach, and a summation approach.
  • 13E #137 (page 171) vs. 13E #203 (page 181): At first glance, these two word problems seem unrelated. But the setup for both problems is identical: we have an unknown price and quantity, and when one goes up by a specified amount the other goes down by a specified amount in order to generate an equivalent revenue amount.
  • 13E #111 (page 167) vs. 13E #170 (page 176): In both cases, we have a terminating decimal in the form of 1 / (2^x * 5^y) where x and y are specified exponents. Both questions entail counting digits.
  • 13E #95 (page 165) vs. 13E diagnostic #13 (page 22): Both questions entail one positive integer divided by another, resulting in a quotient and a remainder equivalent to .12. Our objective is to use the .12 to deduce the value of the divisor in one question and to identify a possible remainder in the other question.
  • 13E #45 (page 158) vs. 2E #57 (page 69): Both problems give us a quadratic equation using the variable x and a constant k. Given one root of the quadratic, we must find the other solution in one problem and the value of k in the other problem.

Repetitive Data Sufficiency Questions

Although the above are the most obvious examples of repetition in Problem Solving, there is much more repetition of concepts across questions. We also can find examples of repetition within Data Sufficiency, albeit to a lesser extent. As two obvious examples, compare the following sets of Data Sufficiency questions:

  • 13E #2 (page 275) vs. 13E #21 (page 276): These two questions have identical setups, although with different numbers in different contexts. In both questions, statement 1 gives us what percent of women have a certain characteristic and statement 2 gives us what percent of men have the same characteristic. The questions both ask what percent of the total are women with that characteristic.
  • 13E #57 (page 280) vs. 13E #59 (page 280): In both cases, we have two different denominations (of bills in one question and of gift certificates in the other). Statement 1 has an identical setup in both questions – the maximum number of the smaller denomination. In #57, the initial information tells us the minimum number of the larger denomination and statement 2 tells us the total value of both denominations combined. In #59, the initial information and that given in statement 2 are swapped.

Conclusions

Why does this repetition happen? Since the GMAT tests a finite number of concepts, these concepts will inevitably appear repeatedly in various forms. But since the GMAC must produce a vast number of questions each year to ensure a fair testing environment, one way to make the question development process more efficient is to borrow heavily from other questions. We also see concepts applied repetitively in Verbal questions, but the questions themselves are not as obviously duplicative as the Math questions that we’ve discussed.

What is the implication of this? One of our “6 Habits of Highly Effective GMAT Students” is to watch out for patterns. As you work through many practice GMAT problems in the Official Guides and elsewhere, you will inevitably encounter similar concepts. With sufficient practice, you will be able to identify the approaches that are most relevant to a given problem. Of course you should not blindly follow the same methodology used on another problem, since the concepts may be applied differently. But the more adept you are at quickly recognizing the relevant approaches to apply to questions on test day, the better your GMAT score will be.