Question Difficulty in GMAT Official Guides

Starting with the 11th Edition of the Official Guides, the Graduate Management Admission Council placed questions in order of difficulty. This is a welcome change from prior editions, because students can now choose practice questions at the appropriate difficulty level. Beginning students need not tackle difficult questions that may cause them to feel overwhelmed and frustrated, nor do advanced students need to waste time on questions that are well below their capabilities.

Mysterious Methodology

The methodology that the GMAC uses to assign difficulty ratings to questions remains a mystery, however. Although there is a clear overall correlation between question placement and GMAT Genius’ assessment of difficulty, the correlation is far from perfect and there are many outliers, particularly with math. Let’s consider the Problem Solving section of the GMAT Official Guide (13th Ed.).

Problem Solving Analysis

There are 230 questions in this section, excluding questions in the Diagnostic Test chapter of the book. GMAT Genius assigns difficulty at five different levels: Super Easy, Easy, Moderate, Hard, and Very Hard. To translate this into numbers, we can assign points of 1, 2, 3, 4, and 5 to each of these levels, respectively. With perfect correlation and distribution, we should see the first 46 questions (230 divided by 5) all at a Super Easy level (average difficulty 1.0), the next 46 questions all at an Easy level (average difficulty 2.0), and so on. Instead, here is how we would rank question difficulty:

As shown in the table, average question difficulty (based on our assessment) does not increase much in the last three quintiles. In fact, the middle quintile is actually slightly harder than the fourth quintile. Surprisingly, we consider five of the top quintile questions “Easy,” including #220, which is supposed to be one of the hardest questions in the entire set. We also consider question #117 in the third quintile “Very Hard,” even though it barely passes the halfway mark. Yet the GMAC considers this question easier than #130, which we rate as “Super Easy.”

Disparity Among Similar Questions

We see this disparity even among similar questions. Questions #178 and #186 both entail overlapping sets, and based on the ordering, the GMAC considers #186 slightly harder than #178. Yet we rate #178 as “Very Hard” and #186 as “Moderate” because the former is a complex application with three groups whereas the latter is a straightforward application with two groups. As another example, consider #137 and #203 that we described in our prior post about repetitive math questions. These two questions are very similar, yet the GMAC considers the latter much harder than the former. By contrast, we rate both these questions as “Hard” and actually consider the latter slightly easier because the numbers are much easier.

Our assessment of difficulty is admittedly somewhat subjective, but probably more realistic than how the GMAC assigns difficulty. Our difficulty ratings take into account our observations of how students tend to find the questions in term of difficulty, the ease of the calculations involved, and the length of the Official Guide explanation, except when the explanation is inefficient or misses a shortcut. For verbal questions, we also take into account the difficulty of the incorrect answers, since process of elimination can be particularly helpful on verbal. Note that our assessment of difficulty skews towards the center; a question would need to be incredibly easy or incredibly difficult to qualify as “Super Easy” or “Very Hard”, respectively. The GMAC most likely has a much greater distribution in its difficulty assignments.

Conclusions

A key takeaway from this analysis is that you should not assume that the Official Guides are an objective measure of question difficulty. Although we don’t know how the GMAC assigns difficulty and would expect a few differences of opinion, we can confidently claim that quite a few questions are ordered incorrectly. So if you encounter a few questions in the Official Guides that are hard for you, it does not necessarily mean that all subsequent questions are beyond your reach. Another implication of this analysis is that the Official Guides do not contain enough difficult practice questions, a complaint we often hear from our advanced students. The GMAC has released two relatively new study products (Exam Pack 1 and Question Pack 1), so let’s hope that Difficult Pack 1 is in the works.