Paul's Internet Landfill/ 2014/ Picking on Certifications

Picking on Certifications

After being so harsh about industry certifications and (by proxy) the terrible private colleges that promote them in my last entry, I felt guilty. So I got my grubby hands on some (older) training materials for industry certs, and looked at some questions, more or less at random. (Before you unleash the copyright demons on me: the excerpts I am about to reproduce are short excerpts for review purposes, so chill.)

Let's start with an easy target, a book named Network+ 2005 Q&A, published by Thomson Course Technology (ISBN 1-59200-794-5). The Network+ certfication is supposed to prove that people understand networking concepts. I opened the book to page 56. Here are the first three questions:

Question 153

What IEEE 801.11 standard signals are less likely to suffer
interference from microwave ovens and cordless phones?

a. 802.11a
b. 802.11b
c. 802.11c
d. 802.11g

EXPLANATION

The 802.11a standard uses multiple frequency bands in the 5GHz
frequency range that are not as congested as the 2.4GHz band.

Question 154

What IEEE 802.11 standards are compatible?

a. 802.11a and 802.11b
b. 802.11b and 802.11c
c. 802.11g and 802.11c
d. None of the above

EXPLANATION 

802.11g, like 802.11b, uses the 2.4 GHz frequency band.

Question 155

What type of signalling does Bluetooth use?

a. DSSS
b. FDM
c. FDDI
d. FHSS

EXPLANATION

Bluetooth is a mobile wireless networking standard that uses FHSS
(frequency hopping spread spectrum) RF signaling in the 2.4-GHz
band.

I could go on, but I am feeling nauseated already. Question 155 is pure trivia. I cannot see how this question tests anything other than the ability to memorize associations between industry buzzwords. There very well may be contexts in which knowing that Bluetooth uses FHSS signalling is useful (for example, knowing that it is a 2.4GHz band can come in handy) but I do not see how memorizing these kinds of associations is a better use of one's time than actually solving problems.

Questions 153 and 154 test domain knowledge that I have in fact found useful. For example, when trying to figure out what wireless bridges we should purchase (in particular, which frequencies those bridges should use). To understand this, I had to learn something about different spectrum frequencies and their properties -- how much interference is on those bands, how well they travel through obstacles like walls, and so on. But again, this is the kind of knowledge that is relatively easy to pick up when I need it, as opposed to memorizing huge information dumps of information without a motivating context.

This underlines the criticism I made of these certifications; they mislead people into thinking that they will be educated if they memorize an enormous number of facts about technology. As it turns out, there are now 802.11n and 802.11ac standards out (which is why this book is the 2005 version of the Networking+ exam and not a more recent one). Some of the knowledge out of these questions (such as potential interference in the 2.4GHz spectrum) are transferable; others are not.

I flipped through a few other pages of this book, and the emphasis was similarly disappointing; the questions were more easily solved by memorizing factoids than by understanding or troubleshooting situations. Maybe this is just a partcularly terrible training guide. Maybe it is just a horribly out of date one, and the Network+ training-industrial complex has awesome books now. But I am not optimistic.

Now let us turn to a second book, Implementing, Managing, and Maintaining a Microsoft Window Server 2003 Network Infrastructure, Second Edition, by J.C. Mackin and Ian McLean, published by Microsoft Press in 2006 (ISBN 978-0-7356-2288-3). This time I turned to Chapter 11, on "Managing Network Security". Here are the first few review questons from this chapter:

  1. You would like to apply a new Registry setting to all the servers on your network. What is an efficient way to perform that task?

  2. You want to prevent users in the domain from running on their local machines all applications except for those from trusted publishers. How would you achieve this?

  3. You need to grant a user the right to create shares on a given server. The user should not be granted any additionmal administrative privileges. To which built-in group should you add the user?

    1. Server Operators on the local machine

    2. Server Operators at the domain level

    3. Power Users on the local machine

    4. Power Users at the domain level

The Microsoft MCSE certifications are considered more prestigious than the Network+ ones, and maybe this is why. These questions are all ones that might legitimately come up while administrating servers. Furthermore, it is not obvious that students are expected to memorize the answers to these questions, and in fact there is some modelling going on. For example, if I was asked Question 1, my thought process might go something like this:

Then I would hop on a search engine and see whether I could find the answers to these questions online. Unfortunately, one cannot consult DuckDuckGo during a MCSE exam, so one is expected to memorize a bunch of things in order to answer this question correctly.

Conceptually, the following things would be useful in answering this question:

It is true that some of these things are easier to learn about on the Internet than others.

The real weakness of this question comes when you look at the answer:

Any Registry value can be added to a security template inf file using Notepad. It should be added to the [Registry Values] section. The next time the template is applied, the Registry setting will be changed. You can write batch scripts to apply the templates to multiple machines, or you can import the template into Group Policy.

This is probably not wrong information, but a lot of it is unfamiliar to me, because it is outdated information. These days people use Group Policy Preferences to push registry edits out to computers, or at worst they write scripts that are deployed via Group Policy. I have never edited a security template inf file in my life, and despite working with Group Policy a nontrivial amount, I probably never will. However, even without reading the chapter I can tell that somebody probably emphasized the different sections of "security template inf files", because it is easy to test whether students have memorized these sections (as if memorizing them would be useful).

And what if I had chosen a different solution than the one proposed in the book? Would I have marks taken off because my solution did not match the textbook one? Probably -- and this is not a problem limited to Microsoft certification exams. While teaching undergraduate courses I have been frustrated time and time again when we punished students for thinking and problem solving their exam solutions instead of giving us the efficient and correct textbook answer. Maybe I just hate all formalized education.

The real issue with these Microsoft Certification training materials is whether they reward problem solving or whether they reward memorization. It is apparent from the other two questions that there is a fairly heavy emphasis on memorization: question 2 is about a very specific Group Policy setting. Question 3 seems interesting because it tries to probe the distinction between local machine and domain rights, but when I looked through the textbook I saw that there was a distressingly long list of built-in groups and their functionalities presented in the textbook. Presumably students are expected to memorize them all, even though many of them are not used in practice and some (such as Backup Operators) are either broken or deceptive. Time spent memorizing these distinctions is mostly time wasted. There are some concepts around these topics that are really important, and a lot of trivia that students are presumably expected to memorize anyway.

Even as I write out these criticisms I feel frustrated, because it is so easy to interpret my criticisms as saying that memorizing domain knowledge is irrelevant and useless. I do not believe that at all; understanding (and yes, memorizing) factoids about particular topics can be very helpful in solving problems, and synthesizing information is crucial to building up good mental models. But factoids are not sufficient; those factoids need to be put in context, and they need to be prioritized. I can think of only two situations in which it is helpful to dive deep into domain knowledge: if the particular domain really interests me, or if I am trying to solve a problem that requires that depth of knowledge. In both of those cases I am more likely to retain the information I have learned than by memorizing factoids out of context. And I would argue that wasting time memorizing factoids that could be better used building up modelling and troubleshooting skills is actively harmful.

Another reason I really dislike the approach of "memorize everything" to technology-related education is that not all knowledge is equally relevant. The training textbooks for the Microsoft certifications are gigantic. They are filled with information, and most of that information is irrelevant. Maybe these tomes are useful as reference books, but when actually solving problems you do not need all of that information. Often you can figure things out either by exploring the interface or by looking at the built-in documentation, and when those techniques fail then a search engine will help you a whole lot. There are things that are important to memorize, but these tend to be the exceptional things: difficult or nonintuitive concepts, and workarounds you need when dealing with buggy software. But this kind of important information tends to get lost in the forests of screenshots and trivialities.

Again, I should offer the caveat that these are older training materials, and that educational materials (and their corresponding exams) might be awesome now. But it is also clear that these materials were used to train a generation of credential-seekers, and unless these students picked up modelling and troubleshooting skills elsewhere I do not think they developed them through these training materials. By extension, I am reluctant to believe that these certifications are all that valuable in themselves, and I would not be inclined to hire somebody based solely on possessing these certs. I stand by my bigotry that seeking out people with troubleshooting and modelling skills is much more relevant than screening people on these certifications and industry buzzwords.