The Digital Reader

The results of a brief study into reading speeds were posted yesterday. The study compared the speed of reading with a:

• paper book
• PC screen
• Kindle
• iPad

You can find the report on the study here. I think this study was flawed. Something was measured, yes, but it was not the reading speed of the participants.

First, the author

If anyone besides Dr. Jakob Nielsen had done this study, I would simply point out the flaws and criticize the conclusions. But there is something you should know about Dr. Nielsen: he has _literally_ written the textbook on usability (several of them, in fact). I’ve used one in a class, and I still have it on my shelf because it  is a very good source of ideas. Have a look at his CV page and you’ll see why I’m impressed.

Second, the study

The study had 24 participants, who each read 1 short story on each of the 4 devices (paperbook, PC, Kindle, iPad). The average reading time was 17 minutes.

I have 2 reasons why I think this study is flawed.

• We don’t know if any of the test group had used an e-reader before. I’m going to assume they haven’t. If the test group had used an e-reader before then Dr. Nielsen would have remarked on it.
• The reading time was too short. If the test group had used an e-reader longer they probably  would have gotten used to them and read faster. This would have changed the results.

Third, the results

I’m just going to quote DR. Nelsen here:

The iPad measured at 6.2% lower reading speed than the printed book, whereas the Kindle measured at 10.7% slower than print. However, the difference between the two devices was not statistically significant because of the data’s fairly high variability.

…

But we can say that tablets still haven’t beaten the printed book: the difference between Kindle and the book was significant at the p<.01 level, and the difference between iPad and the book was marginally significant at p=.06.

So let’s put the results in perspective: Among the people who (1) picked up an e-reader for the first time and (2) briefly read on it, the reading speed was 6% to 10% slower than with a paper book. What he really test here was not the reading speed. No, the data we have here is on adaptability. He answered the question: How much of a speed loss will you see when you pick up an e-reader for the first time? That’s the only question that can be answered from this data. The data says nothing about the reading speed of an experienced user, and thus it is at best incomplete.

Here’s the thing: If he really wanted to test reading speed then he should have had a test group of people who already knew how to use e-readers. This way he could eliminate the chance that the test group  are slowing themselves up with newbie mistakes.

Speaking as someone who has had his hands on goodness know how many e-readers: you cannot draw a valid factual conclusion about the usability of a device based on the first 20 minutes. (Emotional, perhaps, but that’s a different matter.) The first 20 minutes is too heavily influenced by the user not being familiar with the device. I know this; I’ve been in that position more times than anyone.

P.S. There is a second set of results on user satisfaction. The PC rated low, and the iPad, Kindle, and paper book rated about the same. No big surprise there.