Excerpts from Make it Stick: The Science of Successful Learning by Peter C. Brown
The book in its entirety is available through Amazon.com:
http://www.amazon.com/Make-Stick-Science-Successful-Learning/dp/0674729013/ref=sr_1_1?ie=UTF8&qid=1431657835&sr=8-1&keywords=make+it+stick
Learning is an acquired skill, and the most effective strategies are often counterintuitive. Learning is deeper and more durable when it’s effortful. Learning that’s easy is like writing in sand, here today and gone tomorrow. We are poor judges of when we are learning well and when we’re not. When the going is harder and slower and it doesn’t feel productive, we are drawn to strategies that feel more fruitful, unaware that the gains from these strategies are often temporary.
Rereading text and massed practice of a skill or new knowledge are by far the preferred study strategies of learners of all stripes, but they’re also among the least productive.
Cramming for exams is an example. Rereading and massed practice give rise to feelings of fluency that are taken to be signs of mastery, but for true mastery or durability these strategies are largely a waste of time.
Retrieval practice—recalling facts or concepts or events from memory—is a more effective learning strategy than review by rereading. Flashcards are a simple example. Retrieval strengthens the memory and interrupts forgetting. A single, simple quiz after reading a text or hearing a lecture produces better learning and remembering than rereading the text or reviewing lecture notes. While the brain is not a muscle that gets stronger with exercise, the neural pathways that make up a body of learning do get stronger, when the memory is retrieved and the learning is practiced. Periodic practice arrests forgetting, strengthens retrieval routes, and is essential for hanging onto the knowledge you want to gain. When you space out practice at a task and get a little rusty between sessions, or you interleave the practice of two or more subjects, retrieval is harder and feels less productive, but the effort produces longer lasting learning and enables more versatile application of it in later settings.
Trying to solve a problem before being taught the solution leads to better learning, even when errors are made in the attempt. The popular notion that you learn better when you receive instruction in a form consistent with your preferred learning style, for example as an auditory or visual learner, is not supported by the empirical research.
People do have multiple forms of intelligence to bring to bear on learning, and you learn better when you “go wide,” drawing on all of your aptitudes and resourcefulness, than when you limit instruction or experience to the style you find most amenable.
When you’re adept at extracting the underlying principles or “rules” that differentiate types of problems, you’re more successful at picking the right solutions in unfamiliar situations. This skill is better acquired through interleaved and varied practice than massed practice.
Interleaving the identification of bird types or the works of oil painters improves your ability both to learn the unifying attributes within a type and to differentiate between types, improving your skill at categorizing new specimens you encounter later. We’re all susceptible to illusions.
You build better mastery when you use testing as a tool to identify and bring up your areas of weakness. All new learning requires a foundation of prior knowledge. You need to know how to land a twin engine plane on two engines before you can learn to land it on one.
If you practice elaboration, there’s no known limit to how much you can learn. Elaboration is the process of giving new material meaning by expressing it in your own words and connecting it with what you already know. The more you can explain about the way your new learning relates to your prior knowledge, the stronger your grasp of the new learning will be, and the more connections you create that will help you remember it later. Warm air can hold more moisture than cold air; to know that this is true in your own experience, you can think of the drip of water from the back of an air conditioner or the way a stifling summer day turns cooler out the back side of a sudden thunderstorm. Evaporation has a cooling effect: you know this because a humid day at your uncle’s in Atlanta feels hotter than a dry one at your cousin’s in Phoenix, where your sweat disappears even before your skin feels damp. When you study the principles of heat transfer, you understand conduction from warming your hands around a hot cup of cocoa; radiation from the way the sun pools in the den on a wintry day; convection from the life-saving blast of A/C as your uncle squires you slowly through his favorite back alley haunts of Atlanta.
Putting new knowledge into a larger context helps learning. For example, the more of the unfolding story of history you know, the more of it you can learn. And the more ways you give that story meaning, say by connecting it to your understanding of human ambition and the untidiness of fate, the better the story stays with you. Likewise, if you’re trying to learn an abstraction, like the principle of angular momentum, it’s easier when you ground it in something concrete that you already know, like the way a figure skater’s rotation speeds up as she draws her arms to her chest.
People who learn to extract the key ideas from new material and organize them into a mental model and connect that model to prior knowledge show an advantage in learning complex mastery. A mental model is a mental representation.
Many people believe that their intellectual ability is hardwired from birth, and that failure to meet a learning challenge is an indictment of their native ability. But every time you learn something new, you change the brain—the residue of your experiences is stored. It’s true that we start life with the gift of our genes, but it’s also true that we become capable through the learning and development of mental models that enable us to reason, solve, and create. In other words, the elements that shape your intellectual abilities lie to a surprising extent within your own control. Understanding that this is so enables you to see failure as a badge of effort and a source of useful information—the need to dig deeper or to try a different strategy. The need to understand that when learning is hard, you’re doing important work. To understand that striving and setbacks, as in any action video game or new BMX bike stunt, are essential if you are to surpass your current level of performance toward true expertise. Making mistakes and correcting them builds the bridges to advanced learning.
When learning is harder, it’s stronger and lasts longer. It’s widely believed by teachers, trainers, and coaches that the most effective way to master a new skill is to give it dogged, single-minded focus, practicing over and over until you’ve got it down. Our faith in this runs deep, because most of us see fast gains during the learning phase of massed practice. What’s apparent from the research is that gains achieved during massed practice are transitory and melt away quickly. The finding that rereading textbooks is often labor in vain ought to send a chill up the spines of educators and learners, because it’s the number one study strategy of most people—including more than 80 percent of college students in some surveys—and is central in what we tell ourselves to do during the hours we dedicate to learning. Rereading has three strikes against it. It is time consuming. It doesn’t result in durable memory. And it often involves a kind of unwitting self-deception, as growing familiarity with the text comes to feel like mastery of the content.
Learning is stronger when it matters, when the abstract is made concrete and personal.
A flight simulator provides retrieval practice, and the practice is spaced, interleaved, and varied and involves as far as possible the same mental processes.
Subsequent studies by many researchers have pressed further into questions of whether repeated exposure or longer periods of holding an idea in mind contribute to later recall, and these studies have confirmed and elaborated on the findings that repetition by itself does not lead to good long-term memory.
In the other situation, students read the material the first time and then waited some days before they reread it. This group, having done spaced readings of the text, performed better on the test than the group who did not reread the material.
What’s the conclusion? It makes sense to reread a text once if there’s been a meaningful lapse of time since the first reading, but doing multiple readings in close succession is a time-consuming study strategy that yields negligible benefits at the expense of much more effective strategies that take less time. Yet surveys of college students confirm what professors have long known: highlighting, underlining, and sustained poring over notes and texts are the most-used study strategies, by far.
Rising familiarity with a text and fluency in reading can create an illusion of mastery. As any professor will attest, students work hard to capture the precise wording of phrases they hear in class lectures, laboring under the misapprehension that the essence of the subject lies in the syntax in which it’s described.
Mastering the lecture or the text is not the same as mastering the ideas behind them. However, repeated reading provides the illusion of mastery of the underlying ideas.
Had he used the set of key concepts in the back of each chapter to test himself? Could he look at a concept like “conditioned stimulus,” define it, and use it in a paragraph? While he was reading, had he thought of converting the main points of the text into a series of questions and then later tried to answer them while he was studying? Had he at least rephrased the main ideas in his own words as he read? Had he tried to relate them to what he already knew? Had he looked for examples outside the text?
The emphasis here is ours. We make it to drive home the point that students who don’t quiz themselves (and most do not) tend to overestimate how well they have mastered class material. Why? When they hear a lecture or read a text that is a paragon of clarity, the ease with which they follow the argument gives them the feeling that they already know it and don’t need to study it. In other words, they tend not to know what they don’t know; when put to the test, they find they cannot recall the critical ideas or apply them in a new context. Likewise, when they’ve reread their lecture notes and texts to the point of fluency, their fluency gives them the false sense that they’re in possession of the underlying content, principles, and implications that constitute real learning, confident that they can recall them at a moment’s notice. The upshot is that even the most diligent students are often hobbled by two liabilities: a failure to know the areas where their learning is weak.
Notwithstanding the pitfalls of standardized testing, what we really ought to ask is how to do better at building knowledge and creativity, for without knowledge you don’t have the foundation for the higher-level skills of analysis, synthesis, and creative problem solving.
Mastery requires both the possession of ready knowledge and the conceptual understanding of how to use it.
But if we stop thinking of testing as a dipstick to measure learning—if we think of it as practicing retrieval of learning from memory rather than “testing,” we open ourselves to another possibility: the use of testing as a tool for learning. One of the most striking research findings is the power of active retrieval—testing—to strengthen memory, and that the more effortful the retrieval, the stronger the benefit. Think flight simulator versus PowerPoint lecture.
For the class in question, at a middle school in Columbia, Illinois, researchers arranged for part of the material covered during the course to be the subject of low-stakes quizzing (with feedback) at three points in the semester. Another part of the material was never quizzed but was studied three times in review. In a test a month later, which material was better recalled? The students averaged A- on the material that was quizzed and C+ on the material that was not quizzed but reviewed.
Both of these cases—the research in the classroom and the experience of Matt Brown in updating his knowledge—point to the critical role of retrieval practice in keeping our knowledge accessible to us when we need it.
One of the best habits a learner can instill in herself is regular self-quizzing to recalibrate her understanding of what she does and does not know. Second Lieutenant Kiley Hunkler, a 2013 graduate of West Point and winner of a Rhodes Scholarship, whom we write about in Chapter 8, uses the phrase “shooting an azimuth” to describe how she takes practice tests to help refocus her studying. In overland navigation, shooting an azimuth means climbing to a height, sighting an object on the horizon in the direction you’re traveling, and adjusting your compass heading to make sure you’re still gaining on your objective as you beat through the forest below.
Various forms of retrieval practice, such as low-stakes quizzing and self-testing, spacing out practice, interleaving the practice of different but related topics or skills, trying to solve a problem before being taught the solution, distilling the underlying principles or rules that differentiate types of problems, and so on.
Reflection can involve several cognitive activities that lead to stronger learning: retrieving knowledge and earlier training from memory, connecting these to new experiences, and visualizing and mentally rehearsing what you might do differently next time.
Recalling it over and over, practicing it over and over. That’s just so important.
We lose something like 70 percent of what we’ve just heard or read. After that, forgetting begins to slow, and the last 30 percent or so falls away more slowly, but the lesson is clear: a central challenge to improving the way we learn is finding a way to interrupt the process of forgetting.
In his essay on memory, Aristotle wrote: “exercise in repeatedly recalling a thing strengthens the memory.” Francis Bacon wrote about this phenomenon, as did the psychologist William James. Today, we know from empirical research that practicing retrieval makes learning stick far better than reexposure to the original material does. This is the testing effect, also known as the retrieval-practice effect. To be most effective, retrieval must be repeated again and again, in spaced out sessions so that the recall, rather than becoming a mindless recitation, requires some cognitive effort.
Yet despite what research and personal experience tell us about the power of testing as a learning tool, teachers and students in traditional educational settings rarely use it as such, and the technique remains little understood or utilized by teachers or students as a learning tool in traditional educational settings. Far from it. In 2010 the New York Times reported on a scientific study that showed that students who read a passage of text and then took a test asking them to recall what they had read retained an astonishing 50 percent more of the information a week later than students who had not been tested.
Pitting the learning of basic knowledge against the development of creative thinking is a false choice. Both need to be cultivated.
The experiment showed a couple of interesting results: the longer the first test was delayed, the greater the forgetting, and second, once a student had taken a test, the forgetting nearly stopped, and the student’s score on subsequent tests dropped very little.
In 1978, researchers found that massed studying (cramming) leads to higher scores on an immediate test but results in faster forgetting compared to practicing retrieval. In a second test two days after an initial test, the crammers had forgotten 50 percent of what they had been able to recall on the initial test, while those who had spent the same period practicing retrieval instead of studying had forgotten only 13 percent of the information recalled initially.
Those students who were tested immediately after exposure recalled 53 percent of the objects on this initial test but only 39 percent a week later. On the other hand, a group of students who learned the same material but were not tested at all until a week later recalled 28 percent.
Thus, and in agreement with later research, multiple sessions of retrieval practice are generally better than one, especially if the test sessions are spaced out. In another study, researchers showed that simply asking a subject to fill in a word’s missing letters resulted in better memory of the word. Consider a list of word pairs. For a pair like foot-shoe, those who studied the pair intact had lower subsequent recall than those who studied the pair from a clue as obvious as foot-s_ _e. This experiment was a demonstration of what researchers call the “generation effect.” The modest effort required to generate the cued answer while studying the pairs strengthened memory of the target word tested later (shoe).
Interestingly, this study found that the ability to recall the word pair on later tests was greater if the practice retrieval was delayed by twenty intervening word pairs than when it came immediately after first studying the pair. Why would that be? One argument suggested that the greater effort required by the delayed recall solidified the memory better. Researchers began to ask whether the schedule of testing mattered. The answer is yes. When retrieval practice is spaced, allowing some forgetting to occur between tests, it leads to stronger long-term retention than when it is massed.
The results were compelling: The kids scored a full grade level higher on the material that had been quizzed than on the material that had not been quizzed. Moreover, test results for the material that had been reviewed as statements of fact but not quizzed were no better than those for the nonreviewed material. Again, mere rereading does not much help.
The testing effect persisted eight months later at the end-of-year exams, confirming what many laboratory studies have shown about the long-term benefits of retrieval practice. The effect doubtless would have been greater if the retrieval practice had continued and occurred once a month, say, in the intervening months.
Patrice Bain’s sixth grade social studies classes continue today to follow a schedule of quizzes before lessons, quizzes after lessons, and then a review quiz prior to the chapter test.
After reading passages on the history of slavery, for example, his students are asked to write down ten facts about slavery they hadn’t known before reading the passages. You don’t need electronic gadgetry to practice retrieval.
Retrieval and elaboration; again, no technology required. Quizzes at Columbia Middle School are not onerous events. Following completion of the research studies, students’ views were surveyed on this question. Sixty-four percent said the quizzing reduced their anxiety over unit exams, and 89 percent felt it increased learning. The kids expressed disappointment on days when clickers were not used, because the activity broke up the teacher’s lecture and proved enjoyable. Principal Chamberlain, when asked what he thought the study results indicated, replied simply: “Retrieval practice has a significant impact on kids’ learning. This is telling us that it’s valuable, and that teachers are well advised to incorporate it into their instructional technique.”
“The quality of discussions in class has gone way up. I see that big a difference in their written work, just by going from three exams to nine quizzes.”
After a delay of two days, the students who took the initial test recalled more of the material than those who simply restudied it (68 v. 54 percent), and this advantage was sustained a week later (56 v. 42 percent).
Another experiment found that after one week a study-only group showed the most forgetting of what they initially had been able to recall, forgetting 52 percent, compared to a repeated-testing group, who forgot only 10 percent.
How does giving feedback on wrong answers to test questions affect learning? Studies show that giving feedback strengthens retention more than testing alone does, and, interestingly, some evidence shows that delaying the feedback briefly produces better long-term learning than immediate feedback.
Another idea holds that frequent interruptions for feedback make the learning sessions too variable, preventing establishment of a stabilized pattern of performance.
In the classroom, delayed feedback also yields better long-term learning than immediate feedback does.
Tests that require the learner to supply the answer, like an essay or short-answer test, or simply practice with flashcards, appear to be more effective than simple recognition tests like multiple choice or true/false tests.
While any kind of retrieval practice generally benefits learning, the implication seems to be that where more cognitive effort is required for retrieval, greater retention results.
Whichever theories science eventually tells us are correct about how repeated retrieval strengthens memory, empirical research shows us that the testing effect is real—that the act of retrieving a memory changes the memory, making it easier to retrieve again later.
Is repeated testing simply a way to expedite rote learning? In fact, research indicates that testing, compared to rereading, can facilitate better transfer of knowledge to new contexts and problems, and that it improves one’s ability to retain and retrieve material that is related but not tested. Further research is needed on this point, but it seems that retrieval practice can make information more accessible when it is needed in various contexts.
Students who have been quizzed have a double advantage over those who have not: a more accurate sense of what they know and don’t know, and the strengthening of learning that accrues from retrieval practice.
Are there any further, indirect benefits of regular, low-stakes classroom testing? Besides strengthening learning and retention, a regime of this kind of testing improves student attendance. It increases studying before class (because students know they’ll be quizzed), increases attentiveness during class if students are tested at the end of class, and enables students to better calibrate what they know and where they need to bone up.
Frequent low-stakes testing helps dial down test anxiety among students by diversifying the consequences over a much larger sample: no single test is a make-or-break event. And this kind of testing enables instructors to identify gaps in students’ understanding and adapt their instruction to fill them. These benefits of low-stakes testing accrue whether instruction is delivered online or in the classroom.
Practice at retrieving new knowledge or skill from memory is a potent tool for learning and durable retention.
Effortful retrieval makes for stronger learning and retention. We’re easily seduced into believing that learning is better when it’s easier, but the research shows the opposite: when the mind has to work, learning sticks better. The greater the effort to retrieve learning, provided that you succeed, the more that learning is strengthened by retrieval. After an initial test, delaying subsequent retrieval practice is more potent for reinforcing retention than immediate practice, because delayed retrieval requires more effort.
Repeated retrieval not only makes memories more durable but produces knowledge that can be retrieved more readily, in more varied settings, and applied to a wider variety of problems. While cramming can produce better scores on an immediate exam, the advantage quickly fades because there is much greater forgetting after rereading than after retrieval practice.
Simply including one test (retrieval practice) in a class yields a large improvement in final exam scores, and gains continue to increase as the frequency of classroom testing increases.
The greater the effort at retrieval, the more will be retained.
Students who take practice tests have a better grasp of their progress than those who simply reread the material. Similarly, such testing enables an instructor to spot gaps and misconceptions and adapt instruction to correct them.
Giving students corrective feedback after tests keeps them from incorrectly retaining material they have misunderstood and produces better learning of the correct answers. Students in classes that incorporate low-stakes quizzing come to embrace the practice. Students who are tested frequently rate their classes more favorably.
A group of eight-year-olds practiced tossing beanbags into buckets in gym class. Half of the kids tossed into a bucket three feet away. The other half mixed it up by tossing into buckets two feet and four feet away. After twelve weeks of this they were all tested on tossing into a three-foot bucket. The kids who did the best by far were those who’d practiced on two- and four-foot buckets but never on three-foot buckets.1 Why is this? We will come back to the beanbags, but first a little insight into a widely held myth about how we learn.
While practicing is vital to learning and memory, studies have shown that practice is far more effective when it’s broken into separate periods of training that are spaced out. The rapid gains produced by massed practice are often evident, but the rapid forgetting that follows is not. Practice that’s spaced out, interleaved with other learning, and varied produces better mastery, longer retention, and more versatility.
But these benefits come at a price: when practice is spaced, interleaved, and varied, it requires more effort. You feel the increased effort, but not the benefits the effort produces. Learning feels slower from this kind of practice, and you don’t get the rapid improvements and affirmations you’re accustomed to seeing from massed practice. Even in studies where the participants have shown superior results from spaced learning, they don’t perceive the improvement; they believe they learned better on the material where practice was massed.
Cramming for exams is a form of massed practice.
Spacing out your practice feels less productive for the very reason that some forgetting has set in and you’ve got to work harder to recall the concepts.
What you don’t sense in the moment is that this added effort is making the learning stronger.
The benefits of spacing out practice sessions are long established, but for a vivid example consider this study of thirty-eight surgical residents. They took a series of four short lessons in microsurgery: how to reattach tiny vessels. Each lesson included some instruction followed by some practice. Half the docs completed all four lessons in a single day, which is the normal in-service schedule. The others completed the same four lessons but with a week’s interval between them. In a test given a month after their last session, those whose lessons had been spaced a week apart outperformed their colleagues in all areas—elapsed time to complete a surgery, number of hand movements, and success at reattaching the severed, pulsating aortas of live rats. The difference in performance between the two groups was impressive. The residents who had taken all four sessions in a single day not only scored lower on all measures, but 16 percent of them damaged the rats’ vessels beyond repair and were unable to complete their surgeries. Why is spaced practice more effective than massed practice? It appears that embedding new learning in long-term memory requires a process of consolidation, in which memory traces (the brain’s representations of the new learning) are strengthened, given meaning, and connected to prior knowledge—a process that unfolds over hours and may take several days. Rapid-fire practice leans on short-term memory. Durable learning, however, requires time for mental rehearsal and the other processes of consolidation. Hence, spaced practice works better. The increased effort required to retrieve the learning after a little forgetting has the effect of retriggering consolidation, further strengthening memory.
The mixing of problem types, which boosted final test performance by a remarkable 215 percent, actually impeded performance during initial learning.
But the research shows unequivocally that mastery and long-term retention are much better if you interleave practice than if you mass it.
The basic idea is that varied practice—like tossing your beanbags into baskets at mixed distances—improves your ability to transfer learning from one situation and apply it successfully to another.
The benefits of variable practice for cognitive as opposed to motor skills learning were shown in a recent experiment that adapted the beanbag test to verbal learning: in this case, the students solved anagrams–that is, they rearranged letters to form words (tmoce becomes comet). Some subjects practiced the same anagram over and over, whereas others practiced multiple anagrams for the word. When they were all tested on the same anagram that the former group had practiced on, the latter group performed better on it! The same benefits will apply whether you are practicing to identify tree species, differentiate the principles of case law, or master a new computer program.
Compared to massed practice, a significant advantage of interleaving and variation is that they help us learn better how to assess context and discriminate between problems, selecting and applying the correct solution from a range of possibilities.
When you have learned under conditions of massed or blocked repetition, you have had no practice on that critical sorting process.
But this is the way life usually unfolds: problems and opportunities come at us unpredictably, out of sequence.
Several studies have demonstrated the improved powers of discrimination to be gained through interleaved and varied practice.
The researchers were wrong. The commonalities among one painter’s works that the students learned through massed practice proved less useful than the differences between the works of multiple painters that the students learned through interleaving. Interleaving enabled better discrimination and produced better scores on a later test that required matching the works with their painters.
To paraphrase a conclusion from one of these studies, recall and recognition require “factual knowledge,” considered to be a lower level of learning than “conceptual knowledge.”
Conceptual knowledge requires an understanding of the interrelationships of the basic elements within a larger structure that enable them to function together. Conceptual knowledge is required for classification. Following this logic, some people argue that practicing retrieval of facts and exemplars would fall short as a strategy for comprehending general characteristics that are required for higher levels of intellectual behavior.
Strategies of learning that help students identify and discern complex prototypes (family resemblances) can help them grasp the kinds of contextual and functional differences that go beyond the acquisition of simple forms of knowledge and reach into the higher sphere of comprehension.
“The reason variety is important is it helps us see more nuances in the things that we can compare against.”
In other words, the kind of retrieval practice that proves most effective is one that reflects what you’ll be doing with the knowledge later.
“Practice like you play and you will play like you practice.”
Book learning is not enough in these cases; actual hands-on practice is needed.
One form of practice that helps us learn from experience, as the neurosurgeon Mike Ebersold recounted in Chapter 2, is reflection. Some people are more given to the act of reflection than others, so Doug Larsen has broadened his research to study how you might structure reflection as an integral part of the training, helping students cultivate it as a habit.
He is experimenting with requiring students to write daily or weekly summaries of what they did, how it worked, and what they might do differently next time to get better results.
Dooley believes that (1) you have to keep practicing the fundamentals from time to time, forever, so you keep them sharp, otherwise you’re cooked, but (2) you need to change it up in practice because too much repetition is boring. The position coaches work with players individually on specific skills and then on how they’re playing their positions during team practice.
With continued retrieval, complex material can become second nature to a person and the mnemonic cues are no longer needed: you consolidate concepts like Newton’s 3 laws of motion into mental models that you use as a kind of shorthand.
It comes down to the simple but no less profound truth that effortful learning changes the brain, building new connections and capability. This single fact—that our intellectual abilities are not fixed from birth but are, to a considerable degree, ours to shape—is a resounding answer to the nagging voice that too often asks us “Why bother?” We make the effort because the effort itself extends the learning.
Third, learning is an acquired skill, and the most effective strategies are often counterintuitive.
