Another study showing students are not learning

A recent article in InsideHigherEd reported on a meeting of higher ed experts to talk about the just-released Academically Adrift: Limited Learning on College Campuses by Richard Arum and Josipa Roksa.  Both the description of the meeting, and the numerous comments made by readers of the article are quite interesting and informative.

The book describes a study of 2,300 college students moving through a range of 2 and 4 year colleges and universities. The authors utilize surveys, transcript analyses, and the Collegiate Learning Assessment (CLA) to evaluate how much the students learn during their college experience. The answer: not much.  As reported in an earlier InsideHigherEd article about this book:

• 45 percent of students "did not demonstrate any significant improvement in learning" during the first two years of college.
• 36 percent of students "did not demonstrate any significant improvement in learning" over four years of college.
• Those students who do show improvements tend to show only modest improvements. Students improved on average only 0.18 standard deviations over the first two years of college and 0.47 over four years. What this means is that a student who entered college in the 50th percentile of students in his or her cohort would move up to the 68th percentile four years later -- but that's the 68th percentile of a new group of freshmen who haven't experienced any college learning.

Participants at the meeting discussed several limitations of the study, including important questions not asked in surveys, and the use of the CLA as a surrogate for “learning”. Nevertheless, the article reports that the participants at the meeting concluded that this is a very important – and disquieting - report that may lead to some positive actions.  

Ignored in all of this discussion seem to be the large number of earlier studies and reports that reach essentially the same conclusions. Derek Bok’s Our Underachieving Colleges summarized a large number of these studies quite effectively.  As he points out, not only do not improve much in critical thinking (CLA), but they improve even less in the important area of postformal reasoning – understanding how to work with ill-structured problems. By the way, the sad results described above from Academically Adrift are almost exactly those reported by Pascarella and Terinzini in How College Affects Students V 2 based on other studies.  Consistency, unfortunately. 

My favorite quote from Bok’s book is:

“With all the controversy over the college curriculum, it is impressive to find faculty members agreeing almost unanimously that teaching students to think critically is the principal aim of undergraduate education......Ironically, the fact that college faculties rarely stop to consider what a full-blown commitment to critical thinking would entail may help to explain why they have been so quick to agree on its importance to the undergraduate program.”

From the InsideHigherEd articles and their attached comments, it seems that some are rethinking the statement that critical thinking should be (or is) the “principal aim of undergraduate education” as they are faced with increasing data that it isn’t happening.  Instead, there are statements along the lines of “ the students are certainly learning domain knowledge.” Unfortunately, that also does not seem to be the case, or more precisely, they are not learning it in the most useful and important ways. In areas where data exist, it seems we often do a good job of giving our students an “algorithmic” understanding of their domains, but few end up with an  understanding of the underlying concepts that really define the domains, or move very far in the development of expert thinking in the domains. (See previous discussion  of expert thinking in  How people learn)

In my own field of physics, tests given to students at hundreds of institutions ranging from community colleges to the best research universities show that students who have finished a physics course on the average think less like experts than when they started the course. There is a “universal” result, valid across institutional size and quality, size of class, etc. that students on the average understand less than 30% of the concepts that were taught in their class, even though they may answer traditional test questions quite well. I wrote about this in A D- in science education. All in all, this is not data that support the idea that students are currently are learning domain knowledge significantly better than they are learning how to think critically.  And, indeed, how can one really learn domain knowledge if one cannot think critically?

The good news, as pointed out in Bok’s book and numerous other works, is that learning research has shown a number of ways to change traditional approaches to teaching that lead to greatly increased student learning. It is “just” a question of implementation.

The comments made in the articles mentioned above about the weaknesses of the CLA and other tests being used to measure student learning certainly are correct. None of the instruments being used are perfect in measuring what they claim to measure.  In addition, as many critics point out, there are certainly major outcomes of education that are not being measured at all. However, a very broad spectrum of studies using a variety of instruments do show consistently that student learning is much less than we commonly claim.  

To me, this suggests two responses. First, we should not continue to spend time picking at the flaws of current instruments that seek to measure learning – rather, we should focus on developing better instruments.  At a minimum, this will require that we in higher education ask ourselves what we believe the valuable outcomes of higher education to be. In addition, however, we need to know what our students and alumni and their employers think are the valuable outcomes of higher education. Like it or not, without the approval and support of those groups, we are an endangered field.  With that information in hand, we can begin to develop better instruments to measure these outcomes.  And using these new instruments, we can work much more effectively to improve learning in our institutions.  (This is happening now in Europe through the Bologna Process. See The Bologna Process- a significant step in the modularization of higher education )

Second, for the shorter term, there are numerous approaches that have been shown to lead to better learning.  These should be more universally adopted while we wait for even better approaches to be developed.  Pressure will continue to grow on academe to show improved outcomes, and the patience of the society that supports us is finite.  

Comments

[Patricia]

Lloyd,

That is a very interesting comment, and quite the double-edged sword! Instructors are not satisfied with their students' learning and complain that they can't integrate what they have learned in theory courses to the real world. We know the students aren't able to put the pieces together they need to, but don't really want to do what is necessary to teach these students how to critically think? Why complain? As an instructor, I know that my students are novice nurses when they graduate and their critical and clinical reasoning skills are truly only going to develop as they work in their clinical settings after graduation. Another thing to consider is mentor programs and preceptorships as ways for new graduates to develop those critical thinking skills. Several nursing boards around the country are considering a leveling of licensing as students work through a successful preceptorship. Once they successfully complete one year with a preceptor, they are moved up to a fully functioning nursing license. Preceptorships, Internships, and mentoring programs are ideas other disciplines may need to consider as ways for students to develop those critical thinking skills.

[Patricia]

I have to wonder about faculty development as instructors show their concern over the fact that students are not learning at the college level. As a nursing instructor, I am bombarded at the conferences I attend on nursing education to initiate all the newest trends in education. I hear educators talking about all the innovative teaching strategies that are out there. There are tons of buzz words that I hear people use when discussing education, but how many of these educators are really walking the walk? It is obvious that lecture is not getting us anywhere, but we are dealing with the millennial students who want immediate gratification in the classroom, or they simple want to be spoon fed! Are we using lecture, PowerPoint slides, and multiple choice questions effectively, or have we gotten so complacent that we are not utilizing these strategies as the original creators intended? We are so disappointed in these strategies that we are now looking on to other means of entertaining, or teaching, our students. Are we properly trained and ready to use, for instance, problem-based learning (PBL) in the classroom? How about the clicker systems or simulation on a $40,000 mannequin in the skills lab of a nursing school? PBL takes weeks of training in order to implement it in the curriculum. Clickers are so great that my institution sent them back because of the lack of interest, or training, for the instructors. Simulation mannequins are wonderful, but difficult to learn how to use and implement in the courses. Training takes weeks! Are college educators getting the faculty development hours they need in order to implement various teaching strategies so the students can improve their learning and critical thinking skills?

Lloyd comments: There is considerable evidence that a great many faculty are not prepared to invest the time necessary to learn new and ofter radically different ways of teaching, even if it is offered. On the other side, many administrators imagine that learning how to use the new pedagogical approaches is relatively easy, and so little time needs to be set aside for faculty development. So the answer to your last question is probably that not enough time is set aside for faculty development, but that many faculty see no need for change anyway.

[Patricia]

Summer,

Do you think "critical thinking" may just be another trend? I hope we aren't using the advanced technology to take the place of teaching. I do agree that educators still need to be facilitators in this process. Sometimes we get lazy when we have all this technology. It seems that I have experienced doctors that lose their hands-on assessment skills because they want the expensive big pieces of equipment to make the diagnosis for them. They lose their skills and don't trust their assessments without this backup of advanced technology. Are we as educators losing our teaching skills because of advanced technology in the classroom?

[Summer]

Schools are quick to provide advanced technology as part of their curriculum, however are we failing students by not teaching them the basics first. I know many middle school students who don't know how to look up a word in a dictionary.

[Karla Bryan]

I think it's hard to establish critical thinking skills when there is a generation that has grown up with technology and tools that will do all of your thinking for you. I even find myself struggling at times to spell a word correctly because I have a word processing package that will take care of any error automatically. As educators we have to start at the basics of letting students know that it's ok to think for yourselves before we expect them to jump into higher order, critical thinking.

[Summer]

You are all right. Students are bored, the words "critical thinking" have lost their meaning, and we fail our students by not allowing them to make connections. The traditional classroom is not effective on the younger generation. The methods and strategies used should allow for connections. When our students see the relationship between the two, then we can begin critical thinking skills. They need to have the foundation first.

[Charles C.]

We educators have to realize that our students are exposed to so much media and technology by the time the reach middle school they are bored with traditional teacher lead learning activities. They have something to say and we have to create an environment that will give them the vehicle to allow their voices to ride.

[Patricia Loy]

I also agree with Bok’s comment from his book that faculty have no clue at the effort that goes into making sure our students know how to critically think. As a nursing instructor, critical thinking and clinical reasoning are two skills that are interrelated and are vital to the success of our novice students; their future passing of NCLEX exams, and their future positions as registered nurses. Instructors want to see that their students can take theory content and apply it to real world situations. In order to do that, I have been researching the benefit of problem-based learning. This is just one teaching strategy that is represented as promoting critical thinking skills. In discussing effort, one must understand that there are no repositories of problem-based learning scenarios for nursing; so therefore, the scenarios must be created. This can be very difficult when you have faculty with time constraints because their work load is split between teaching in the classroom and in the clinical setting. The process of using PBL as a teaching strategy requires program development for the instructors because their role as instructor changes to facilitator in the classroom. An environment that once was teacher-centered is now student-centered. This will take a great amount of adjustment for some educators. You also have to strategize how to break the students into groups, which depends on student enrollment in the course and the size of the classrooms. What new resources do we need to implement this strategy? How to we rank students’ success and measure improvement in their critical thinking skills? This is just one teaching strategy that I have discussed here. Look at the many aspects that have to be considered in order for this strategy to be implemented in a course, program, or even across a curriculum.

[Ji]

I feel “critical thinking”, and “civic engagement” etc, those terms have become so persuasive in the daily conversations of the academics that the terms have lost their meanings. The more urgent task is to develop credible and reliable instruments to measure these skills and to help instructors develop adequate commitment and knowledge base to teach these skills.

Despite all the persistent criticisms, I think US higher education is still at an advantageous position compared to higher education at other nations. It is truly a luxury to even ask: whether students have developed adequate critical thinking skills. In developing nations such as China, the majority of students are busy getting into colleges, whichever they can get in. It is until rather recently that people have begun to question the quality of higher education instruction and whether students have developed the necessary skills. For example, when national surveys such as CIRP and NSSE have been employed in the U.S. for decades to understand students’ learning and to help institutional growth, the first survey of such kind, the NSSE China, was initiated in 2007. The tight governmental control may have hampered the development of such survey instruments, yet, it is until quite recently that China has had the “luxury” to measure students’ engagement and learning in college.

[Sarah]

I don't think you could have put this any better. New education technology, even those furthered by online schools, are becoming increasingly popular and have shown to be pretty competitive. I wonder what changes we will see in another 10 years with all the new legislation about ed tech? Hopefully more education tech tools within every students reach.

Great article! You're welcome to come visit our education site http://www.university-bound.com

[Brink Gardner]

The most important course that I took during my 7 years of studying in HE was Philosophy, where I was taught that it was the duty of every human being to think through everything and anything critically. Now in business for nearly 44 years, and with no intention to stop working. This was the one course which has more than anything else influenced my life. The wonderful old Professor use to tell the story of the two students who were walking just in front of him when the one asked the other: "So what course have you enrolled for?" The other answered "Philosophy" The next question was " What can you do with that?" The answer came...."I dont know, but the Prof said, that it is more a question of what stdying this would do with you !"

[Nina Wells]

Teaching methods seem to have consistently been reviewed over the years and, in examining various learning and teaching styles, changes have been implemented and effective, when applied with effort and care for the learners. In my opinion, critical thinking skills are developed in learners through and by clearly understanding the educational needs and processes of those we teach. This, combined with keeping with the current movements (i.e. technology, globalization), all serves as the important ingredients needed to create a wholistic learning environment where critical analysis is not only welcomed but hungered.