Research Methods and Standardized Tests

By Sally Raskoff

This weekend I helped a young relative with her homework. She’s six and in first grade. Her spelling homework consisted of various words, most with the “ook” ending, for which there were sentences to be copied and pictures to be drawn.

We spent quite a few minutes learning about how we took ourselves to the brook, put a worm on a hook, and, well, you can see where this is going.

I was amazed to see some of the worksheet as it didn’t make much sense to me, much less to her. One phrase had to do with how the worm “shook the hook” which, of course, left it empty. I say “of course” sarcastically since I did not assume that was what it meant but that is what the last sentence said.

This exercise left me thinking about education in general and intelligence and achievement tests in particular. We are certainly living in a time for which testing is of the utmost importance. With the remnants of the federal policy “No Child Left Behind” affecting us even at the college level, standardized testing is  everywhere.

There is the social class bias inherent in all forms of testing, especially in standardized testing. From the formation of the first intelligence tests to the SAT and more recent forms of such testing, we know that social class norms inform these tests. When I was helping that first grader with her homework, I wondered what a child living in an urban area would do with such a sentence about brooks and fishing since those would probably not be familiar in their environment. In fact, at least three of the people who were with us at the time had a conversation about what a brook is and how it might differ from a river or a creek. These were people who had done a lot of camping in their lives.

Do standardized tests really capture what we’ve learned? Do they accurately measure intelligence? Sociological research suggests that relying solely on quantitative measures such as these tests is problematic.

Looking at research methods helps us understand why. If we were to rely solely on surveys, let’s say online surveys to understand everything, would this be sufficient to fully understand human lives and society? How about if we only had observations at our disposal? What about laboratory experiments in which we simulate real life situations?

No one research method is adequate to fully understand social realities in the same way that no one type of test can really capture what we know. Each can measure pieces of those realities but to get a full picture, more information gathered from different modalities is paramount.

Replication of studies is also an important factor to consider. In the world of scientific research, we know that no one study is definitive by itself. We need to replicate or repeat studies to ensure that we really know what we think we know. The earlier study may have had some problem or things might have changed over time. Re-doing the research is important.

In the same way, testing a child and relying on that test from one time period to document their learning may not be adequate. What if on they day they took the test they had a headache, a family member had just died, or were just having a bad day? Would their performance on that one type of test be truly reflective of their knowledge and abilities?

These thoughts led me to ponder Max Weber and his approach to research methods. He wrote about the importance of verstehen , how we must take into account the perspective and experience of those who live the lives we are studying. We must use all data we can possibly gather, from those who can be interviewed about their experiences to the documents and statistical output created by institutions. In order to assess learning, we need to understand and address the need for diverse testing strategies so as to capture a holistic sense of what is learned.

In sum, scientific research and educational testing at all levels must remember that to gather accurate data, one should be using multiple sources and types of data. No one test will provide a definitive answer to a scientific question or proof of learning.

Comments

[JD Edwards User Lists]

Standardized tests are required for any freshman application to MIT. That said, tests are certainly important and you should prepare for them as best you can.

[Minh Tran]

Standardized testing used too much is akin to the same sort of elitism-justifying methods that social Darwinism and bullshit research like The Bell Curve espouse.

Its only there to make whoever's employing it subtly shift the burden of proof (in regards to educational achievement) from themselves to the people they're serving (students) instead.

"Oh, these kids scored low? They must be stupid, then! Because the test they took is the same for everyone, everyone's OPPORTUNITY to prepare for the test must also be the same, right?"

Never mind that there's cram schools like Kaplan or Elite where you can pretty much buy your SAT score. Or places like Blueprint where you pay for an LSAT score. Or private tutoring companies now where you pay to bump your child ahead of the other kids whose parents for whatever reason aren't also able to afford it...

And never mind that exams don't include any scores for teacher performance...although that's also being proposed nowadays.

Testing has become this efficiency-of-scales monster that is trying to replace the human part of education, teaching, and learning. If only the test companies cared enough to stop pushing it onto everyone. But then again, with how much money it saves and how everything's tripled in price now, do they really even need to try that hard to make everyone use it so much?

[stephen]

Standardized Tests are important to study for, due to our culture, but they shouldn’t be. At very least they need to be improved to accurately measure intelligence. While an objective number is good, in reality is not that simple, and we are bonds from fully understanding it. There is also a big cultural bias, as pointed out. Tests need to be reformed, or removed.

[Liam Cyr]

Standerdized tests are very important, but you have to take into account what happends after they are out for a few years. Teachers start to understand the nature of these tests, and while no student is the same or has the same learning and understanding abilities teachers start to taylor their teaching styles to just standerdized testing; This makes the tests not accurate and wastes the students time due to after highschool they will be doing much more than standerdized testing. Standerdized tests should exist but their should also be more applying concept tests that are not multiple choice timed tests.

[Susan]

Not all tests accurately measure ones intelligence. For example, the ACT. It is a timed test just focusing on certain subjects, rather than a wide variety. The timing of this test is a great extraction to the students, and therefore is not accurate. It is a test to base intelligence and what they know, not how fast they can get the answer right. If it were not timed, and students could actually do the test based on their knowledge, with no pressure, now that would measure intelligence.

[Zander Espinoza]

Standardized Tests are important to the commmon knowledge of the students, it help students understand the nature of test. It directly meant to show intelligence of the student but to show how the test should be taken.

[Ryheem Stokes]

I completely agree with you when you talk about how we do need to check our research methods, and make sure that we are using different methods. The point that really drove this home to me was when you stated that, "No one research method is adequate to fully understand social realities in the same way that no one type of test can really capture what we know." Sure the ACT and SAT are important to getting into college, but what about the students who do struggle with taking tests? Or the ones that might have a learning disability? All students should have a chance to go to college, and a simple standardized test should not hold a student back from achieving a college degree.

[vibram five fingers outlet]

Colleagues Who Can Make You Fat.
Ni zhge ren hen baichi
By yzi10

[Marcus L]

This article struck my interest because I am currently enrolled in an education class. I have recently been pondering becoming a teacher. This is the type of things that we have covered in the course; in fact, this is a particularly hot topic of debate in the world of education. The idea that the students are being judged on one set of tests that inaccurately demonstrates their abilities seems very unprofessional when it is really looked into. And just to add a point, if I recall correctly, these tests are not even given at the end of the courses, they are given some place in the middle, where is the sense in that? By now, I think that this problem has been identified, but coming up with a new form of examination is the real challenge.
Sociologically speaking, this article’s points are extremely relevant. The understanding of the young people’s education is necessary in order to improve the education system, and without an accurate reading of what they are learning that is going to be impossible. Maybe this is why not much has improved in the past years? If we haven’t been getting an accurate read, then it is possible that we are fixing the wrong things. Once we start using multiple research methods and replication, I think we can expect to see improvements on this matter.

[Katherine S]

I do agree with the ineffectiveness of standardized testing indicating our intellectual and scholarly capabilities. The inconvenient test taking anxiety is common among students. For a student taking many advanced classes, is a leader among their peers, and is a competent deep thinker may test lower than their average peers because they don't work well under pressure, or were simply having a bad day. Sadly, these unrepresentative tests determine how much money schools receive or if students will be admitted into their university of choice. The real question though is, how would test makers formulate and tailor a test that would truly measure the students aptitude. All students are unique and are more than a number, a grade, or a GPA.

[Robin]

I think the reason for the uproar against standardized testing is that parents believe that it's the ONLY way learning is measured. But, as you've pointed out, the best way to assess anything important is through multiple methods, not just one. One way to measure learning IS via standardized testing. I don't think we should ever throw it out. But we should add other methods to assess student learning.

When it comes to assessing teachers, the best (albeit subjective) method that I know is to ask parents. Parents will tell you, with scary accuracy, exactly who the good and bad teachers are at any school.

[Jane]

Standardized tests are an unfair judge of intelligence. The "common knowledge" is different for kids across the country so testing them all on the same material gives some kids an advantage over others. It is difficult to get rid of standardized testing, however, because there isn't one reliable way to test intelligence.

[dustin]

Standardized testing has its advantages and its disadvantages. It is an advantage to see how smart you are in core areas in comparison to the country. but it comes as a major disadvantage to those who focus their learning on other topics, and are just as smart in those areas, even though their standardized test scores say otherwise. these tests definitely have their place, but rankings shouldnt be solely based on them.