School Reform and the Crises of Educational Testing in Taiwan

One of the biggest issues in Taiwan is the development of a new system of education. The MOE is currently trying to replace a test-driven, hierarchical system of selection with one more in-line with models derived in Western countries. The system they seem intent on implementing would be based more on direct application to schools who can then select students on a criteria they feel better suits the needs and identity of the school. High-placed MOE officials have spoken of their respect for the University of California system, but important details appear yet to be ironed out.

These reforms have come under heavy criticism. It is not clear to me how much of this criticism is aimed at the concept of reform or at the way these reforms have been progressing. Nevertheless, one of the main foci of criticism has been the significantly smaller role that paper-and-pencil tests will play in selecting and promoting students.

Paper-and-pencil placement tests have been until very recently a very large part of schooling as developed for Taiwan by the KMT government. The public rationale for this is often centered on the term ‘fair’. In this case, ‘fair’ has come to be defined as an objective number that would be replicated if the same test score were counted again by a different grader. Until only a few years, almost all educational decisions related to student placement were bound up in a web of paper-and-pencil tests administered centrally through the Ministry of Education and the Examination Yuan.

Recently, I had a significant experience relating to why the MOE would appear so desperate to change this system. And in the course of this experience, I have come to agree that such serious flaws exist in the system of high stakes competitive testing established under KMT rule that only massive reform can prepare the nation for a competitive future.

The Problem

The problem of testing in the Republic of China on Taiwan is straight forward. These tests are not at all fair. That is, they are not effective at selecting for those who best know the subject being tested. The result of this is a whole host of significant problems that include a huge washback effect that damages students and interferes with educational innovation.

Does the system of testing developed by the KMT’s MOE and the Examination Yuan actually select the best candidates? Although I implied above that the answer is no, the real answer is that no one knows. Neither the MOE nor the Examination Yuan carry out the kind of analysis on their tests that mark the Western concept of a scientific test. They are constructed according to legal mandates conceived early in the life of the KMT’s Republic of China.

You have all heard the complaint that selection by testing in the end selects for people who are good at tests. This complaint is perhaps more widespread in Taiwan than anywhere you are likely to go. Those of us whose lives are defined by a world of school and institutional excellence are not likely to come across this belief, but it is rife among Taiwanese. I was first exposed to it by a former roommate who was a 4th year Sociology student at NTU. He explained to me it was unlikely local shopkeepers in our working-class neighborhood thought he was particularly ‘smart’. Instead, he related to me, they probably thought he was just “good at taking test.”

The all-or-nothing test system creates a vast number of other problems. From the proliferation of expensive preparation schools to the stress that robs children of their childhood, newspapers are full of complaints. The MOE has its own concerns. Emphasis on unchanging goals stifles the drive for innovative teaching techniques. There are many very skilled public school teachers in Taiwan who want desperately to make their students more talented. This is difficult in a system where life changing knowledge is defined by paper-and-pencil tests whose basic design has not been changed in decades.

The Anatomy of a Good Test

While the macro aspects of the testing problem are well understood, the micro aspects are not at all clear. There is good testing available, and by this I mean, it is well understood how to create exams that consistently select for a defined standard. Tests such as TOEIC, TOEFL, LSAT and others of the Educational Testing Service are designed along these criteria. I would never defend the ETS as the bastion of fairness, but at least I do know what their tests test are selecting for. The same can not be nor could have ever been said for those of The ROC’s Ministry of Education and Examination Yuan.

The construction of the kind of tests I am talking about is complex and beyond the discussion here, but I would like to point out some aspects of the kind of tests I am talking about.

Language tests are conventionally divided into achievement and proficiency tests. For example, a test that purports to determine a student’s ability to answer the phone in English would be an achievement test. Tests such as the TOEIC or TOEFL that are designed to examine a generalized English ability are proficiency tests. It is not clearly understood what kind of test the English section of the JCEE or even other entrance examinations are supposed to be.

The design and planning of a proficiency test is quite complex and should involve the piloting of questions and an examination of results using powerful statistical procedures. It is crucial that proficiency test questions have certain statistical properties. Questions on exams such as the JCEE were written according to procedures that allowed for use on only a single administration of the test. To assure that the testing was ‘fair’, following an administration of the test, questions and correct answers were made public in newspapers.

We all know that the MOE’s English tests do not select for proficiency in English. Everyone knows that students at top schools like National Taiwan University (NTU) have achieved tremendous scores on the English section of their entrance test. This has always been true. Yet despite this, the communication ability of an average NTU student is extremely poor – unless they have received outside instruction from a commercial language school. These tests could in no sense be construed as proficiency tests. Yet if the test is not a proficiency test, what ability is it selecting for? Could it be that language exams such as the JCEE are neither proficiency nor ability tests? The thought of this distorts my mind.

Testing in Taiwan

An Examination Yuan official once described to me how professional selection examinations are made. Test writers are selected from a list of approved people and assigned writing tasks by the member of the Examination Yuan whose job it is to supervise that test’s construction. The questions are collected together and then sent across the road to the Ministry of Examinations to be copied and assembled. For this process, the selected staff of the Ministry of Examinations are sequester. Following the administration of the tests, the qualified person who wrote the examination marks the questions that he or she wrote. Selection examinations for high schools and universities use a multiple-choice format, but the point remains the same that none of the standard procedures for test development or verification are carried out on official examinations in Taiwan.

The problems created by this system of test construction are strange. Because there are only limited guidelines for the construction of test items, there is as much leeway in marking an exam as there would be in a classroom test. The answer to the question “What is the law?” depends very much on who is grading the test. As a result, the leading test preparation buxibans keep a close watch on the experts used to write professional selection tests and also on their research and class teaching materials. I have been told of bar exam prep schools finding out who is writing certain sections of the test and then prepping their students on what that professor teaches in his or her class.

The problems of the construction of multiple-choice examinations are even more difficult to understand. In the past, the Joint College Entrance Examination contained compulsory examination of subjects that included English. While multiple-choice examination seem straightforward – either the answer is correct or incorrect – this is not true. In fact, it is not true in Mathematics, and it is not at all true for language testing.

My Story

Questions can be written in such a way that actually knowing the material is a distraction to finding the correct answer. That’s right. What I’m saying is there are questions about English that have a clear and reproducible answer that are made more difficult to find by a strong knowledge of English. This fact is well understood by Taiwanese, but its full significance did not drive home to me until a very recent experience gave me insight into what kind of skills a good test taker in Taiwan truly has to master.

One of my projects at Ming Chuan is working as part of a team developing a testing system to replace the current test used for in our English program. As part of this program we’re using Rasch Modeling to assess the usefulness of individual items for future tests. Several other team members are MBA students in the business program here.

The other night, we were discussing the criteria for placing items on a test and how some items that seem good intuitively are in fact very bad items. Rasch Modeling uses a statistic called point-biserial correlation to determine the quality of items on a test. Briefly, the point-biserial correlation measures the relationship between answering a particular item correctly and your score on the entire test. Did testees who answered an item correctly also get better overall scores on the entire test? And did testees who answered the item incorrectly get lower overall scores on the entire test? The statistical test results in a number between 1.0 and -1.0, where acceptable items have as high a positive number as possible.

One of the things that our new development program has pointed out to me is just how hard it is to write test items with high positive point-biserial correlations. The program has been able to identify a large number questions from our previous testing program that had negative point-biserial correlations. That’s right. There are actually questions that students whose total test score is high are more likely to get wrong. And all of them were written by highly experienced language teachers. I suspect that what most classroom teachers think of as ‘hard’ questions or ‘detailed’ questions are in fact questions that select against language proficiency.

But back to Taiwan. Taiwanese know all this. They know about it so well there are buxibans established with the purpose of teaching students how to identify and answer this type kind of question when it appears on an official examination – such as a university entrance test. Ask your Taiwanese friends about this. All of my friends and students know answering such questions correctly is necessary for admission to the most prestigious institutions. In fact, it may even be more important than actual language proficiency, which would explain how those with poor proficiency could get higher scores.

What Does All This Mean for Taiwan?

It’s true that the high stakes, test-driven system powered Taiwan industry during its decades of growth. In fact, similar systems existed in Japan and Korea during their economic miracles and exist in China right now. But there are other characteristics of schooling that are not taken into account with this explanation, and it may be these characteristics that were more significant. And it may be that the benefit of these characteristics are coming to an end.

In a world where almost no one has the money to study at the top universities in America or Europe, cash is an advantage. In the coming competition of this century, paying tuition will not be enough, and a system that selects for students with questionable ability is bound to affect competitiveness.

The fact is that current government and education officials have all been selected by a meritocratic system based on paper-and-pencil test results and as such are extremely conservative about accepting reforms that would dramatically alter what defines excellence. Despite this, there has been so much public outcry over the high stakes testing system and its consequences that they have been forced to act. But, as I said earlier, what the final results will be is anybody’s guess.

January 21, 2007 | Permalink


Fantastic post. What are the English tests selecting for? Students with psychological insight into the construction and taking of tests (“they probably think I’m just good at tests”). Students with a demonstrated willingness to commit to the test-dispensing regime to the extent that they reshape themselves to survive in that System. Highly intelligent people who have completely internalized System values….


Posted by: Michael Turton | January 22, 2007 at 10:08

The internalization of system values…that’s a good point. I think that was my roommates point that any reform in membership selection process, no matter how small outsiders may see it, is seen by this group as radical change. ‘Outsiders’ I suppose would include not just foreigners, but Taiwanese who not members of this internally consistant group.

Posted by: Scott Sommers | January 22, 2007 at 11:12

You capture much of problem well Scott, and I think although Michael is being cynical, the internalization of values is exactly the point. However, where I part ways with Michael (I suspect) and maybe you, is that the “fair” part is not what you point out at all. Everyone knows the exams are somewhat random in their quality and consistency, I mean just watch the news during exam time, it is always a lead story how something was screwed up. Yet, at least the exams are made by real teachers, and those people are NOT government employees, but rather the job moves around to different universities, bringing in different people over time. There is a long, complex, and expensive process of putting question makers away for weeks, closing not only doors but all windows, sending food up in elevators, without any personal contact, etc. Why? Because although the exam system has many flaws, it is an attempt to get away from the biggest flaw that would swamp it in a minute: corruption. This is what the reaction to reform is based on. With so many interview-based admissions now (each with a fee), parents feel schools are just sucking down cash flow while actual admitted students are getting in due to guanxi and money.

Thus, the system is “fair” only in so far as it avoids the straight out payment for positions. But this is not the main idea either. Fair comes from the fact that when a student does do well in the exam, she is totally free to choose whatever school she likes. The exam system actually has nothing to due with rankings or school choice. For example, Tai Da has no rules about scores for students entering. There is no system of filtering based on score. The “fair” is that all the school ranks and all the department entrance scores are decided totally by the choices of the students. In this way, it is a very democratic system. Let me repeat that, because I know when I first saw it in action it totally flew in the face of what I thought the exams stood for: The exams only determine who chooses their department first, not which department they will choose.

In reality, however, by the time one actually internalizes the whole testing paradigm, which is a full time job from elementary school on, the objective becomes very blurred, and the pressure to do as others have done before is huge. Thus a high school student may have a strong interest in electronics, but he has a score that can get him into Tai Da’s animal science department. Well, parents and family may encourage that choice over the young man’s own interest. And I ask, is that wrong?

Wu Jing’s reforms should have broken this dilemma by making so many school choices, but in the end what we have gotten are many bad schools who’s goals are clearly profit. Students graduate lacking basic skills in the areas they hold degrees in. Is it any wonder that we are then right back where we started, pushing our kids to get into the school with the best name because it buys so much here?

It sounds so easy to hear things can be made “better,” yet, as your post points out, the complexity is very high, and I argue that the underlying assumptions in the society, the expectations, the behaviors (generally labeled culture) push everything right back, or at least confound such attempts at change. As a parent with children in the system I can say I don’t find the current reforms any better than the “unreformed” system. The whole thing is more complex than ever and all the same people are all still competing to get those high exam scores. Thus, all the reform is meaningless to the demographic that participated in the system previously. The big change is more university students in the system and the elimination of the vocational system. Rather than drop outs, you have students, and rather than skilled workers for a production economy you have service workers. Seems like things are working.

Posted by: Clyde Warden | January 23, 2007 at 00:08

Clyde, thank you very much for comment, It helps bring a side of this issue that both Michael and I lack. Still, I am confused about whether your concern is about the way in which this reform is being done or whether reform is even a good idea.

I think I need to restate my position on this. I am not advocating either reform or a neoconservative testing regime. My position is that the testing regime itself leaves out talented people and that it does this not just by accident, but systematically. I speculate that it is this that the MOE is concerned about. The reforms themselves are the result of a long struggle begun initially with the protests of public school teachers about curriculum, textbooks, and testing started almost as soon as teachers were legally able to speak publicly on these matters.

The place of parents in policy is complex position. While they are a key stakeholder, their interests are hardly paramount. Education as an element of a nation’s economic and business policy instructs that schools should find the most efficient balance between the workplace needs of a nation and the skills of the workforce. As such, it is important if a student with no interest at all ends up in veterinary school instead of engineering. In fact, I suspect it is one of the factors I hear expressed by Western professionals working in Taiwan when they speak of how professionals here lack autonomy.

While I can understand that some parents would be confused by the evolving standards under discussion, this is only one of the considerations that the MOE should be taking into account. The reality is that even with alternative entrance standards in place, the majority of students are still admitted via examination, and at national schools, it is probably almost everyone.

My interpretation of Taiwan is that it is a highly homogeneous society comprised of many different groups with differing histories, identities and needs – cultures if you will. While some parents no doubt ascribe to a vision of education tightly circumscribed by conservative examination, I have found a very different picture among urban professionals. Speaking on their experience and expectations about education, many have told me that public education did not at all prepare them for the current demands of their professional work lives. As such, they have no expectation it will do so for their children. Many in this group are actively seeking alternative routes to educate their children through supplementary education or even schooling outside Taiwan. Not coincidentally, it is this group that is currently fighting with the MOE over bilingual kindergartens and other forms of supplementary education.

Posted by: Scott Sommers | January 24, 2007 at 17:54

The reform is great, in so far as more competition in the market, but my point is that reform to the examination system is almost meaningless. It is not the exam that creates the mismatch of education with reality, but the choices of those who are involved in the exam system.

Those professionals you talk to may tell you one thing, but I observe they act out another. The reality is that there are only limited numbers of spaces at NTU, and everyone wants in. If you eliminate the exams in order to prevent the backwash issue, then just get ready for another type of backwash. Maybe instead of cramming for exams from the first grade, students will need to start bank accounts to prepare the red envelopes needed for payoffs when college entrance interviews come.

Again, my point is that the backwash is not from the exams, but from the choices made by those participating! Those behaviors are not going to be modified by any entrance system anyone can come up with. The exam system doesn’t place anyone, it is the choices of the students that make the placements! It is not an “all or nothing” system, and never has been. Sit down with a high school student and look at the paperwork they complete for the exams. The way they choose universities and the way the scores impact those choices. The system is only all or nothing for those who select the same schools and departments that everyone else is chasing after. This results in so many students targeting so few spaces that they all their choices are missed.

Most students know this risk very well and always have a back up choice that is known to be lower (and for some students this may even be a preferred choice, that their parents wouldn’t approve of). I think you are misinterpreting the exams and you are buying into standardized scripts that everyone in Taiwan knows how to repeat by heart. I’ve sat in rooms with parents waiting for our kids to come out of cram school classes, or exams, and we all say the same thing: “the system sucks, so much pressure, no fun, so confusing,” bla bla bla. But those same parents turn right around and push their kids into departments the children have no interest in just because it is at a national university. This is the heart of the issue, not the exams!

Posted by: Clyde A. Warden | January 27, 2007 at 22:01

Clyde is right — I was thinking in my comments about how what kind of student is being created by the system, not what kind of system there is.

To answer Clyde’s question, the system is markedly inefficient and definitely unfair. I do not see it as democratic at all; testing and the test system are fundamentally authoritarian in nature. This is for practical reasons — it favors nuclear families over extended families, mandarin users over Taiwanese users — but also because the test system, and education in CHinese culture in general, is viewed as a weeding out process rather than an enhancement process. When you make it all the way through the System, you belong to an elite, and now the choices are yours. The idea of educating everyone to that elite level, which underlies democratic education in the US, is foreign to here. Thus the system is hugely inefficient.

Clyde’s observation that families force kids into things they don’t like is spot on. That’s a gross system inefficiency.

Further, you can claim that it is producing fewer technical workers than it needs, and that might be true, but nowhere in the system does there appear to be a process that assesses future economic needs and then assigns parts of the educational system to fulfill them. Instead, intakes are regulated by the MOE according to the infrastructure of the school, not according to the needs of the economy. So there is no point of view from which the system can be viewed as efficient.


Posted by: Michael Turton | January 28, 2007 at 14:23

Michael, once again, I like your point. Thank you for helping with clarity.

Clyde, I think there are a number of points being confused here. My post was about the poor design of entrance exams. The significance of this is that poorly designed tests can not consistently select for anything. In particular, I used the example of English tests that seem to select for nothing for all. In fact, it seems from a strictly technical point of view that these tests are not tests in any sense that we would have learned about in school, baring much more resemblance to the teacher evaluations we talked about earlier. They seem to be, as Michael puts it so well, merely obstacles.

The term “all or nothing” is incorrect and that is why I did not use it in my post. The term I used is “high stakes testing”, which is a widely used term in education and correctly used in this case. My point is that high stakes testing results in structural aspects of education that are not desirable for a nation in the current position of Taiwan.

The issue of parents is something I brought up only because it is the focus of your comment. As I pointed out above, parents are only one of the stake holders the MOE should be considering. In fact, it could be argued that parents are among the least significant of these stake holders.

Anyway, the professionals I talk to have children that are either in elementary school, so no one knows what they will do when their children are old enough, or they are sending their children overseas for schooling. Regardless of what they will feel when their choices comes down to it, there is a clear feeling expressed that MOE schools can not provide education in the skills they know professional workers need. In fact, I would say that this struggle between professionals and the MOE has been one of the defining characteristics of schooling since the conception of the ROC. But that’s another story.

Posted by: Scott Sommers | January 28, 2007 at 20:51

The weeding out direction Michael brings up is good. Can you imagine any other way here in Taiwan, we are talking about a fundamentally punishment oriented culture, not a reward one. But I don’t see that the US approach is any better, where everyone gets to be above average and one third of your students cannot pass high school exit exams.

I agree, these exams, and all exams, cannot be equally fair to everyone, that is a fact. My point is that this is as good as you can get in a setting where social pressures can easily push things to be even more unfair. The MOE often tweaks with score adjustments to address the very issues you raise Michael. You say that the elite get to choose, but my point is what is it they choose? It is their attitudes, so deeply ingrained, that perpetuate the problems. Can’t we even say a big so what? With schools like CYUT, why can’t students get an even better education there–more aligned with economic realities, and their entrance was not so influenced by exam scores? Do the scores even matter? You and I know the answer, I think, and that is that even these students are so bought into the system’s assumptions that nothing changes.

Of course, it would be great if the exams could be more aligned with some economic realities, and then the prep for the exams would help everyone involved build up useful skills, but hey, let’s look at reality. These kids spend how much time on Chinese character memorization alone–without any chance in hell a more rational approach like simplified characters (not to mention bigger changes) being implemented.

Who can decide such things in a political environment? Wouldn’t it be nice for the MOE to get out ahead and do as you say, making a more systematic approach to the exams. I agree. This would benefit everyone, even those who do not do so great on the exams will at least not be wasting their time with silly exam topics that have use in reality. It doesn’t have any chance of ever happening though. Anyone who has sat in academic meetings about the exam topics and been in the scoring committees knows the very strong currents to test details that are detached from reality.

Posted by: Clyde Warden | January 28, 2007 at 21:27

I suspect this last comment came in before my response, but here goes anyway.

Clyde, the key point you make comes at the end of your comment,
“Wouldn’t it be nice for the MOE to get out ahead and do as you say, making a more systematic approach to the exams…Anyone who has sat in academic meetings about the exam topics and been in the scoring committees knows the very strong currents to test details that are detached from reality.”
This is probably why the MOE is trying to hand more of this responsibility over to the schools and other responsible parties. Despite this, the reality is almost all students are still admitted through some kind of examination.

I can understand the fear that parents have with changes coming to system that has shown only the tiniest of change since its conception 3 decades ago, but the MOE does have to consider greater issues. The situation now is that talent is systematically being bypassed because it does not conform to the test-based placement system.

It’s true that professional parents will respond to whatever system is on place to get their children the label and skills they believe they need. On the other hand, it is these parents who have the strongest belief that the system just doesn’t do it. And it is these parents’ whose hearts the MOE must win in a democratic society.

Posted by: Scott Sommers | January 28, 2007 at 21:43

This is a big assumption:
“The situation now is that talent is systematically being bypassed because it does not conform to the test-based placement system.”

With nearly everyone now taking the entrance exam getting some acceptance at some school, how is it that talent is being bypassed? Isn’t what you mean to say is that individuals don’t get to just apply to departments and get accepted based on the department’s definition of skill? That is exactly what everyone is afraid of!

Currently many departments can admit up to 1/3 of their students in exactly this way. Do you think those applying are exhibiting skill or even interest in the specific department? The answer is no. They go to cram schools where they get all the points on how to prepare a project to show the entrance committee, they get told what to say in interviews, etc.

I’ve sat on these entrance committees and watched as high school buses pulled up with 50 or more kids all ready for their interviews, and guess what, like robots they all had very similar, and at times exactly the same, things to say and show. Your whole concept of professional simply is confounded by the local traditions and behaviors Scott. The people you talk to are giving you pre-canned scripts, especially useful when faced with a Western intellectual, but the values they hold are in a very different direction.

Posted by: Clyde Warden | January 30, 2007 at 13:17

Clyde, I’m a little confused by your response. I think I mentioned above that the reforms we are talking about began under pressure from Taiwanese school teachers and parents in the early 1990’s. For the most part, anything I say could be a paraphrase of remarks that appeared in teacher’s professional magazines, reported in communique of the Legislative Assembly, or said in public groups set up specifically with the purpose of addressing educational issues such as ubiquitious test with political content.

I can agree that the current government is not doing a good job of these reforms, but the pressure to limit the affects of testing and change textbooks began more than a decade ago by Taiwanese urban professionals. In fact, I would say the historical record shows this goes back almost a century, but that’s another story.

Posted by: Scott Sommers | January 30, 2007 at 13:53

Let me make this point with this example: The MOE “limited” the effects of the exams by not allowing schools to continue classes during vacation time (used to cram more for the exams). Great.

The result is, before vacations teachers ask students who volunteers to come to school during the break (normally everyone volunteers). And because the time is officially not school time guess what–parents get to pay in cash to support the effort. My daughter is in class now even though the semester is over.

Words differ from behavior and I live in the world of behavior Scott, and I’m telling you there is NO change among a certain large segment of the population in their own emphasis on the exams (and as I said earlier, it is not the exams but the opportunity to choose a result that is socially preferred) and nothing done by anyone has changed it over the past twenty years I’ve been here. Do you see what I’m saying? There have been many structural reforms, but I’m adding to your point by saying those reforms do not change behavior and it is the behavior that causes the issues not the government regulations.

Posted by: Clyde Warden | February 02, 2007 at 22:56

I don’t doubt that some segments of Taiwan have not changed at all perhaps even in decades. This might (or might not) be a large segment of Taiwan or even a majority. My point is that what is called Taiwan ‘society’ is a heterogeneous group composed of diverse groups with largely different histories, interests, and cultures.

But in fact, in all the excitement of these comments, I have become sidetracked from what was intended as my main point. Clyde, your comments focus on university entrance exams, but my post was intended as a larger critique of the way that testing has been used in the ROC. Ironically, my critique of entrance and professional testing was the same as your comments about teacher evaluations; without proper measurement of results, test questions can not be written so they systematically select for their stated purpose. In Taiwan, this problem has been compounded by test specifications determined not by professionals themselves but by legal mandates. As I put it in this post,
test questions frequently address matters irrelevant to the practice of the profession being selected. Professionals and even Examination Yuan officials do not want such questions included.

Posted by: Scott Sommers | February 03, 2007 at 00:17


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: