Students missing test questions due to computer interface issues

I’ve had a series where I’ve been looking at Common Core exams delivered by computer looking for issues. Mathematical issues did crop up, but the more subtle and universal ones were about the interface.

Part 1: Observations on the PARCC sample Algebra I exam
Part 2: Observations on the PARCC sample Algebra II exam
Part 3: Observations on the PARCC sample Geometry exam
Part 4: Observations on the SAGE sample exam

While the above observations were from my experience with design and education, I haven’t had a chance to experience actual students trying the problems.

Now that I have, I want to focus on one problem in particular which is on the AIR samples for Arizona, Utah, and Florida. First, here is the blank version of the question:

Here is the intended correct answer:

Student Issue #1:

In this case, it appears a student didn’t follow the “Drag D to the grid to label this vertex” instruction.

However, at least one student did see the instruction but was baffled how to carry it out (the “D” can be easy to miss the way it is on the top of a large white-space). Even given a student who missed that particular instruction, is the lack of dragging a letter really the reason you want students to miss the points?

Also, students who are used to labeling points do so directly next to the point; dragging to label is an entirely different reflex. Even a student used to Geogebra would get this problem wrong, as points in Geogebra are labeled automatically. I do not know of any automated graphical interface other than this test which require the user to add a label separately.

Student Issue #2:

Again, it appears possible the full directions were not read, but a fair number of students were unaware line connection was even possible, because they missed the existence of the “connect line” tool.

In problems where the primary activity was to create a line this was not an issue, but since the primary mathematical step here involves figuring out the correct place to add a point, students became blind to the line interface.

In truth I would prefer it if the lines were added automatically; clearly their presence is not what is really being tested here.

Student Issue #3:

This one’s in the department of “I wouldn’t have predicted it” problems, but it looks like the student just tried their best at making a parallelogram and felt like it was fine to add another point as long as it was close to “C”. The freedom of being allowed to add extra points suggests this. If the quadrilateral was formed automatically with the addition of point “D” (as I already suggested) this problem would be avoided. Another possibility would be to have the D “attached” the point as it gets dragged to the location, and to disallow having more than one point being present.

Advertisement

Some comments on the Khan Academy videos

Making a teaching video is much harder than it appears. My Q*Bert video was built out of 30 second chunks and many, many retakes. Even then I had to do some sound editing of ‘well’ and ‘um’ and so forth. I don’t have this problem teaching to students in person (even if I’m doing most of the talking) because I can always pause; “dead air” isn’t nearly as deadly. Salman Khan’s use of continuous takes at 8+ minutes is quite a feat.

Video is not just spoken text. Salman doesn’t fully utilize the video format, but he wouldn’t be able to be so prolific otherwise. Even so, what he does is non-trivially different from text. Consider Why Gravity Gets So Strong.

Here’s a shot of the video early on…

… and here’s a shot later.

In a book format one could have a diagram that is progressively developed (but often is not for space concerns), but even given that in the video there is focus and movement going on that gives a more tactile sense of mass that a static image cannot convey.

Video cannot utilize student response the same way as other forms of interaction. I had several points in my Q*Bert video with requests to pause, because it was intended for front-of-classroom use by a teacher. However, some students have seen (and used) it solo. While I did receive one email inquiring about the puzzle at the end, the number of students working alone that paused the video in the middle when prompted I estimate roughly between zero and zero.

Hence, saying “let’s think about this” and giving several minutes wait time is not plausible. (I do have some ideas for how I might get students more willing to pause and may incorporate them in my upcoming video, but I don’t have high odds that they will work.)

Salman Khan cares getting students to understand why things work more than he is alleged. His Slope and Y-intercept intuition video, for instance, shows off an interactive applet that students can use to explore. Even taking a non-interactive portion of the site, like Proof: log a + log b = log(ab) nets a quote like:

What I actually want to do is stumble upon the logarithm properties by playing around. And then, later on, I’ll summarize it and then clean it all up.

Of course, optimally, we’d like the students to discover things and clean them up for themselves. That is asking for something different than a video (or at least how we normally think of video, presented linearly). Even then, for the students who worked through an exploration on their own but still didn’t get it, it’s nice to have a video leading through the same logic as a backup.

Now, it’s not to say there aren’t issues, but some of the bobbles and mistakes — at one point fixed by a pop-up box on Youtube, but I believe that video has been remade now — come through more as humanity and charm rather than obscuring factors.

I can hardly claim speed-recording as many lectures as Salman Khan has allows him to fully utilize the video format. There’s all sorts of dynamic aspects of design that would help with presentation, but they are of course time-intensive. It’s like complaining about Wikipedia and recommending Scholarpedia instead for all work. Scholarpedia only has a fraction of Wikipedia’s content and in all likelihood a particular topic X will not be found in Scholarpedia. It’s only usable as a resource in reverse: looking at the index and reading from there.

Still, in many topics, Khan’s not the only game in town. A common sorted index akin to how The Online Books Page sorts books would be welcome. Khan has a consistent quality that makes him usable; random Youtube choices often lead to some extraordinarily dull math videos. Competition between videos needs to open up in a coherent way that students can navigate; this will push in some better lessons in the gaps Khan has.

Example: By all rights, Dr. Taton’s videos should have more views than they do, try say–

The Complex Number i is NOT the square root of negative one!

However, I’m not going to blame Salman for getting famous and being dubbed The Messiah of Math and so forth. Critiques should be measured and backed by solutions: there’s a chance here to build something new.

A system for helping language learners read mathematics

I have twice before now (here and here) pondered over the issue of how to help language learners past the reading of mathematics, particularly in a standardized testing environment.

Our school has implemented a system by Larry Bell termed UNRAAVEL, with posters in every classroom. Here’s the Math UNRAAVEL poster:

Underline the question
Now predict what you think you need to do to solve the problem
Read the word problem
Are the important words circled? (Especially the clue words?)
Apply the steps you chose to solve the problem
Verify your answer (Is it reasonable? Does it make sense?)
Eliminate wrong answers
Let the answer stay or rework the problem

I have tried to implement this in my classroom, with little success. Here are the issues:

1. The acronym is terrible as a memory aid. The first letter, U, suggests an immediate verb (Underline) which stands on its own without the rest of the phrase. The second letter, N, stands for Now, which suggests nothing at all. Recently, attempting to reconstruct the list from memory (and having taught it multiple times) I got only 5 out of 7 right; I’m fairly certain the students would do worse. A successful acronym would have every letter cued to a specific verb so students only have to remember single words.

2. It isn’t sensitive to misunderstanding by single word changes. To go back to Kate Farb-Johnson’s example:

“There were 90 employees in a company last year. This year the number of employees increased by 10 percent. How many employees are in the company this year? A)9, B)81, C)91, D)99, E)100

What if a student dutifully highlights “10”, but misses the percent? Optimally the student shouldn’t circle individual words, but the entire phrase “increased by 10 percent”.

3. It’s too much fiddly work. By “fiddly” I mean “stuff that’s part of the procedure that isn’t useful for every problem but we’re doing it anyway as part of the procedure.” Some math problems consist of only the question, so underlining the question followed by predicting what is needed (already observed) and reading (already done by the act of underlining) is overkill. Even when I hand-selected problems I felt rewarding to a lengthy process, the students rebelled and skipped doing most of the steps.

Going back to the employees problem, let’s see what could happen:

Underline the question: How many employees are in the company this year?

Now predict (etc way too long): Er … we’ll get numbers and we add them or something? With this (and many other math problems) there just isn’t enough information to do a prediction at this stage.

Read the word problem: With the aid being the question is underlined so we have the “punchline”.

Are the important words circled?: Problematic for reasons I already mentioned.

Apply the steps you choose to solve the problem: Taking the acronym literally, the student will have already chosen the steps by this point. Putting that aside, the process doesn’t provide the holistic understanding, the mental image of (or equivalent to) this:

Verify your answer: On this particular question past the point of answer to know what “reasonable” is, unless one happened to catch on rereading a missed important word (that one was previously ignoring because they forgot to highlight it).

Eliminate wrong answers: Wait, we are doing this now? On multiple choice tests even in a reading context doesn’t that come earlier?

Let the answer stay or rework the problem: The acronym’s the kicker here; students remember the “let the answer stay” part but not the “rework the problem” part.

So after a year of fussing and finagling, I have my own process, which I present here. It’s still “fiddly” by my definition, but it’s the best I’ve been able to summon so far:

Underline the question: Still not a bad starting point.
Read the problem: Prediction is one step too many.
Box words you don’t know: Half-baked psychology here, but the idea is to “contain” the tricky words, which may be entirely unnecessary to the problem. This can be coupled with a method of working-definitions-from-context.
Circle key phrases: NOT key words.
Diagram the problem: In my mind, the largest omission of Larry Bell’s original system. Not every problem needs this, but the ones that do are hurt by the reading-emphasis approach.
Eliminate wrong answers: Check earlier rather than later.
Rephrase if needed: From my blog experiments the most powerful technique seemed to be rephrasing.
Solve and verify: As a separate step, students never verify. I want them to have the impulse that every time they solve they also verify.

New acronym: UR BCDE RS

It doesn’t resemble a word (and probably is better for it because students don’t have to remember to misspell UNRAVEL) but it’s cute enough to use as a memory aid.

A reading experiment

In my post “When vocabulary isn’t the issue” I got the impression it was difficult to “step inside the head” of a student who misunderstood that particular problem from a reading perspective, so I thought I’d give an example that has a better chance of simulating the experience.

This is a puzzle called Slitherlink. I gave it to my students and asked them to attempt to work it out simply from the directions, but out of 100 or so students only a handful managed without extra assistance. (They were given that the word “adjacent” means “next to”, so the vocabulary was not a problem.)

I have given this to adults who also needed extra explanation, so don’t feel bad if you’re unsure at first what to do.

Draw a closed loop by connecting dots. Each number in the puzzle specifies how many adjacent sides are included in the loop. A zero means no part of the loop passes next to that number.

Here are four sample puzzles of the type:

This example is less than optimal in that (unlike the last post) I know how to teach reading for understanding here, but still, I’m curious: did you have difficulty, and how did you extricate yourself? How would you teach reading the instructions to this puzzle?