EFF sues Proctorio on behalf of student falsely DMCA'd

I finished up the coursework for my masters degree at the start of the pandemic. My university was quite flexible for how instructors would examine us, given how sudden everything had to change.

One of my courses, which only had about 8 students and two instructors, decided to do an oral examination, which ended up being basically a very in-depth, one on one conversation about the course material and based on the expectations set in the syllabus (so, no surprises).

While obviously not practical for large rosters, this was by far the best exam format that I have ever done in my many, many years of schooling. I'm sure not everyone would prefer it, but the students unanimously agreed to try it (wouldn't have done it that way otherwise), and it was just so great. It was not at all like an oral thesis defense, which was what I was a little worried about.

In my alma mater for most courses we had to actually discuss what we wrote(and maybe some other topics) with the lector as part of the exam. Some were open book, some were closed. Regardless of that however when you get to talk to the professor it's very hard to fake knowledge and especially logical thinking. It sure takes more time(we usually had one on one for about 20-30 minutes per student), but you can size a person way better than just reviewing what he wrote/selected. And there is no obstacle(other than time constraints, which can be worked around) to continue doing this even during the pandemic. I think we are overestimating technology such as AI and its applications. Some things are just plain better when done by a person. I had a very positive experience with one professor who chose to round up my grade instead of down, because my errors were technical(carelessness mostly) not logical, so he then asked me questions from subsequent chapters to see my reasoning. I really didn't care that much about the grade, but that discussion and the overall experience of failures that could lead to positive change through introspective analysis is something that you can't get from an AI or any pipeline processing.

Thank you! I'm sure it takes more work to make an open book exam, but it's definitely to the benefit of your students.

If you can't see them, doesn't that open up the possibility of students taking the exam together?

When I was involved in the teaching of a course that allowed this, one student basically reduced all the course notes onto a single two-sided page, except they also had flaps on it to fit more in. The exam invigilator took it from them deeming it (correctly imo) to not be in the spirit of the allowance. We decided we'd mark their exam, and then determine what we should do about this cheating-cheat-sheet. Given they got something like 5% in the exam anyway, we decided to let it slide.

My own experience of making up those cheat sheets is the same as yours, the effort put in to create them is an excellent teaching tool in itself, as you're studying and distilling all the stuff you don't know well, and by the time you're done, you hardly have to refer to it in the exam anyway.

For physics, they'd just give us a page with all the formulas but with no context aside from the letters in them, so if you knew roughly what it should look like, you could find it and you're not at so much risk of making a basic error under pressure.

I still have a couple of the cheat sheets my study group compiled -- they're so fun to look back on. One for Computer Organization is almost unintelligible to me now; we were so proud of the density of that sucker.

This was common in my engineering school too. I enjoyed creating the cheat-sheets more than any other form of studying, because what better way of learning three chapters of thermodynamics than having to distill them into 10 lines of text on a notecard.

I always permitted/encouraged this when I taught (I'm retired now). Every once in a while, we'd find some enterprising student selling pre-made cheat sheets, which were of course of little value to the purchaser.

At one point in college, I printed "Don't Panic" in large letters on one of these one page cheat sheets.

This raised eyebrows of the TA's proctoring the exam, but like, if you don't know the stuff, a page isn't going to help you at that point.

I recall at some point that I started writing that type of cheat sheet, even if I didn't get to refer to it in the exam, for that reason.

Just as a counter example. I have 2 "advanced" degrees - in engineering and in management (btw I did NOT do them because I was smart - but rather because I was curious and had a lot of time on my hand. I definitely struggled in both of them - though really really enjoyed them both). I am not what you call a model test-taker. I got barely passing marks in all my exams and yet my "assignments" I was consistently a high scorer in (sadly assignments at the time only accounted for atmost 20% of your grade with 80% being what you can cram-and-dump in an exam).

I still cannot "remember" how to perform a discrete-wavelet-transform from memory (my honor's thesis) but i found myself digging into it a couple of months ago (just fiddling on pet projects). An hour on googling got me on the track. Point is there is so much even those with advanced degrees have to know and having to retain it all in memory all the time is both infeasible and wasteful. Yet expecting kids/young adults to do this is truly hypocritical and disingenuous.

I'm a physics grad student, and this is completely wrong. The purpose of graduate classes (at least in physics) is to teach you how to navigate a the standard reference texts. For example, the main thing I learned in my grad E&M class was where to find information in Jackson (aka "Classical Electrodynamics"). Classical mechanics was a mix of Goldsmith and Landau and Lifshitz. Quantum mechanics used Sakauri. And statistical mechanics used Pathria. Also, referring to these classic texts by author is very common, much moreso than by title.

My experience with people who are PhD is that they know things you expect to memorize just because they use it a lot. They learn formulas or whatever not because they spend 5 hours memorizing it, but because they needed it once, so they looked it up. The second time they needed it, they weren't 100% sure, so they looked it up again just in case. Every time they search for it, they need it less and less and at the end they know it by memory. That's how you learn most of the stuff these days and that should be for everything. Natural learning. Why memorizing things you dont need to know?If you use it often enough, you'll end up memorizing it. Same for concepts or any type of knowledge.

1. PhDs don’t “do exams”. You can’t cheat your way to publishing original research (exceptions of course) and have external expert researchers review it and accepted on a reputable journal. Given that that’s a requisite for obtaining your degree, there’s no point on a final exam. Most defenses nowadays are partly ceremonial (exceptions of course). Both the material and your ability to do research has been checked months before by your advisors and graduation committee.

2. Master’s degree courses follow all kinds of schemas for examination. From oral exams, to in person no extra material allowed, to open book, and take home. Being on both sides (taking the exams and creating them) I can say that it really doesn’t matter if you allow people to take the exam home and collaborate among each other. These types of exams are designed to really test a deep understanding and ability of the material. There have been exam questions where an entire class of >20 students are not able to solve it. People that are really good are able to have a shot at it and maybe make some progress, and that differentiates the good from the exceptional.

3. Nowadays undergraduate degrees are a commodity so it feels they need to make sure only the good students get one. But in the end it doesn’t really matter, since most employers (exceptions of course) will want to see how much value the candidate add, which does not correlate with having a degree or good grades

When I have discussed this, most people I know argued the opposite (for Mathematics). The undergrad courses are supposed to lay the foundation. Stuff you need so often, and stuff you need to recognize when useful, you should know it by heart.

At graduate level, there is too much stuff. You know the outline, you know where to find stuff. But you don't need to know everything exactly. If you forgot one passage from a definition, that should not cause a failure. Because in 'real Mathematics' you get to look at references.

A nice trick for open book is to make it time-infeasible to just look up everything. But allow students access to the materials incase they have a brainfart.

It depends on the degree. If you're becoming a medical doctor you'll have to do a ton of memorization, but in other fields doctorates are the least memorization-intensive part of your education, because the focus is on generation and synthesis, on the assumption that you've already learned the rote knowledge. You don't need to memorize anything to write a dissertation but that doesn't make it easy (and you'll end up memorizing everything anyway).

This is why PhD's typically have comprehensive exams. It varies (a lot!) by university and even department, but panel led oral exams are pretty common. At some places they are even open to the public (no pressure). These are distinct from a thesis defence, which happens at the end of your degree; they are likely to happen about a year in and are meant to ensure you have a solid background.

Overall I think an oral exam run by a skilled examiner is the best of all worlds, but it isn't practical for a section of 500 undergraduate students. It's quite doable for a seminar or manageable up to say 10-15 students, becomes difficult after that.

Often the tests at this level are difficult enough that the need for memorization is implicit.

Open book is also fine in this case because without a base of memorized understanding an open book isn't that helpful anyway.

> Except for some contrived or rare circumstances someone will now point out.

I don't know about you, but I tend to want my pilots, surgeons, and lawyers to tend towards self-competent, and I don't think those circumstances are contrived or rare. It's not to say that they can't ask for help before or during flight/operation/trial -- and in fact asking for help and crew-resource management is important for those fields -- but not all fields are like software engineering or aerospace where there may be dozens or hundreds of people that can help you in a timely fashion if you have completely no idea what you're doing. I'd prefer pilots, surgeons, and lawyers that didn't cheat their way into their jobs.

That only works if the answer can be looked up.

Real life isn't always kind enough to only give problems that someone has already solved and published somewhere you can find.

Honestly, any tests I took that were open book were long enough that if you were constantly looking things up you'd never finish. Same would occur if you were talking to someone and they were having to relay the info.

Part of the benefit of open book tests is that it brings up the "floor" of resource access, limiting the potential differential between honest and cheating students. Obviously, this could be negatively compensated for with other factors, like having more questions or stingy grading.

> I think people here are underestimating the volume of cheating in school.

Not defending Protorio here, but I didn't really understand how rampant cheating was, either, until I became friends with someone who works at a University.

Growing up, my friends and I would never dream of cheating and would fear the consequences of getting caught, so the entire concept feels foreign. We also grew up in the era of in-person classes, before cellular phones were common.

Some of the cheating stories I hear from my friends in University are mind-blowing. Everything from (college) students thinking the proctor won't notice them using their cellular phone during tests, to people trying to hire test-takers who show up with fake IDs. They had to start giving proctors photos of each student because checking IDs wasn't enough any more.

The situation is much more complicated than it sounds. As soon as one person gets away with cheating, it becomes a cheating arms race as even the good students feel the need to cheat to keep up. If an entire institution becomes notorious for rampant cheating, the value of every graduate's degree goes down.

Obviously the Proctorio solution is terrible, which is why I expect we'll see Universities push to return to in-person classes sooner rather than later while keeping remote as a 2nd class option.

While I was at university for computer science, I often found many "senior" students (both graduate and undergraduate) who were truly astonishingly unversed in the most basic things (I think my "favorite" was the time I did a group project involving a student who had had an internship working in the kernel at Sun the prior summer and he turned in his portion of the project (in C) with code like "[...] char* foo = ""; foo += "bar"; [...]")...

...and yet I never thought to wonder how they managed to keep passing exams.

Maybe we should stop endlessly pushing and insisting on degrees...

Maybe skills tests in an interview process is not such a bad thing.

As an employer, I don't care what piece of paper you have from what school. I only care about how well you can do the job I need to be done.

Caltech has a strong culture of academic honesty and take-home exams.

Maybe some institutions are simply selecting their students wrong?

There's software to detect copy-pasted code or plagiarized (it analyses the syntax tree so renaming the variables and changing indentation won't fool it!).

The best however, is a 1:1 code review with a TA or Prof. Randomly jumping to files, reading the code and asking questions about it. Extremely hard to copy a codebase and learn it well enough you can explain it but really easy to do if you wrote it yourself.

Sure projects are generally better than exams. But cheating on projects and papers is also rampant. Students just pay someone else to do the work.

This is the answer.

No one wants to hear this answer because it means actual work for the teaching staff (compared to, say, automated scoring of multiple-choice tests), but it is, in fact, the answer.

You sound like a good teacher. My experience at university was that the good teachers were a minority. Most would do their research and the minimum for teaching.

> The distribution of grades the last exam was normal with a mean of 80%, pretty much in line with what I get during a typical semester.

The distribution is one thing, but grade correlation with other statistics might be more revealing. In particular, how do you know that wealthier students aren't hiring people to take the exam for them?

Their employer will pay the consequence later.

The university will pay repetitional consequences for claiming that the student learned particular things which they clearly didn't.

While I despise software like this and how high-stakes and petty it can be, keep in mind that it's in the institution's interests to not only provide a quality education, but to defend the value of their brand.

You might have worked very hard for a diploma from University X, but if it was found that University X was handing out diplomas like candy to other students that were not as academically rigorous, it weakens the value of your diploma.

When your institution's reputation is at risk, you want to prevent it. Honest students don't get a lesser grade; that's not the problem. If the cheaters want to cheat their way through life, that's their problem. My problem is to make sure the rest of my students end up with a well-respected degree. I.e. the problem is being solved for the honest students, the ones I want to spend my time on.

Realistically, if all existing diplomas were invalidated and the entire system of diploma-granting were to be stripped away from educational institutions, an equivalent would quickly be re-established by the free market - and it would probably resemble Proctorio.

> So my question is: who are we trying to solve the problem for? What is the expected outcome, knowing that there will always be people who "cheat" their way through life too?

For one thing, all the harm that incompetent professionals can cause - and in a moddern society, the scope of that is not inconsiderable - in fact, Proctorio is probably an example.

Let's not lose focus on the real problem here, which is Proctorio being an aggressive vendor of garbage.

  Higher level education is largely voluntary, and it's up to
  the person taking it up to decide what they gain from it. 
  If they are only in it for a diploma, they'd get there one 
  way or another.

I had experience having an exam with the proctorio. And I see many ways how it can be cheated, at least technically. They also say they monitor your head and eyes movements, which is supposed to show something unnatural if you're cheating. Though I'm not sure it's really possible, and maybe just a security theater.

My university had an honor system, proctors were explicitly not allowed in the room during the test except to make announcements. Worked well enough as far as I know. Cheating was dealt with harshly when reported.

In a room, I can make sure that there is no other person there, that there is no big monitor with solutions, notes, whatever. I can also not see whether the students has another browser open with one of the cheat-websites where one of his faster colleagues has already uploaded the solution. I'm luckily in a field where we can do open-book exams without too much problems, which makes the "hidden notes" a non-issue, but one wants to be sure that it's actually the student who solves the problem. That is much easier in a class room.

Higher education is already quite expensive. It would be great to have highly qualified professors spend more time directly interacting with students, but who will pay for those extra professors?

I guess this is a consequence of the Covid pandemic, where such a thing is not allowed at the moment.

If the email mentioned particular students by name then that in and of itself is a violation (at least in California). We're absolutely not supposed to share any student performance information with anyone other than the student, without permission.

If you shoot yourself in one foot you get a nasty limp. Two is much better, for balance.

No worries dang, mistakes happen :P.

> But isn't "cheating" the norm in real-world jobs too? It's rare that I or any of the developers I work with complete a task without looking something up or asking a question of someone more knowledgeable.

One part of one my math exams in a previous university was (I feel like) modeled around this idea, where you were allowed to use a non-programmable (graphical) calculator and to bring basically any written material in to help yourself solve that part of the exam. Of course there were some restrictions: none of the solutions to the various homework assignments etc. That part alone for the ~2 hours of the whole exam would've been nice if there hadn't been the other half: no calculator, no helping materials apart from those maybe provided on the exam sheet. This part (of course) was the one containing the questions about specific definitions, one or more things to write a proof on and calculating things like double and/or triple integrals, deriving complicated expressions.

In other courses (Databases 1 and 2, Web Dev) at my 2nd/current uni with some specific professors there was a clause that you could bring with you help in the form of: "DIN A4 sheet paper, hand-writing on one side, non-copied" and the professor or the TA if one was present would pass through the rows during the exam to check the student ID, have the student sign a presence sheet and whether the help sheet was compliant (also if maybe there were some answers to the exam), signed that help sheet and would collect that together with the exam upon completion/timeout.

Although our course was only ~80-90 people at the start of my first and ~30 at the start of my current uni, so we had to be thrown together with some other disciplines/outlines ("Studiengänge" in Germany) for exams of the same courses to have an exam that made sense for everyone.

Please don't muddy the waters like that. I'm sure you understand perfectly well that when a job asks you to do X, they just want to get X done, whereas when a school asks you to do X, they want to test your skills or make you practice.

I don't disagree with you, but our current system that we have in reality does not handle things well if you just allow rampant cheating.

Restructuring is a good idea but you don't want to throw the baby out with the bathwater in the meantime.

I actually agree with you here, our incentive structures are flawed for sure.

The problem is, hiring is expensive, hiring mistakes are expensive, and new grads are a combination of “lack of signals” and “need lots of time to pan out”. The result is companies look for any possible signal - this more or less becomes a degree, and within the degree, the GPA. Thus incentivizing the student to get the degree with a high GPA by any means necessary, and companies will still give you a year to ramp up.

I agree, I am not saying the lawyers should be charged for perjury necessarily, just that they know the seriousness of what they're doing when they do this.

Without reviewing the pasted code it's hard to know if the claim is fraudulent, but I can assure you the DMCA is not the right tool for the job as-is. Using the word "expeditiously" implies there is some immediate threat that must be dealt with to prevent grave damages to copyright holder. In reality maybe some sections of code were pasted for critique, clearly not to steal property from (c)holder.

If only the most severe cases need to be handled "expeditiously" then MAYBE people wouldn't look at DMCA/DRM as a censorship tool. Right now though, its an automated nightmare which has no means for discriminating fair use of a small incomplete/unusable section of code vs someone posting full compiled binary.

Great for them if they can prove the claim was fraudulent, it can only aid in the fight to fix this busted law that has been abused repeatedly to harm competition and critique. We have to fix it now before the cops start using it to shut down streamers by playing copyrighted music at them (oh wait this IS happening, we need to fix this stupid law already...).

Small correction: he’s being sued (by a US company) in Canada, under a new Canadian law.

SLAPP suits are absolutely a thing in the EU as well.

The irony is, 1984 is used as a manual of sorts by some.

Unless, of course, such documentation will be surpressed. Which is pretty Orwellian, in fact, such operations are the protagonist's job.

Not if it gets Fahrenheit 451'd :-)

The software sounds faulty. The black screen is terrible UX; it should just silently set a bookmark and let the student continue.

The overall outcome (and the student's experience and stress) is ultimately a staff issue.

The professor should have been on call, and should have been able to get the student back into the questions, or, failing that, to offer another sitting. I have done both of these, among other remedies, and setting student expectations ahead of time is crucial for avoiding massively stressful problems (stuff like "If something goes wrong, stop and email me. I can add lost your time back and get you into the test again, no matter how long it takes to sort out. You won't lose marks from this." and actually follow through). Software inevitably fails sometimes, and it is just a tool. Staff need to make their own decisions.

I think you're extrapolating way too far, and hence your arguments aren't hitting the mark. The total amount of invigilation in the entirety of each course I'm involved in is 3-5 hours. This has been the case for many decades. This year, for some students, it is aided by software, due to the fact that they're unavoidably thousands of kilometres away. It is not a slippery slope, and none of your dystopian ideas seem like they could result from it.

This kind of software does not monitor computer use outside of the test. It monitors the student during the test, like a human invigilator (yes, it's recorded, so not identical). During the test, the student is only working on the test, which is not private or secret. When the test is finished, the software exits.

I don't see how you could have the resources for an infrequently used testing centre, but not enough to pay one person for an hour or two. That is not what this kind of software is useful for.

It's a problem for institution reputation, and for remaining accredited to confer professional degrees controlled by external organisations, like engineering degrees.

Yes, but my subject is maths :-). 1st and 2nd year engineering maths don't really have projects, presentations, or writing as options, as we mostly care about whether they know particular fundamental mathematical techniques and skills. All those options also have the problem of knowing who did the work.

From talking to remote students, I don't think they feel like they're being treated like cheaters. Instead, they seem happy we're making their study possible, and accepting of what they're asked to do. They know it's important that they can demonstrate unequivocally that they have particular skills.

Programs like this usually attempt to detect and forbid being run in a VM.

That's all interview questions.

So this was about a decade ago and only a couple years of my life; I really wouldn't want to misrepresent my experience at the time as being reflective of what it's like now. So I'll just touch on the things that likely are the same now.

The bidding process for defense contracts means a lot of Big Design Up Front, and an inability to change things easily once signed, means that though there are attempts at being agile, they likely involve only the technical delivery side (CI/CD pipelines, automated testing, etc), not the interaction with product and other stakeholders (no fail fast and pivot, real MVP, etc). I know there's been some attempts at moving in this direction, but I honestly can't see it happening, since it's innately at odds with the short term incentives that Congress has (and the Pentagon operates under).

The place I worked at had decent perks, but nothing like Silicon Valley tech companies. We had a cafeteria and a Starbucks in the building, but neither was subsidized. Vending machine snacks, again, at cost. Pay and benefits were reasonably competitive for the area as best I could tell (but didn't look that much; when I was ready to move on I wanted different geography as well).

General culture was fairly laid back; only the month or so before something was due did it feel crunchy, and then only for certain people. A lot of dead weight, but a lot of job security, especially for those who delivered.

The projects could be interesting. I worked on a distributed data visualization system that got me exposed to the CAP theorem back in 2011ish, which helped set the trajectory of my career, though I didn't realize it at the time. I also encountered people who said, half jokingly, that their skills had stagnated to the point they weren't hireable elsewhere. I doubt that as true, but certainly at the time there was work on technical things that didn't translate outside of defense (but many of the skills did even if not the technology).

In terms of would I go back to the defense industry? Probably not if I have a choice. While there are a lot of problems in public sector tech companies too, and many with the same issues as defense contractors, there are more of them, and the constraints placed on them tend not to feel as daunting or arbitrary as some of the ones placed on defense companies. Ultimately the impression I got was that defense companies tend to be very stable and predictable, and I just don't appreciate that as much as I appreciate being able to suggest changes and seek improvements.

Approaching this the other way - stepping into my first public sector company, I got to appreciate being agile, actually working with stakeholders to understand and address their needs directly. I got to make technology choices with the team based on what would address the problem the best (and that we were interested in supporting) instead of being told what they were based on what the contract said. And I got to work on smaller more focused teams. Longer term, I've gotten better compensation I feel like (though haven't directly compared; is there more than just a ~15% bonus offered at defense companies now?), the feeling of a lot more things 'done', and the ability to change jobs (without having to change geographic locations) when I felt like I needed a change.

I'm not sure why it's so hard to say "Yes, company X did a lot of things well (Y, Z, AA) but could have improved in sectors AB, AC, AD." Actually, I do understand why -- you may be risk averse, and the fear of losing your job or rocking the boat precludes you from making a critique even if other people get hurt. Maybe in part because you too have mouths to feed and folks who will get hurt if you do so. So you downvote and make up excuses for it. After all, that's easier than addressing the cognitive dissonance, no?

It's understandable, but I still can't agree that it leaves you without some amount of ethical culpability. Maybe significantly less than an executive. But still, some. It's more understandable for roles that don't have as strong a position in the labor market as engineers, but I find it a little bit less so for myself, as someone who works in engineering.

I think you (and anyone else downvoting) should read Eichmann in Jerusalem [1]. It's about this exact ethical quandary. I would hope it would change your opinion on these things, but if it doesn't, agree to disagree. And certainly don't expect any sympathy from me or the rest of society.

[1] https://en.wikipedia.org/wiki/Eichmann_in_Jerusalem

Then why ask? All job interviews are weird mind games. My assumption when I'm asked why I left my last employer is that they're looking to second-guess my other answers based on what I say, or checking to see if I'm a disagreeable person who is willing to badmouth somebody generous enough to employ me. This is just taking the latter in the opposite direction.

Bingo. If you discard my resume because I refuse to badmouth a previous employer in an interview then that's just a sign that I'd probably rather not work for you.

And as other people mentioned, especially in an interview where you're being intensely judged and have to give second thought to everything you say, I'm not about to give anything other than a neutral answer that's supposed to appease most people.

Being able to pick your job that way is a privilege, and it certainly isn’t granted to everyone in all fields

>I might not ask about C-specific context, but if you don't know how to write a value to a sequence of bytes, you have no business being a SWE at Google

I know quite a few engineers who passed Google interviews and work there, and they wouldn't know how to do this off the top of their head. I also interviewed with Google a few times, and not once was I asked a question like this.

Also seems weird you would say that they have "no business being an SWE at Google", given that this problem is pretty trivial, and any competent engineer would be able to figure it out after some quick googling. It isn't some difficult algorithmic problem, it is a very specific and small piece of trivia.

https://www.imdb.com/title/tt0140000/

I recon the majority of english speakers will not think of the Greek word for anus. Especially educated English speakers who have had exams proctored before.

Google Trends: "proctor" is dramatically more common than "proctologist". I'd go so far as to say you're objectively wrong.

https://trends.google.com/trends/explore?geo=US&q=proctor,pr...

I think the majority of English speakers are completely unaware of this meaning. “Proctor” on the other hand, is something they would’ve heard of in an exam setting.