Content Moderation Case Studies: Stopping Malware In Search Leads To Unsupported Claims Of Bias (2007)
Summary: As detailed in a thorough oral history in Wired back in 2017, it's hard to overstate the importance of Google's Safe Browsing blocklist effort that began as a project in 2005, but really launched in 2007. The effort was in response to a recognition that there were malicious websites out there that were attempting to trick people into visiting in order to install various forms of malware. Google's Safe Browsing list, and its corresponding API (used by pretty much every other major browser, including Safari, Firefox and more) has become a crucial part of stopping people from being lured to dangerous websites that may damage or compromise their computers.Of course, as with any set of filters and blocklists, questions are always raised about the error rate, and whether or not you have too many false positives (or false negatives). And, not surprisingly, when sites are added to the blocklist, many website operators become upset. Part of the problem was that, all too often, the websites had become compromised without the operator knowing about it -- leading them to claim they were falsely being blocked. From the oral history:
One interesting thing that happened was related to how we communicated with web masters who were affected by Safe Browsing alerts. Because very quickly when we started looking into the problem of how users might be exposed to malware on the web, we realized that a lot of it came from websites that were actually benign, but were compromised and started delivering malware via exploits. The site owners or administrators typically did not realize that this was happening.Of course, at the same time, when sites are being put on blocklists when they don't even realize that their sites are compromised, they often assume that there is some sort of bias against them or the content of their websites.Matt Cutts, who for many years was in charge of stopping web spam at Google, wrote a few blog posts in the early years responding to people who accused Google of blocking websites whose content they disagreed with, with Matt explaining how he responds to such complaints. In that linked post, Cutts responds to complaints from a vocal Google critic, who claims that it was blocking the website of a think tank that had expressed opinions that were different than Google's regarding net neutrality.Cutts points out that this is not accurate and that Google had only blocked a specific page on that think tank's website, and the reason was that that page itself was compromised and visiting it would install malware on the visitor's computer:
If you visit pff.org/issues-pubs/, you'll see that it's a web form. It looks like pff.org stored their data in a SQL database but didn't correctly sanitize/escape input from users, which led to a SQL injection attack where regular users got exposed to malicious code. As a result, normal users appear to have loaded urls like hxxp://www.ausbnr .com/ngg.js and hxxp://www.westpacsecuresite .com/b.js < --- Don't go to urls like this unless you are 1) a security researcher or 2) want to infect your machine. Notice that even in this case, Google didn't flag the entire pff.org site, just the one directory on the site that appeared to be dangerous for users. I never like it when people accuse Google of flagging a site as malware just because we don't like it for some reason. The bright side of this incident is that pff.org will find out about a security hole on their site that was hurting their users (it looks like pff.org has disabled the search on the vulnerable page in the last few hours, so it appears that they're responding quickly to this issue). Flagging malware on the web doesn't earn any money for Google, but it's clearly a Good Thing for users and for the web. I'm glad we do it, even if it means that sometimes we have to write a generic malware post to debunk misconceptions.Decisions for Google:
Questions and policy implications to consider:
- How do you determine which sites are included in the Safe Browsing blocklist?
- Should the blocklist cover entire domains, or just specific pages within a domain?
- How should Google inform website operators that their sites are distributing malware?
- How do you determine which sites are deliberately distributing malware, and which are simply compromised?
- How should Google respond to accusations of false positives?
Resolution: Google took a number of steps to try to alleviate the concerns, starting initially with a diagnostic tool for sites that were put on the blocklist.Over time that expanded to Google's Search Console for websites. From Wired's oral history of Safe Browsing, here is Panos Mavrommatis, the Engineering Director of the program describing how and why they built the Search Console:
- If Google is too transparent, will that actually help those with malicious intent get around the blocklists?
- Since Google is not making money from this blocklist, how many resources will it take to keep the list accurate and to handle appeals and questions from blocked sites?
- Should a single company be making these decisions?
In our first interactions with web masters they would often be surprised. So we started building tools dedicated to web masters, now called Search Console. The basic feature was that we would try to guide the web master to the reason that their website was infected, or if we didn't know the exact reason we would at least tell them which pages on their server were distributing malware, or we would show them a snippet of code that was injected into their site.Along the way, they have faced mistakes and errors, including a technical error in 2009 that, for a time, labeled every single Google search result an error.Google also put together an appeals process for those who felt their site did not belong on the blocklist, and enlisted a third party, the non-profit StopBadware.org which was formed out of Harvard's Berkman Klein Center in 2006, to help with reviewing appeals when a site gets listed in Safe Browsing.This partnership represents an early version of having an independent third party help review content moderation choices on a platform.
Read more here
posted at: 12:00am on 05-Sep-2020
path: /Policy | permalink | edit (requires password)
Students, Parents Figure Out School Is Using AI To Grade Exams And Immediately Game The System
With the COVID-19 pandemic still working its way through the United States and many other countries, we've finally arrived at the episode of this apocalypse drama where school has resumed (or will be shortly) for our kids. It seems that one useful outcome of the pandemic, if we're looking for some kind of silver lining, is that it has put on full display just how inept we are as a nation in so many ways. Federal responses, personal behavior, our medical system, and our financial system are all basically getting failing grades at every turn.Speaking of grades, schools that are now trying to suddenly pull off remote learning for kids are relying on technology to do so. Unfortunately, here too we see that we simply weren't prepared for this kind of thing. Aside from all of the other complaints you've probably heard or uttered yourselves -- internet connections are too shitty for all of this, teachers aren't properly trained for distance learning, the technology being handed out by schools mostly sucks -- we can also add to that unfortunate attempts by school districts to get AI to grade exams.This story begins with a parent seeing her 12 year old son, Lazare Simmons, fail a virtual exam. Taking an active role, Dana Simmons went on to watch her son complete more tests and assignments using the remote learning platform the school had set students up on, Edgenuity. While watching, it became quickly apparent how the platform was performing its scoring function.
She looked at the correct answers, which Edgenuity revealed at the end. She surmised that Edgenuity’s AI was scanning for specific keywords that it expected to see in students’ answers. And she decided to game it. Now, for every short-answer question, Lazare writes two long sentences followed by a disjointed list of keywords — anything that seems relevant to the question. “The questions are things like... ‘What was the advantage of Constantinople’s location for the power of the Byzantine empire,’” Simmons says. “So you go through, okay, what are the possible keywords that are associated with this? Wealth, caravan, ship, India, China, Middle East, he just threw all of those words in.”“I wanted to game it because I felt like it was an easy way to get a good grade,” Lazare told The Verge. He usually digs the keywords out of the article or video the question is based on.And Lazare appears to have been right, as he now gets perfect scores on all of his tests. This is obviously both lazy teaching and lazy technology. Relying on software to grade tests that are essentially short-form essay tests, as opposed to multiple-choice Scantron style tests, make zero sense. Human grading is needed.But the technology is quite lazy as well. How can a platform that is grading exams of this nature not build in a check against proper grammar, for instance? The fact that a student can simply toss in a bunch of disjointed words at the end of an answer, like some kind of keyword metadata, and get away with it is crazy. Especially when Edgenuity informs everyone that it's supposed to work this way.
According to the website, answers to certain questions receive 0% if they include no keywords, and 100% if they include at least one. Other questions earn a certain percentage based on the number of keywords included.Whatever that is, it sure as hell isn't good education. And while testing practices in education are generally under scrutiny wholesale at the moment, there is little reason to issue tests at all if everyone involved is going to be this lazy about it.And, to be clear, this is happening all over the place, with students finding more than one way to game the system.
More than 20,000 schools currently use the platform, according to the company’s website, including 20 of the country’s 25 largest school districts, and two students from different high schools to Lazare told me they found a similar way to cheat. They often copy the text of their questions and paste it into the answer field, assuming it’s likely to contain the relevant keywords. One told me they used the trick all throughout last semester and received full credit “pretty much every time.”Another high school student, who used Edgenuity a few years ago, said he would sometimes try submitting batches of words related to the questions “only when I was completely clueless.” The method worked “more often than not.”I think it's fair to say that Edgenuity probably doesn't get a passing grade for its platform, now widely used thanks to COVID-19.
Read more here
posted at: 12:00am on 05-Sep-2020
path: /Policy | permalink | edit (requires password)