Creating Family Friendly Chat More Difficult Than Imagined (1996)

Furnished content.


Summary: Creating family friendly environments on the internet presents some interesting challenges that highlight the trade-offs in content moderation. One of the founders of Electric Communities, a pioneer in early online communities, gave a detailed overview of the difficulties in trying to build such a virtual world for Disney that included chat functionality. He described being brought in by Disney alongside someone from a kids' software company, Knowledge Adventure, who had built an online community in the mid-90s called KA-Worlds. Disney wanted to build a virtual community space, HercWorld, to go along with the movie Hercules. After reviewing Disney's requirements for an online community, they realized chat would be next to impossible:

Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: "I'm confused. What standard should we use to decide if a message would be a problem for Disney?"The response was one I will never forget: "Disney's standard is quite clear:No kid will be harassed, even if they don't know they are being harassed."..."OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs," we replied.One of their guys piped up: "Couldn't we do some kind of sentence constructor, with a limited vocabulary of safe words?"Before we could give it any serious thought, their own project manager interrupted, "That won't work. We tried it for KA-Worlds.""We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words - the standard parts of grammar and safe nouns like cars, animals, and objects in the world.""We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he'd created the following sentence:

I want to stick my long-necked Giraffe up your fluffy white bunny.

It was standard fare: Collect stuff, ride stuff, shoot at stuff, build stuff Oops, what was that last thing again?"kids can push around Roman columns and blocks to solve puzzles, make custom shapes, and buildings.", one of the designers said.I couldn't resist, "Umm. Doesn't that violate the Disney standard? In this chat-free world, people will push the stones around until they spell Hi! or F-U-C-K or their phone number or whatever. You've just invented Block-Chat^TM. If you can put down objects, you've got chat. We learned this in Habitat and WorldsAway, where people would turn 100 Afro-Heads into a waterbed." We all laughed, but it was that kind of awkward laugh that you know means that we're all probably just wasting our time.

• Is there a way to build a chat that will not be abused by clever kids to reference forbidden content (e.g., swearing, innuendo, harassment, abuse)?
• Can you build a chat that does not require universal moderation and pre-approval of everything that users will say?
• Are there ways in which kids will still be to communicate with others even without an actual chat feature?
• How much of a community do you have with no chat or extremely limited chat?

• Is it possible to create an online family friendly environment that will work?
  • If so how do you prevent abuse?
  • If not, how do you handle the fact that kids will get online whether they are allowed to or not?
• How do you incentivize companies to create spaces that actually remain as child-friendly as possible?
• If the kids will always find a way to get around limitations, does it make sense to hold the companies themselves responsible?
• Should family friendly environments require full-time monitoring, or pre-vetting of any usage?

"The Disney Standard" (now a legend amongst our employees) still held. No harassment, detectable or not, and no heavy moderation overhead.Brian had an idea though: Fully pre-constructed sentences - dozens of them, easy to access. Specialize them for the activities available in the world. Vaz Douglas, our project manager working with Zoog, liked to call this feature "Chatless Chat." So, we built and launched it for them. Disney was still very tentative about the genre, so they only ran it for about six months; I doubt it was ever very popular.

Sure enough, chatters figured out a few simple protocols to pass their secret code, several variants are of this general form:User A:"Please be my friend."
User A:"Come to my house?"
User B:"Okay."
A:[Move the picture frames on your wall, or move your furniture on the floor to make the number 4.]
A:"Okay"
B:[Writes down 4 on a piece of paper and says] "Okay."
A:[Move objects to make the next letter/number in the code] "Okay"
B:[Writes] "Okay"
A:[Remove objects to represent a "space" in the code] "Okay"
[Repeat steps as needed, until]
A:"Okay"
B:[Enters secret code into Toontown software.]
B:"There, that worked. Hi! I'm Jim 15/M/CA, what's your A/S/L?"

For example, let's say you have a secret code (1hh 5rj) which you would like to give to a toon named Bob.First, you should make clear that you want to become their SF.
You: Please be my friend!
You: (random SF chat)
You: I can't understand you
You: Let's work on that
Bob: Yes
Now, start the secret.
You: (Jump 1 time and say OK. Jump 1 time because that is the first thing in your code. Say OK to confirm that was part of your secret.)
Bob: OK (Wait for this, as this means he has written down or otherwise recorded the 1)
You: Hello! OK (Say hello because the first letter of hello is h, which is the second part of your secret.)
Bob: OK (again, wait for confirmation)
Repeat above step, as you have the same letter for the third part of your secret.
Bob: OK (by now you should know to wait for this)
You: (Jump 5 times and say OK. Jump 5 times as this is the 4th part of your secret)
Bob: OK
You: Run! OK (The 5th part of your secret is r, and "Run!" starts with r)
Bob: OK
You: Jump! OK (Say this because j is the last part of your secret.)
Bob: OK
At this point, you have successfully transmitted the code to Bob.
Most likely, Bob will understand, and within seconds, you will be Secret Friends!

0 comments, click here to add the first

Las Vegas Police Are Running Lots Of Low Quality Images Through Their Facial Recognition System

Furnished content.


Even when facial recognition software works well, it still performs pretty poorly. When algorithms aren't generating false positives, they're acting on the biases programmed into them, making it far more likely for minorities to be misidentified by the software.The better the image quality, the better the search results. The use of a low-quality image pulled from a store security camera resulted in the arrest of the wrong person in Detroit, Michigan. The use of another image with the same software -- one that didn't show the distinctive arm tattoos of the non-perp hauled in by Detroit police -- resulted in another bogus arrest by the same department.In both cases, the department swore the facial recognition software was only part of the equation. The software used by Michigan law enforcement warns investigators search results should not be used as sole probable cause for someone's arrest, but the additional steps taken by investigators (which were minimal) still didn't prevent the arrests from happening.That's the same claim made by Las Vegas law enforcement: facial recognition search results are merely leads, rather than probable cause. As is the case everywhere law enforcement uses this tech, low-quality input images are common. Investigating crimes means utilizing security camera footage, which utilizes cameras far less powerful than the multi-megapixel cameras found on everyone's phones. The Las Vegas Metro Police Department relied on low-quality images for many of its facial recognition searches, documents obtained by Motherboard show.

In 2019, the LVMPD conducted 924 facial recognition searches using the system it purchased from Vigilant Solutions, according to data obtained by Motherboard through a public records request. Vigilant Solutions—which also leases its massive license plate reader database to federal agencies—was bought last year by Motorola Solutions for $445 million.Of those searches, 471 were done using images the department deemed “suitable,” and they resulted in matches with at least one “likely positive candidate” 67% of the time. But 451 searches, nearly half, were run on “non-suitable” probe images. Those searches returned likely positive matches—which could mean anywhere from one to 20 or more mugshots, all with varying confidence scores assigned by the system—only 18% of the time.

Clark Patrick, the Las Vegas attorney representing [Alexander] Buzz, told Motherboard that neither the LVMPD nor the Clark County District Attorney’s office ever informed him that investigators identified Buzz as a suspect using, at least in part, facial recognition technology. The Clark County District Attorney’s office did not respond to an interview request or written questions.

0 comments, click here to add the first