Roblox's introduction of mandatory face verification to chat is one of the most biggest examples of how people in tech can get so deep in trying to create a solid technical solution, that they completely miss the human problems it creates.
You could create the best possible face verification system that processes everything completely locally, uses CPU security features to make sure the photos stay exactly where they're supposed to, etc etc. You could design the best possible chat age segregation system that makes sure nobody can ever get groomed over chat again. You can get so deep that you forget you're forcing children to take pictures of themselves, and fail to consider the wider effects this will have on the safety of those kids in general.
How's Jimmy supposed to know that taking a picture of himself for roblox.com is okay, but taking a picture for somescamwebsite that he found in a Roblox game is absolutely not okay? This solution creates a much worse problem. Sane parenting would tell kids to never take pictures of themselves or put it on any website, but now we're clearly shifting the role of parenting to tech companies and we are going to see bad consequences of this.
If 11-year-old Jimmy is anything like I was a lifetime ago (in terms of understanding tech), he knows how to ask an LLM to take his picture and make him look like he's 18... and none of it matters anyways.
Ideally Roblox would be able to rely on the platform to tell them whether the device is child-locked or not. It would be up to parents to make sure their kids only have access to devices with appropriate locks turned on. Parents could rely on vendors to make devices where it’s easy to set appropriate locks, and rely on stores not to sell unlocked devices to kids.
But we don’t live in that world.
Also, the are trying to prevent adults from pretending to be kids, which is much harder than preventing kids from accessing adult sites.
> Ideally Roblox would be able to rely on the platform to tell them whether the device is child-locked or not. It would be up to parents to make sure their kids only have access to devices with appropriate locks turned on
Yes, for people who live from selling tech, it is ideal when parents have to buy a separate device for everyone. But for people who do not live from selling tech, they prefer one or two tablets for the whole family, you know it is cheaper.
And being cheap is one of reasons for roblox popularity. Kids could have play it without buying anything (except that tablet).
This feels like a bit of a reach. It's not really clear that adding face scanning as a blocker for chat makes anyone more likely to fall for scams. These hypotethical god tier engineers should just make scam prevention software in addition to face scanning anyway?
You could create the best possible face verification system that processes everything completely locally, uses CPU security features to make sure the photos stay exactly where they're supposed to, etc etc. You could design the best possible chat age segregation system that makes sure nobody can ever get groomed over chat again. You can get so deep that you forget you're forcing children to take pictures of themselves, and fail to consider the wider effects this will have on the safety of those kids in general.
How's Jimmy supposed to know that taking a picture of himself for roblox.com is okay, but taking a picture for somescamwebsite that he found in a Roblox game is absolutely not okay? This solution creates a much worse problem. Sane parenting would tell kids to never take pictures of themselves or put it on any website, but now we're clearly shifting the role of parenting to tech companies and we are going to see bad consequences of this.