Read through it all, it still comes down to "trust us". Apple can sign and authorise an update at any time that will backdoor it, and the government is the stroke of a pen away from forcing them to, all completely silently.
I get that there's benefit to what they are doing. But the problem of selling a message of trust is you absolutely have to be 100% truthful about it, and them failing to be transparent that people's data is still subject to access like this poisons the larger message they are selling.
While true, the gap between "we send your data to our datacenters but we don't look at it" to "we look at it a little bit without telling you" is much smaller than "we leave your data on your device alone" to "we upload data from your device", both on a technical and policy level.
Even if the org has been trustworthy to this point, I think this step makes it more likely (maybe still unlikely, but more likely) that in the future they do look at your data, as less things have to change for that to happen.
That's true, but also it should be possible to make an OS that people can trust without trusting you, and as users we should encourage movement in that direction.
I understand the sentiment, but it's impractical to live in a trust-less society. If you've ever had dental work done, you've put an awful lot of faith in a stranger pushing a drill into your head. Ditto for riding buses and bus drivers, etc etc.
Trust can be abused, certainly, but it also allows collaboration and specialisation, and without those I doubt we'd have gotten very far.
I'm happy to trust many kinds of people, dentists included, just not the kind people who find themselves at the helm of companies like Apple and Google.
Better to trust many people narrowly (e.g. I don't trust the bus driver to drill my cavities) than to trust a small handful of people broadly (e.g. like Apple expects of their users).
Any kind of practical OS would contain code of unpractical amounts to manually review and audit.
That's not to mention the argument that any software of a LOC count of higher than some number is impossible to audit because of complex state handling. Rice's Theorem applies to your brain too, probably, to some extent. Idk about purely functional Haskell though.
Because there are so freaking many of us, and some of us trust each other. If we were better at coordinating about which parts of the code we trust and to what degree, we could determine which parts of it are untrustworthy and patch the problem out of it.
The GrapheneOS people are doing this, for example. It's not crazy to consider your device vendor as part of your threat model, because like it or not, they are a threat.
That’s a good point. It would be interesting if there was a “git blame” style command, but that showed a trust score for every line/block based on who has touched it.
I'm working on a system for data annotation that might support apps of that nature.
I'm focusing on simpler datasets for now, I want people to be able to annotate a paper restaurant menu with notes about allergens in a way that other people with those allergens can summon those annotations and steer clear--all without participation from the restaurant. Like an augmented reality layer for text.
I hope it grows up into something that would let you ask:
Good luck. It’s much easier to talk about it. The last open OS I have seen reach a semi-mainstream level of adoption was started in the early 90’s, more than 30 years ago, by some Linus guy.
And (basically) nobody running linux is individually verifying the source code of every little piece of software that goes into it (maybe Linus is), so you're still trusting someone.
How does one understand what is going on with out a clear set of documentation?
Either one is so smart they can review everyline of everything they are running. My contention is that it is hard enough to find authoritative documentation much less developing an understanding of the code needed or running.
The choice is either many little trust relationships or a giant leap of faith. I feel better served by a giant leap of faith and access to all the technology I can use.
I prefer no documentation and consistent behavior to no documentation and a bunch of internet howtos on what might work.
That's true, but if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere. The trouble with remotely pushing data somewhere is that you can't tell if anything has changed even if you wanted to. (Attestation only works if it's not compromised, and for obvious reasons, there's no way to know that an attestation mechanism is compromised.)
That said I really don't disagree with this point at all in terms of it being a valid problem. It's not a fixable problem either (it comes down to, again, building trustworthy computers) but it could be biased way towards being solved whereas today it is still "trust me bro". I don't think Apple will be the company to make progress towards this, though.
> if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere
If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.
> I don't think Apple will be the company to make progress towards this, though.
> If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.
Y'know though, when you put it that way, it sounds inherent that security vulnerabilities will pop up, which is kinda true, at least for the foreseeable future, but to be pedantic, the security vulnerabilities are already there, it's discovering them that's the problem. If we could make secure computers... (time to formally prove everything from the ground up I guess.)
But, that said, I wasn't overlooking this, I'm just looping "getting pwned" into "active involvement". If you have some sufficiently isolated machines, they're probably fine indefinitely. The practicality of this is limited outside of thought experiments. However it's definitely worth noting that unlike a compromised remote, it is at least technically feasible to work on the problem of making local compromise more evident, whereas a remote compromise is truly impossible to reliably be able to detect from the outside.
Your argument is no different than what Apple could do to your iPhone. The fact that it happens on the server changes nothing. Apple could push a button and have your iPhone upload whatever they want to their servers. In other words, based on your argument, you shouldn't trust anything, including locally run AI. You're probably right, but it isn't practical.
Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well:
> Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good.
> In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
I think he has a nice pragmatic view on things. I’m EU enterprise we basically view things like picking cloud providers as a question of who we want to spy on us. Typically it comes down to AWS or Azure if you’re pocking a “everything included” service. That being said, I’m not really sure I’m on board with this part:
> As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
Isn’t that sort of where the pragmatism ends? All the parts aren’t going to be secure… Unless I misunderstood his intention, I think the conclusion should be more along the lines of approaching the cloud without trust.
I get that there's benefit to what they are doing. But the problem of selling a message of trust is you absolutely have to be 100% truthful about it, and them failing to be transparent that people's data is still subject to access like this poisons the larger message they are selling.