How abliterated models can get you pwned (projectdiscovery.io)

Because it's not the abliteration that does this. It's the "untrusted artifact" that does.

Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).

That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.

___

Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.

This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.

Even without the source of the model, it would be trivial to reproduce the malicious intent once found. You have the weights - you can find the prompt to produce the bad behavior. So where is that proof?

I'm much more concerned with both the private data I send and tokens executed on my computer doing god knows what from cloud hosted providers. Especially those that don't even show the reasoning traces. So the model's true intent can be hidden even while it's openly thinking of acting against you.

This is scare mongering that implicitly suggests something much less secure.

Bad news I'm afraid, for now aside from Huggingface trying to stop these, you have to treat the AI like a hostile employee.

- Don't give your AI secrets access, scope it's fs access as a limited user.

- Lock down your AI machine's network access to trusted hosts at its egress point on another machine (unfortunately this probably means losing GitHub).

- Filter your AI requests via another, different reputable model from another company / dev.

- Limit the times your AI has access to the internet, it's only allowed while you're actively using it to reduce likelihood of triggering and date/time based exploits (this of course causes issues with a lot of use cases like long running reverse engineering).

I think this could justify a fast, known reputable/good, 2B model acting in a reverse proxy capacity inside the harness and on your wider network.

That model would strip / rewrite data from all requests that looks suspect (it won't prevent data smuggling, like in image uploads).

It would also block domains BAU for a proxy using an allowlist.

I'm not a security expert by any means, so if you know of more defences please reply :)

Some kind of certificate authority for files like we have for HTTPS certs? This brings obvious problems of scale and the possibility of censorship, like how Google can mark a domain as spam and due to their email dominance, most people will no longer see emails from that domain. 2600 Magazine ran in to this problem.

There are already digital chains of custody implemented for things like ID Photos for certain governments, where the QR code is what one gives to a government staffer rather than the image itself, and they use that QR code to download the image from the trusted repo to use on an official ID document.

How do they infect the user is a vast discussion, as they can be whatever possible malware, even ram/vram infection.

Solutions: 1 Avoid external coding tools/repos. 2 Own your system/kernel. 3 Firewalls got your back. 4 Policy works. 5 Internet connection is the front and backdoor.

How many weights is a minimal abliteration? Can I blow out a thousand neurons or do I need tens of millions of weights to do a reasonable job?

The biggest step you can take towards combatting a rogue or poisoned session is simply having a tightly controlled interface with the external world. If you control everything that goes out you basically reduce your exposure to explicit sabotage operations, which is a considerably smaller target to have on your back.

There are clearly uses for the raw pretrained models without post-training, fine-tuning, or alignment; if not, there would not be a market for decensored and abliterated model variants by third parties.

If reputable model publishers could instead publish their models in verifiable parts: pretrained weights, fine-tuning/alignment adapters, and a combined intended-use model - this would reduce the attractiveness of malicious fine-tune publishers. I'm not holding my breath for that though, quite the opposite. I expect the economic and power incentives will encourage the big training players to push for ever-higher but nebulous safety regulations until everyone else is choked out except for them.

Until training costs come down, and perhaps even after that, all AI should be treated as untrusted rogue agents, constrained by default regardless of source. That's just the sensible thing to do, but the economics will have something to say about that.

Running them on your host hardware with access to your whole machine because its running in your main account is unhinged bats*t crazy insane.

Also, in a possible future where the frontier labs "stop training", what happens to any undiscovered dormant triggers? Is it something they can patch away relatively cheaply and quickly?

Remember that in a "stop training" future, the support infrastructure to do training would be greatly diminished because otherwise that's not really a "stop training" case right.