Right now the biggest threat to their IPO's is that people realize that local models are good enough for whatever they're peddling...
...plus the recent price increases by AI companies, made me actually think the opposite: that there might be another additional "run" for memory and/or GPUs.
Therefore, yesterday I decided to order an additional RTX 5060 with 16 GiB VRAM for the ~500$ that I saved during the last months (to be added to the RTX 5070 12 GiB that I bought last year to play games in 4k + my old RTX 3060 12 GiB which I recycled a few months ago after noticing how nice it is to run llama.cpp locally without having to worry about subscription costs).
The original 24 GiB VRAM were actually quite enough for some of the stuff that I do (e.g. transcribe text of image scans of old magazines, coding with Aider, etc - I usually use Q5_K_M quantizations of Qwen & Gemma by Bartowski as lower ones delivered sometimes weird results and/or looped forever in "thinking"-mode), but I guess that with 40 GiB I should be bullet-proof for my pessimistic view of our future :o)
I was running in Gentoo "6.18.18" (amd64) and the exploit worked (and all other shells which I PREVIOUSLY opened could then just execute "su -" without password to become "root") -> doing temporarily a "modprobe -r algif_aead" on-the-fly did not fix it as I was still able to swap to "root" from the unprivileged user by executing just "su -".
"6.18.25" fixed it (module "algif_aead" still running).
- Maybe older Kernel versions that don't contain the fix should be blacklisted?
- FYI in Gentoo I had to recompile "sys-fs/zfs-kmod" after the minor kernel upgrade (I initially skipped it, but after rebooting with the new kernel I could not mount my raidz1) -> the same might be needed for other external modules.
Yeah in theory genkernel should handle zfs but since I’m zfs_on_root because I like living dangerously I have a one liner that genkernels and then re-emerges zfs and then rebuilds the initramfs.
My performance when using an RTX 5070 12GiB VRAM, Ryzen 7 9700X 8 cores CPU, 32GiB DDR5 6000MT (2 sticks):
- "qwen2.5:7b": ~128 tokens/second (this model fits 100% in the VRAM).
- "qwen2.5:32b": ~4.6 tokens/second.
- "qwen3:30b-a3b": ~42 tokens/second (this is a MoE model with multiple specialized "brains") (this uses all 12GiB VRAM + 9GiB system RAM, but the GPU usage during tests is only ~25%).
- qwen3.5:35b-a3b: ~17 tokens/second, but it's highly unstable and crashes -> currently not usable for me.
So currently my sweet spot is "qwen3:30b-a3b" - even if the model doesn't completely fit on the GPU it's still fast enough. "qwen3.5" was disappointing so far, but maybe things will change in the future (maybe Ollama needs some special optimizations for the 3.5-series?).
I would therefore deduce that the most important thing is the amount of VRAM and that performance would be similar even when using an older GPU (e.g. an RTX 3060 with as well 12GiB RAM)?
Performance without a GPU, tested by using a Ryzen 9 5950X 16 cores CPU, 128GiB DDR4 3200 MT:
> The main reason for this is lack of competition for DB in Germany
Cannot be - there is no competition in Switzerland, but things run pretty smoothly -> in the case of Germany I'd rather say: "lack of oversight, controls, 'konsequent zu sein'" -> in the case of Germany's DB I think that nobody at all levels gives a *hit about its problems.
I interpreted your post like what "krupan" posted in the separate sub-thread ("This is a much tighter circle than any of us should be comfortable with"), but maybe others interpreted it differently (the words of your post are quite generic...). Cheers :o)
And there's also pulseaudio, which I had to run in some games with PULSE_LATENCY_MSEC=90 %command%, other games you can only run in lutris, other games you can't even minimize it or it will mess up with the screen entirely.
(you're right - I was wondering the same thing 1h ago :o) )
reply