I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
like this
reshared this
I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
like this
reshared this
AndresFreundTec
in reply to AndresFreundTec • • •I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
like this
Haelwenn /элвэн/, gnu/m43 and clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛 like this.
reshared this
Ian Coldwater 👻🌿, Haelwenn /элвэн/, gnu/m43, bert hubert 🇺🇦🇪🇺, Peter Upfold, Shae Erisson, Michał "rysiek" Woźniak · 🇺🇦, void friend, clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛, abadidea, aeva, datenwolf and Jasnah Kholin reshared this.
djinn & juice
in reply to AndresFreundTec • • •Glyph
in reply to AndresFreundTec • • •clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛 likes this.
LisPi
in reply to Glyph • • •That is true.
Binary artifacts have no business existing in Free Software (or near-binary considering how auditable pre-generated config scripts end-up being). The way it was compromised in this case is almost certain to have happened before and reminds me of the SourceForge malware debacle (so arguably that's another famous example of it happening before).
I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.
Glyph
in reply to LisPi • • •LisPi
in reply to Glyph • • •In this case the actual malicious vector was the near-binary injected code in the practical binary of the unaudited autotools vomit (always autoreconf) which was then bundled in the actual binary artifact that was the compromised tarballs.
None should have ever been part of the project.
As for the test files, I still think that having a hex dump with comments explaining what flaws particular parts test would be desirable in a lot of cases.
Hector Martin
in reply to LisPi • • •@lispi314 @glyph The actual backdoor code was part of binary test vectors for the LZMA algorithm (and is an actual amd64 binary object file). Autotools goop was just how it was triggered/injected into the build system.
In this case the build system side was rather noisy, but you could make it a lot more subtle. Something like a subtly incorrect glob could be engineered to "accidentally" match a test data file that then gets linked into the build process in some way.
reshared this
alcinnz reshared this.
LisPi
in reply to Hector Martin • • •Right, I suppose I did omit to mention that one.
Though that suggests my initial statement of "no binaries, full stop" is the right idea.
Alexander The 1st
in reply to LisPi • • •It's my understanding that at that point though, you're trusting the developers of the CPU assembly/byte/machine code.
The problem isn't necessarily the binaries - it may have been harder to do so, but they could've hid it in a commonly distributed compiler ala ( cs.cmu.edu/~rdriley/487/papers… ) - it's a matter of trust in the person doing the development.
Part of the problem is that the developer's submissions were pushed with less review than necessary.
LisPi
in reply to Alexander The 1st • • •Indeed, unfortunately until we have Libre FPGAs and the ability to verify their design and configuration, it is necessary to trust the hardware.
In my post here: udongein.xyz/notice/AgMU728f3a…
One of the articles (I'm not sure which again) explicitly links to the Trusting Trust problem & documentation.
It is possible, if tedious, to bootstrap a Forth implementation in a minimal amount of machine code in an hex editor or other basic input method, and then retain the ability to audit both the initial binary bootstrap seed and then bootstrap everything else from source.
Of course in this case the Guix project contributors decided that a small Scheme (which iirc isn't standard-compliant) interpreter would do.
The commits weren't adequately reviewed, yes.
That said, if binary artifacts like tarballs weren't accepted, and pratically binary by its difficulty of auditing autotools vomit was deemed equally unacceptable (better use autoreconf on whatever dev machine is used to build the project as necessary), it would be far more difficult to replicate this incident.
LisPi
2024-03-30 01:12:05
Hector Martin
in reply to LisPi • • •@lispi314 @glyph @AT1ST You should look up Precursor:
bunniestudios.com/blog/?p=5921
TL;DR there is a very legitimate argument that backdooring an FPGA generically is impractical, so libre FPGA configuration (and ideally libre FPGA toolchains) plus inspectable hardware generally suffice for a high level of trust.
However, this will never be practical for high performance general purpose compute, only for very limited applications like this. I don't think anyone has any good ideas on how to solve the trustability problem for things like desktop PC or smartphone class hardware. It might never be possible.
Introducing Precursor « bunnie's blog
www.bunniestudios.comIan Coldwater 👻🌿
in reply to AndresFreundTec • • •Mark Eichin
in reply to AndresFreundTec • • •clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛 likes this.
abadidea
in reply to AndresFreundTec • • •railmeat
in reply to AndresFreundTec • • •AndresFreundTec
in reply to AndresFreundTec • • •One more aspect that I think emphasizes the number of coincidences that had to come together to find this:
I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.
AndresFreundTec
in reply to AndresFreundTec • • •Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.
Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.
Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().
clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛 likes this.
AndresFreundTec
in reply to AndresFreundTec • • •AndresFreundTec
in reply to AndresFreundTec • • •like this
clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛 and Haelwenn /элвэн/ like this.
reshared this
clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛 and Patricia Aas reshared this.
Cyril Brulebois
in reply to AndresFreundTec • • •Anisse
in reply to Cyril Brulebois • • •BD
in reply to AndresFreundTec • • •Matt "msw" Wilson
in reply to AndresFreundTec • • •dgilman
in reply to AndresFreundTec • • •AndresFreundTec
in reply to dgilman • • •Rihards Olups
in reply to AndresFreundTec • • •This is insane. I expect full-fledged articles out soon, but another interesting bit in news.ycombinator.com/item?id=3… :
"the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features""
This is CVE-2024-3094 for easier tracking.
#JiaT75 #CVE20243094
Very annoying - the apparent author of the backdoor was in communication with me... | Hacker News
news.ycombinator.comreshared this
M. The Crystalline Entity and Charlie Stross reshared this.
Rihards Olups
in reply to Rihards Olups • • •@dgilman
From the same thread:
"Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.
> If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released."
Dianora (Diane Bruce)
in reply to Rihards Olups • • •John Regehr
in reply to AndresFreundTec • • •Haelwenn /элвэн/ likes this.
silverwizard
in reply to AndresFreundTec • •Hypolite Petovan likes this.
Billy O'Neal
in reply to AndresFreundTec • • •LisPi
Unknown parent • • •GCC as well.
Ref:
- guix.gnu.org/blog/2019/guix-re…
- guix.gnu.org/blog/2020/guix-fu…
- guix.gnu.org/en/blog/2023/the-…
Guix Reduces Bootstrap Seed by 50% — 2019 — Blog — GNU Guix
guix.gnu.orgdeliverator
Unknown parent • • •@malwaretech saw reddit comment to the effect: don't cause perf issues for database people, they will wreck you.
Nice job Andres!
Graham Perrin
in reply to AndresFreundTec • • •thank you, I see the paragraph that begins:
"To reproduce outside of systemd, …"
I have someone claiming:
"The exploit requires systemd. …"
Can both be true?
Postscript: thanks to @vi for helping me to realise my misunderstanding.
Apologies for the noise.
Vi •
in reply to Graham Perrin • • •@grahamperrin the exploit only works if sshd is patched to depend on libsystemd as that's what pulls in the compromised liblzma.
You don't have to *run* sshd via systemd to be compromised, it can be started manually because the binary is still linked with the malicious library.
@AndresFreundTec
Mina
in reply to Vi • • • •