With glibc, you can use -ftls-model=initial-exec (or the corresponding variable attribute) to get offset-based TLS. The offset is variable per program (unlike local-exec), but the same for all threads, so it's more efficient. Using too much initial-exec TLS (potential across multiple shared objects) eventually causes dlopen to fail because the TCB cannot be resized. This is not a problem if the shared objects are loaded through dependencies at process start.
If initial-exec TLS does not work due to the dlopen issue, on x86-64 and recent-enough distributions, you can use -mtls-dialect=gnu2 to get a faster variant of __tls_get_addr that requires less register spilling. Unfortunately glibc and GCC originally did not agree on the ABI: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874https://sourceware.org/bugzilla/show_bug.cgi?id=31372 This has been fixed for RHEL 10 (which switched to -mtls-dialect=gnu2 for x86-64 for the whole distribution, thereby exposing the ABI bug during development). As the ABI was fixed on the glibc side in dynamically-linked code, the change is backportable, but it's a bit involved because the first XSAVE-using change upstream was buggy, if I recall correctly. But the backport is definitely something you could request from your distribution.
Note that there was a previous bug in __tls_get_addr (on all architectures that use it), where the fast path was not always used after dlopen: https://sourceware.org/bugzilla/show_bug.cgi?id=19924 This bug introduced way more overhead that just saving registers. I expect that quite a few distributions have backported the fix. This breaks certain interposed mallocs due to a malloc/TLS cyclic dependency, but there is a workaround for that: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0...
The other issue is just that the C++ TLS-with-constructors design isn't that great. You can work around this in the application by using a plain pointer for TLS access, which starts out as NULL and is initialized after a null check. To free the pointer on thread exit, you can use a separate TLS variable or POSIX thread-specific data (pthread_key_create) to register a destructor, and that will only be accessed on initialized and thread exit.
I see a lot of people asking for a real use case. If you follow the reference chain in the first aside, you'll find this blog post of mine https://pvk.ca/Blog/2020/07/07/flatter-wait-free-hazard-poin.... where we use value speculation to keep MOVS out of the critical path in an interrupt-atomic read sequence for hazard pointers.
> This is what Smalltalk did, and the problem is it's very hard to understand what a program does when any part of it can change at any time.
I don't think dissolving this difference necessarily results in Smalltalk-like problems. Any kind of principled dissolution of this boundary must ensure the soundness of the static type system, otherwise they're not really static types, so the dynamic part should not violate type guarantees. It could look something like "Type Systems as Macros":
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there).
Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work).
Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance this will work for LLMs too.
This was a passion project of mine in my last few months at IBM research :).
I am convinced there is a deep connection to understanding why backprop is unreasonably effective, and the result that you can train low precision DNNs; for those note familiar, the technique is to compute the loss wrt to the low precision parameters (eg project to ternary) but apply the gradient to high precision copy of parameters (known as the straight through estimator). This is a biased estimator and there is no theoretical underpinning for why this should work, but in practice it works well.
My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.
There is a very cool paper about structuring query compilers that I've read a couple years ago, and came back to recently when doing some experimentation.
The paper[0], "How to Architect a Query Compiler, Revisited", presents an approach to structuring compilers based on Futamura Projections. In practice this means that you write most of your high-level code as though it was an interpreter, but then mainly rely on constructors and operator overloading (and occasionally special functions) to emit compiled code for your expressions and nodes under the hood.
The paper should be interesting to anybody who is also interested in the posted link.
Once you're past the fundamentals, if find yourself interested in high-performance networking, I recommend looking into userspace networking and NIC device drivers. The Intel 82599ES has a freely available (and readable!) data sheet, DPDK has a great book, fd.io has published absolutely insane benchmarks, ixy [1] has a wonderful paper and repo. It's a great way to go beyond the basics of networking and CPU performance. It's even more approachable today with XDP – you don't need to write device-specific code.
If you're interested in something with a bit more features, check out the Bangle.js[0]. The benefits are you have Bluetooth, GPS, accelerometer, vibrator, and a colour screen. The main downside is that the battery lasts considerably less than 3 years.
just start with the Feynman undergraduate lectures (listed). The easy-mode of that is "Six Easy Pieces" (read it on kindle) which are the easiest 6 lectures.
At some point you'll need math, I recommend https://www.amazon.com/No-bullshit-guide-linear-algebra/dp/0... (I actually started here), and for calculus, "No BS Guide to Math/Physics" by the same author. These books both include a review of high school math (i.e. trig) which i needed. For DiffEq I currently recommend Logan's "A First Course in Differential Equations", this is where I am now and I found this the most gentle after trying several textbooks recommended from r/math. Context: I am an adult with an engineering degree from 20 yrs ago.
Other flags that can be useful not mentioned here:
* -fvisibility=internal (stronger than hidden and must be used with a LOT OF CARE, e.g. never pass a function pointer to a function not marked explicitly with hidden or default visibility in that case as its ABI may change)
* -Bsymbolic / -Bsymbolic-functions / -fno-semantic-interposition (interesting explanations: http://maskray.me/blog/2021-05-09-fno-semantic-interposition)
* -fno-stack-protector (not recommended for libz aha)
* -fno-plt / -fno-ident (very small effect, more for the sake of completeness)
* -fvirtual-function-elimination (surprisingly not done by default, here I can sometime get back a few percent through this on OO heavy codebases)
* -ffunction-sections -fdata-sections -Wl,--gc-sections (largest improvement in my experience, sometimes this literally halved binary size for me when coupled with lto)
* -Wl,--as-needed
* -Wl,--icf=all (I heard horror stories about this but it works fine here)
in some cases, when all used together (as there's some synergy between the various optimizations of these flags) this can have double-digits effects on binary size reduction percentage
The reason I got them was that I wanted small desktop displays to both replace my “now playing” Raspberry Pi display and act as smart speakers, and there didn’t seem to be anything event in the market (there are plenty of cheap Chinese tablets, but I wanted something with a standalone PSU and a proper speaker).
These are not Google Home devices (they run Android Things, but a “corporate” version tailored for Teams and Zoom calling) but they are functionally equivalent and surprisingly capable. After installing Firefox and PlexAmp, they do everything I could possibly want from a kitchen-top display, and I am considering getting another one.
Like the OP, I am fascinated with the fact that Google keeps shooting itself in the foot regarding home devices—-these things are as capable as any Android tablet and far more useful if you install a browser, so I honestly don’t see the point of nerfing them with fancy “home” UIs that do absolutely nothing useful.
If Google stopped messing with its partners and standardized on a a more open, more third-party friendly Android Things release with just a browser, a media player and the Play Store (plus maybe a better “family” calendar UI, which I baked in with an Outlook web view), I’m betting there would be plenty of cheap Chinese clones as well…
And they are more reliable to boot, since you won’t forget to charge them (no, a tablet on a dock isn’t the same thing).
The Illustrated Transformer is fantastic, but I would suggest that those going into it really should read the previous articles in the series to get a foundation to understand it more, plus later articles that go into GPT and BERT, here's the list:
I, random person on internet, highly recommend resmed airsense. Make sure it’s apap model. Their naming schemes are confusing. I’ve used Phillips and resmed and resmed is so much nicer in every way.
The relational model (and generally working at the level of sets/collections, instead of the level of individual values/objects) actually makes it easier to have this kind of incremental computation in a consistent way, I think.
There's a bunch of work being done on making relational systems work this way. Some interesting reading:
That is essentially how Allen Downey approaches statistical education: the analytical solutions came first because we lacked the computational power. Now that we have cheap computation, we should exploit that to develop better intuition. His Bayesian book[0] is available as Jupyter notebooks.
Wasn't "my" usage per se since I'm not the original comment author, but yes, I too find this term confusing in this context. A stack trace to me usually means a textual dump of a call stack printed from some point of the program execution (usually an error site and/or a breakpoint), not compiler output that contains no information about the compiler's runtime stack.
No extra tooling, no symlinks, files are tracked on a version control system, you can use different branches for different computers, you can replicate you configuration easily on new installation.
I'll plug my site, which visualizes solutions to the Schrödinger equation in 1d potentials. Try the first two exercises, no math required: https://ridiculousfish.com/wavefiz/#exercises
Install it (burn it to a USB drive using Rufus, for example, or to a DVD). Then use a KMS key and one of the many public KMS servers that let you activate Windows without having a key. Cmd as admin:
“A Primer on Memory Consistency and Cache Coherence” is part of the "Synthesis Lectures on Computer Architecture" which are 50-100 page booklets on topics related to HW components. All the booklet PDF's are available online [1].
edit: only those PDF's with a checkmark are available as PDF to download, the rest can be bought. Quite a few actually available for download.
I'm referring to what people want when they say they want "higher-kinded types" in Rust. That is: the ability to have typeclasses with higher-kinded type parameters.
I'm well aware that the formal definition of an HKT is just a type with a higher kind, and that's irrelevant to this discussion.
No, it's not relevant. When people say they want HKT, what they mean is that they want to create typeclasses that abstract over types of a higher kind and only those types, with a type system that can make those guarantees.
Saying C++ has HKT because it doesn't have typeclasses is like saying Python has all of Haskell's type system features because it doesn't have static types. There's a sort of vacuous sense in which it's true, but it's not a particularly meaningful or interesting thing to say.
If initial-exec TLS does not work due to the dlopen issue, on x86-64 and recent-enough distributions, you can use -mtls-dialect=gnu2 to get a faster variant of __tls_get_addr that requires less register spilling. Unfortunately glibc and GCC originally did not agree on the ABI: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113874 https://sourceware.org/bugzilla/show_bug.cgi?id=31372 This has been fixed for RHEL 10 (which switched to -mtls-dialect=gnu2 for x86-64 for the whole distribution, thereby exposing the ABI bug during development). As the ABI was fixed on the glibc side in dynamically-linked code, the change is backportable, but it's a bit involved because the first XSAVE-using change upstream was buggy, if I recall correctly. But the backport is definitely something you could request from your distribution.
Note that there was a previous bug in __tls_get_addr (on all architectures that use it), where the fast path was not always used after dlopen: https://sourceware.org/bugzilla/show_bug.cgi?id=19924 This bug introduced way more overhead that just saving registers. I expect that quite a few distributions have backported the fix. This breaks certain interposed mallocs due to a malloc/TLS cyclic dependency, but there is a workaround for that: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=018f0...
The other issue is just that the C++ TLS-with-constructors design isn't that great. You can work around this in the application by using a plain pointer for TLS access, which starts out as NULL and is initialized after a null check. To free the pointer on thread exit, you can use a separate TLS variable or POSIX thread-specific data (pthread_key_create) to register a destructor, and that will only be accessed on initialized and thread exit.
This sort of question is probably more suited to libc-help: https://sourceware.org/mailman/listinfo/libc-help/