if we're talking about logs from our own applications that we have written, the program should know because we can write it in a way that it knows.
user-defined config should be verified before it is used. make a ping to port 25 to see if it works before you start using that config for actual operation. if it fails the verification step, that's not an error that needs to be logged.
What about when the mail server endpoint has changed, and for whatever reason, this configuration wasn’t updated? This is a common scenario when dealing with legacy infrastructure in my experience.
the whole point of the essay here is that you should make a distinction between errors that you care about and plan to fix, and errors that you don't care about and don't intend to do anything about. and if you don't intend to do anything about it, it shouldn't be logged as error.
i'm following the author's example that an SMTP connection error is something you want to investigate and fix. if you have a different system with different assumptions where your response to a mailserver being unreachable is to ignore it, obviously that example doesn't apply for you. i'm not saying, and i don't think the author is saying that SMTP errors should always or never be logged as errors.
when the mailserver endpoint has changed, you should do the thing that makes sense in the context of your application. if it's not something that the person responsible for reviewing the logs needs to know about, don't log it. if it is, then log it.
So when the random error on a remote party happens at one time your system ignores it, bu when it happens at another time, it prevents the server from booting? That's a very brittle system.
Would it make sense to consider anything that prevents a process from completing it's intended function an error? It seems like this message would fall into that category and, as you pointed out, could result from a local fault as well.
SMTP clients are designed to try again with exponential backoff. If the final attempt fails and your email gets bounced, now that's an error. Until then, it's just a delay, business as usual.
There are a lot of European “cloud” providers, but there’s not one that offers anything even close to AWS/GCP/Cloudflare. If you need more than compute and S3, you’re pretty much SOL.
Absolutely not. There's a gazillion cloud providers out there with hosted postgres+kafka+redis and the other big open source softwares. Hetzner is just not one of them.
Hetzner, OVH and Upcloud. All of them have object storage, managed Redis,Postgres and K8S.
Most of the time the missing things are homegrown SaaS offerings of big 3 and identity services. You will not find equivalent IAM or BigQuery in indie clouds.
Welcome to the real world. I’m not saying this is a good thing, but this is what’s happened. That’s why everyone is doing AWS. To pretend this is a non-issue and can easily be fixed by investing a bit in development does not reflect reality.
Because they don’t offer every service AWS offers? They offer plenty of hosted databases, queues and what not that it shouldn’t be too hard to move things over. Especially not if you are on Kubernetes. Not if you are all in on lambdas of course but that is a problem in and of itself.
I had created this comment on your other comment on one of my other comments that there are no cloud providers and so I am wishing to talk about both of these at the same comment perhaps
I do not know about the others but OVH (even with all of its flaws) is definitely a cloud provider
I got so damn overwhelmed looking at all the services offered by OVH once and I found some niche services which would most likely be underrated by many but if one wants at scale cheap cold storage, I recommend OVH's cold storage because they cost only 2$ per month/TB storage long term but have only 12$/TB ingress/egress compared to the egregious 100$/TB (or so I have heard) AWS's outage and where you have to play this little dance of shutting down AWS itself or something to not pay it but I genuinely think that OVH has a lot of features
I am not kidding but when I say overwhelmed, I truly meant it so much that I had to take a walk outside to put things into mind, I was looking for partnership opportunities for OVH tho that time but in my mind I have rejected them because they are too big to partner at a small scale In my opinion
OVH has the 2nd most meaningful high content or similar metric (I forgot the website which shared it) after AWS, it had more high traffic websites overall even more so than gcp
Personally I do not like this complexity. Just give me compute and storage and let me handle the rest but I don't really like OVH thaat much (my opinions change overtime too) but please look more into it if you genuinely want european cloud provider, I am interested to know about it, What are the metrics which qualify for the "long shot", I am genuinely curious and wishing to discuss honestly.
But to give you another example (from the article): Try migrating Google Workspace to an EU solution. Actually impossible. I tried it myself, and gave up. The closest you’ll get is Proton, which isn’t EU to begin with and doesn’t even have half of the features Google Workspace offers.
Well proton isn't Eu but its switzerland and I think that proton is going to move some of their servers to germany after switzerland had this questionable decision which feared privacy services (including proton)
Proton recently got proton sheets/proton docs
Personally one of the largest issues I have with proton is the lack of extensibility. Like google has app scripts and similar api's but proton's lack of api's have frustrated me so much that I have built an api over scraping/using a puppeteer instance over it but its still in very finnicky state
Well, I am not sure about the newer BTP-based stuff, but the main ABAP-based core of an SAP S/4HANA system certainly does not need those capabilities, as it is still basically the same running in on premise systems. The priority of a couple of BTP apps might be quite low, if they are not starting from scratch.
Yes. Let’s take an extreme example: you think you exit in Japan, but you’re actually exiting in China. This means your traffic will be analyzed and censored by China.
The routers don’t care about where the provider says the IP comes from. If the packet travels through the router, it gets processed. So it very much matters if you do things that are legal in one country, but might not be in another. You know, one of the main reasons for using VPNs.
A more general case is for legal and SLAs. If a company uses one of these vpns to make sure their traffic only travels through a specific legal path, and then it's found that their traffic entered a different territory, there can be a lot of consequences.
The case I can think of most accessible would be anything that streams copywriten video.
I've wondered about jurisdiction in copyright for a while -- if I access a USA website from a Swedish server, make a copy on that server, then stream it to a French location for viewing all the while being in UK. Where has any crime/infringement occurred; which courts have jurisdiction?
Anyone know of any caselaw addressing these issues.
Are any VPN's getting China wrong? It would be pretty obvious. In fact, common VPN's I'm looking at don't even support China as an option. Obviously no VPN's are mixing countries up where it becomes clear from what you're allowed to browse.
But so "if you do things that are legal in one country, but might not be in another" is what I'm specifically asking about. Ultimately, legality is determined by the laws that apply to you, not the country your packets come out of. So I'm asking for a specific example.
And I already said, that if a site is attempting to determine permissions based on the country, it's doing so via the same list. E.g. when the country is actually Greenland, but you think it's the UK, and Netflix also thinks it's the UK. Which is why I'm saying, at the end of the day, is there any real consequence here? If both sender and receiver think it's the UK, what does it matter if it's actually Greenland?
China was just an example. Try to extrapolate on your own.
Take someone from Russia, Iran, wherever, trying to access information they aren't allowed to access, or sharing information they aren't allowed to share. They think they're connected to a neighboring country, but in reality are exiting from their own country. Therefore, the traffic gets analyzed and they fall out a window.
Imagine Snowden sharing information about the NSA while using a VPN that actually exited from the US. Things might have developed differently.
Yes, it won't matter for most services. But as soon as states or ISPs are involved, you're fucked if you get it wrong.
No need for the snark. Obviously we're not talking about somebody in Iran or Russia connecting to a VPN that just leads back into their own country, that would be idiotic. None of the VPN providers are providing anything like that. Those don't even make sense conceptually. A Western VPN provider that an Iranian or Russian is using isn't even legally allowed to operate nodes inside of Iran or Russia due to sanctions.
I'm talking about the realistic mix-ups that the article is using as examples. Where Somalia is actually going to France or something. That's why my original comment started with "Is there any real-life situation..."
No VPN providers are accidentally routing into an oppressive dictatorship.
I really don't understand what you're not getting here. I'm not trying to be condescending, but I explained it as simple as possible. But let's break it down again:
1) You currently reside in Iran for whatever reason.
2) You download, or have downloaded previously, a VPN software that does not tell you where you exit truthfully.
3) You connect to Pakistan, because you want to spread information that is illegal in Iran, but legal in Pakistan. You choose Pakistan because it is near you, so you get better latency.
4) In reality, your VPN exits not in Pakistan, but in Iran. Because they lied.
5) Iran is now able to monitor both your connection traffic to the VPN, and your VPN's exit traffic.
6) You die.
Simple as that. I don't see why this is not a realistic use case in your mind? One very prominent selling point for VPN providers is exactly this. Allowing reporters and other minorities to still safely access the internet in areas in which it is not allowed by law. You don't have to be an Iranian for that. You can just be there, as an international correspondent, using a western VPN, for example. Or you're visiting family after purchasing that VPN in Europe somewhere.
> No VPN providers are accidentally routing into an oppressive dictatorship.
The entire point of this article is that you as the user can't know that. And almost every country is applying some kind of censorship that may or may not affect you. As I mentioned in my previous example, Snowden is a real life situation in which this exact thing would have mattered. He didn't live in an oppressive dictatorship, yet a VPN exiting in Canada vs. exiting in the U.S. would have made a significant difference in safety for him.
You know what? Adding another example from a "non oppressive dictatorship" country: Germany.
Every few years, Germany tries to push for [Vorratsdatenspeicherung](https://de.wikipedia.org/wiki/Vorratsdatenspeicherung), which forces ISPs to store all connection data for later analysis by law enforcement. This is against the German constitution, so it gets overturned after a few years and the cycle starts again.
Point is, if you want your data to not be analyzed by law enforcement, and you live in Germany, you may want to use a VPN and connect to, for example, the Netherlands, where no such law exists.
Now, as we established, the VPN providers lie about it and maybe they don't have an exit point in the Netherlands, so they just let you exit in Germany again.
Same situation as before, but less extreme. Germany now will be able to put together metadata and find out what you were actually doing.
Are you going to jail for that? Maybe, if you did something highly illegal. You probably deserve it in this case.
But this isn't the point of discussions like these. It's always fine until something like Hitler happens and you happen to be jewish. You just can't know how the future will play out, and if your data is stored somewhere, it can be abused. VPNs promise to be a solution to this issue, and they apparently aren't. Which is a problem, even in a healthy democracy.
People like you pretending this isn't a problem, are a problem. It's the same argument as "I don't have anything to hide." Yes, maybe you don't have anything to hide today. Tomorrow? That might no longer be true. Take abortions in the US as a recent example.
An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series", as is the de-facto standard in NPM dependencies. Therefore, it's up to each dependency-user to validate newer versions of dependencies before publishing a new version of their own package. Lots of manual attention needed, so a slower pace of releases. This is a good thing!
Another example: all Debian packages are published to unstable, but cannot enter testing for at least 2-10 days, and also have to meet a slew of conditions, including that they can be and are built for all supported architectures, and that they don't cause themselves or anything else to become uninstallable. This allows for the most egregious bugs to be spotted before anyone not directly developing Debian starts using it.
You forgot to mention it is also tied to provable namespaces. People keep saying that NPM is just the biggest target...
Hate to break it to you but from targeting enterprises, java maven artifacts would be a MASSIVE target. It is just harder to compromise because NPM is such shit.
Maven Central verifies the domain used for the package namespace, too. You need to create a DNS TXT entry with a key.
This adds a bit more overhead to typo squatting, and a paper trail, since a domain registrar can have identity/billing information subpoenaed. Versus changing a config file and running a publish command...
Maven central also requires package signing. You're not stealing my signing key. It's on a yubikey. Game over, you can't publish malware in my name using my key.
> An example: Java Maven artifacts typically name the exact version of their dependencies. They rarely write "1.2.3 or any newer version in the 1.2.x series"
You can definitely do this.
To be honest, you just end up with the same thing via dependabot/renovate.
You can specify a dependency version range in Maven artifacts. But the Maven community culture and default tooling behaviour is to specify exact versions.
You can specify an exact dependency version in npm packages. But the npm community culture and default tooling behaviour is to specify version ranges.
Even if a maintainer uses a bot to bump dependency versions, most typically they will test if their package works before publishing an updated version, and also because this release work is manual (even if the bot helps out), it takes some time after the dependency is released for upstream consumers of it to endorse and use it. Therefore, nobody consuming foo 1.0.4 will use dependency bar 2.3.5 until foo 1.0.5 is released... whereas an npm foo 1.0.4 with bar dependency "^2.3.0" will give its users bar 2.3.6 from the very moment bar 2.3.6 is released, even without a foo 1.0.5 release.
Other languages seem to publish dependencies as self-contained packages whose installation does not require running arbitrary shell scripts.
This does not prevent said package from shipping with malware built in, but it does prevent arbitrary shell execution on install and therefore automated worm-like propagation.
You have separate people called "maintainers", and they're the ones who build and upload packages to the repository. Crucially, they're not the people who write the software. You know, like Linux has been doing since forever. https://wiki.debian.org/DebianMaintainer Instead of treating your package repository like a trash can at a music festival, you can treat it more like a museum, curated by experts. Unfortunately, this isn't quite the devil-may-care attitude the Node ecosystem is so accustomed to, and will be met with a lot of whining, so it never happens. See y'all in two weeks when this happens again.
That literally makes no difference at all. You’ll just vendor the malicious versions. No, a lock file with only exact versions is the safe path here. We haven’t seen a compromise to existing versions that I know of, only patch/minor updates with new malicious code.
I maintain that the flexibility in npm package versions is the main issue here.
You are using the word differently than everyone else I think. I’ve never heard someone using that word to mean maintain private forks. Then again, even private forks don’t protect you much more than package lock files and they are way more overhead IMHO.
You still need some out-of-band process to pull upstream updates and aside from a built-in “cool down” (until you merge changes) I see that method as having a huge amount of downside.
Yes, you sidestep malicious versions pushed to npm but now you own the build process for all your dependencies and you have to find time to update (and fix builds if they break) all your dependencies.
Locking to a specific version and waiting some period of time (cool down) before updating is way easier and jus as safe IMHO.
Vendoring literally just means grabbing the source code from origin and commit it to your repo after a review.
The expectation that every repo has important regular updates for you is pure FOMO. And if I don't do random updates for fun, nothing will every break.
> Version locking wont help you all the time, i.e. if you build fresh envs from scratch.
I'm confused on this. I would imagine it would protect/help you as long as releases are immutable which they are for most package managers (like npm).
> Vendoring literally just means grabbing the source code from origin and commit it to your repo after a review.
Hmm, I don't think it always necessarily means grabbing the source, it can also mean grabbing the built artifacts in my experience.
My biggest issue with vendoring dependencies is it allows for editing of said dependencies. Almost everywhere I've worked that vendored dependencies (copied source or built versions in and committed them) felt the siren song of modifying said dependencies which is hell to deal with later.
You are right about version locking, bullshit on my side, not sure what I was thinking.
I personally don't have a problem with the general ability to change vendor code. The question is whether you want it in an specific case or not. If you update frequently then certainly not. But that decision should be deliberate team policy.
> I personally don't have a problem with the general ability to change vendor code. The question is whether you want it in an specific case or not. If you update frequently then certainly not. But that decision should be deliberate team policy.
Fair, in the instances I ran into it was code that was downloaded and unzipped into a "js-library-name" folder but then the code was edited, even worse, the `.min.js` version wasn't touched, just the original one which led to some fun when someone "helpfully" switched to the min versions that didn't have the edit. IMHO, if you want to edit a library then you should be forking and/or making it super obvious that this is not "stock" library X. We also ran into issues with "just updated library Y" and only later realizing someone had modified the older version.
But yes, if it's deliberate and obvious then I don't have an issue modifying, just as long as the team understands it's "their" code now and there is no clean upgrade path in the future.
I'd require vendor code being committed to git and integrated into the CI/CD pipeline. It should be treated as if you own it, just with a policy whether you want to change it or not.
The difficulty to make changes obvious is same for forks and vendored commits, imho. You can write big warnings in commit messages, that's it I guess. Which kind of boils down to deliberate team policy again. But I generally prefer monorepos for various reasons.
I think some system would need to dynamically analyze the code (as it runs) and record what it does. Even then, that may not catch all malicious activity. It's sort of hard to define what malicious activity is. Any file read or network conn could, in theory, be malicious.
As a SW developer, you may be able to limit the damage from these attacks by using a MAC (like SELinux or Tomoyo) to ensure that your node app cannot read secrets that it is not intended to read, conns that it should not make, etc. and log attempts to do those things.
You could also reduce your use of external packages. Until slowly, over time you have very little external dependencies.
Hire an antivirus company to provide a safe and verified feed of packages. Use ML and automatic scanners to send packages to manual review. While Halting problem prevents us from 100% reliably detecting malware, at least we can block everything suspicious.
reply