This reminds me of a time at Red Hat when a worm was going around and infecting Red Hat systems, one of the engineers reverse engineered the worm and wanted to release it in the wild to fix the bug, legal wouldn’t let them. I think legal was right (for a public company) but this kind of shows the actual right response, in my opinion.
Keep in mind in like 1999, you didn’t expect upgrades via package managers online for most large customers so this was an appealing release vector.
It also sounds like the time Max Butler exploited a buffer overflow in BIND to patch a bunch of DOD systems. As we later found out, he added some extra "functionality" to that patch. Who's to say FBI hasn't done that in some small fraction of cases?
Yeah I can understand legal's approach and maybe not wanting test the waters by going to a judge and all that work.
Microsoft and the DOJ have established a track record of getting judicial approval and so on. I'm sure now it is a much more known quantity / outcome legally for them than Red Hat back in 1999. I can imagine there is a good chance of a judge in 1999 think "You're who? and you want to wut wut the wut wut?"
If I'm understanding this correctly, the DOJ authorized the FBI to exploit the exploit to remove the exploit from exploited servers? This proactivity is something I remember hearing recently that the NSA wished they could have
The mechanism seems to be a search warrant. The FBI applied for a warrant to "search" all compromised Exchange servers in the United States, and to "seize" the illicit malware on those servers by executing a specified series of commands.
A few excerpts from the above link:
"FBI personnel now seek authorization to search the compromised Microsoft Exchange Servers and uninstall the web shells on those servers". (6th page of the PDF)
"This warrant authorizes the United States to seize and copy from Microsoft Exchange Servers located in the United States the web shells identified in Attachment A, and to delete the web shells from those servers." (11th page)
> a magistrate judge ... has authority to issue a warrant to use remote access to search electronic storage media and to seize or copy electronically stored information located within or outside that district if ... the media are protected computers that have been damaged without authorization and are located in five or more districts.
> A warrant may be issued for any of the following:
(1) evidence of a crime;
(2) contraband, fruits of crime, or other items illegally possessed;
(3) property designed for use, intended for use, or used in committing a crime; or
(4) a person to be arrested or a person who is unlawfully restrained.
Interesting, I had missed that. The language allowing remote searches when computers "have been damaged without authorization and are located in five or more districts" seems to be recent, added by Congress in a 2016 amendment: https://www.justice.gov/archives/opa/blog/rule-41-changes-en...
Usually warrants have to be pretty specific. This one is, as to the thing that they are searching for. But it isn't, not as to the places to search. Anyways, I doubt the warrant will be challenged, so it works. It's quite the innovation. I... like it, provided they really do only uninstall the malware and close the vulnerability.
It's very specific about the places to search. They are listed in appendix A -- every server that has a web shell installed is enumerated by the URL of the shell.
The first time they innovate like this, they'll be very careful to do it in a way that seems beyond reproach, probably only uninstall the malware and close the vuln etc.
Once it gets established as a legal thing, they'll keep pushing it...
Is this setting any new precedent for a warrant allowing removal of a file from a server, or is it already common practice when shutting down illicit servers?
can they also copy anything suspicious in "plain view", like whatever is brought by a simple search on the hacked Exchange? That is one huge dragnet. Of course any good citizen wouldn't mind when a law enforcement would come take a look around after the crooks have already broken into the place.
Edit: to the iudqnolq's reply below - warrant doesn't have to explicitly permit it. My understanding [IANAL] is that, at least in the physical world, whatever gets in "plain view" of the officer during any authorized law enforcement activity also becomes a fair game. I.e. they were called for the noise and upon entering see a kilo of heroin laying on the table - the heroin comes into play even though they didn't have a search warrant for it. So i'd expect that the same principle would be applicable in the virtual world too.
> can they also copy anything suspicious in "plain view", like whatever is brought by a simple search on the hacked Exchange?
If it isn’t passively visible, it is not, by definition, in plain view. If they have to do a search, however simple, beyond what is explicitly authorized in the warrant, to find the information or to find whatever would give them probable cause to believe it is contraband or evidence of crime, they can neither seize it nor get a search/seizure warrant based on their observation of it under the plain view doctrine.
No. "This warrant authorizes the use of remote access techniques to search the electronic storage media identified in Attachment A and to seize and copy from the electronic storage media identified in Attachment A the web shells, used by actors to communicate with and distribute files to victim computers to infect them with malware, as evidence and/or instrumentalities of the computer fraud and conspiracy in violation of Title 18, United States Code, Sections 1030(a)(2) (theft from a protected computer), 1030(a)(5)(A) (damage to a protected computer) and 371 (conspiracy). This authorization includes the use of remote access techniques to access the web shells and issue commands through the web shells to the software running on the electronic storage media to delete the web shells themselves.
This warrant does not authorize the seizure of any tangible property. Except as provided above, this warrant does not authorize the seizure or copying of any content from the electronic storage media identified in Attachment A or the alteration of the functionality of the electronic storage media identified in Attachment A."
Yeah. I don't like the idea that law around search warrants can be used to authorize the government to do things like this. That's not what a search warrant is for. I think the judge was wrong to authorize it, but I don't count on judges to resist law enforcement requests.
If this sort of thing is a good idea, there should really be legislation about it specifically.
More like – FBI wanted to patch affected servers remotely without getting every owner's consent but weren't sure of legality so they asked a judge to sign off on it.
Actually, according to the brief they actually didn’t patch the servers, but only removed the web shell: “This operation was successful in copying and removing those web shells. However, it did not patch any Microsoft Exchange Server zero-day vulnerabilities or search for or remove any additional malware or hacking tools that hacking groups may have placed on victim networks by exploiting the web shells.”
Asking a judge for a really broad constructive warrant and pretending that the founding fathers anticipated this use case in their aggregate infinite wisdom is very common
It's bizarre to think that we should be bound forever more to only do the things that people in 1789 thought appropriate. But it seems to me that they followed the procedure outlined in the 4th amendment; they went to court and got a specific warrant to remove a specific bit of malware from a specified set of servers.
Madison deliberately left many provisions in the Constitution and Bill of Rights open-ended because he wasn't an idiot and knew that the founders could not anticipate everything. Had the FBI acted on its own it would be overreach. They went to court, got a warrant, and did pretty much the minimum required to eliminate the threat.
The statement says they were authorized to send commands to vulnerable servers through the exploited services/webshells which removed said webshells from the system.
What if servers that got “fixed” weren’t American? Would that mean the FBI went outside its jurisdiction and could be seen as an illegal act across international borders?
A previous comment quoted officials stating that no patch was applied. They only removed the specific malware in the warrant and left the servers as they were. Which means they could simply be exploited again.
I'm kind of annoyed by some of the general negative tone of some of the comments here: "Ha! The FBI is guilty of hacking", or "But they didn't patch the root cause!"
In my understanding, the FBI:
1. Applied for and received a lawful court order
2. To make as minimally invasive as change as possible to help the targeted networks
3. While making a best effort to contact the network owners to tell them what they were doing and then
4. Widely publicizing what they did.
Not everything is some big "gotcha" conspiracy. We can just say "thank you" and move on.
Though their intentions are probably noble, I get a niggling feeling when I hear this.
To me the FBI's behavior is the equivalent of "Since there have been several break-ins in our district and our local police dept. has determined that the internal door locks of business offices in our district are faulty, so we have broken into each business, and fixed the locks ourselves, we have attempted to trace you and notify you if you have publicly available contact info". Noble, yet still chilling.
It's actually not uncommon for police patrols to notify people if they see unsecured property. And that can be hard to do if no one is home and they don't want to alert actual criminals. To enter the property they often need a court order (unless there's an immediate threat), but I've seen efforts by police to drop notes wherever they see unsecured doors or windows that could be targeted for a break-in. Especially if there have been break-ins targeting certain weaknesses (e.g. open windows), police will look for those and alert owners.
The problem with a hack is that it can be done within hours on properties everywhere, so just dropping a note won't be fast enough for most. Where there is an equivalent in real-world (i.e. immediate threat of a break-in), police can and will act to secure that property.
So I don't see the chilling effect, it's something that (at least in Europe) is common part of preventive policing.
Not unlike glaziers boarding up a broken window to prevent intrusion or weather damage (although that also serves as a good opportunity to advertise the services to a newly-discovered prospective customer)
>To enter the property they often need a court order
Exactly, in this case they got the equivalent of a court order to enter every house and check the locks.
Which no judge would be likely to approve if it were actual physical houses. But servers...
The weird thing is the warrant authorizing entry into pretty much any computer anywhere. Could they get a warrant to enter any computer anywhere to look for child porn? Or "terrorism" or whatever? Probably not, that's not the way warrants work, except now are they going to?
> so we have broken into each business, and fixed the locks ourselves
... which is not what happened.
And - "open door" metaphors do not work well for cybersecurity due to an entirely different threat model. These vulnerable machines are globally available for anyone to break into with almost no effort.
I don't know that anybody is frustrated about what they did. It's that anybody else who could have benevolently done the same thing, much faster, but without all the process, would be in massively hot water. I think mostly it's just frustrating to want to do the right thing, know that you could, also know that you can't, and then see somebody get to do it officially so much later.
Top that all off with the fact that it's really hard to trace who and what were involved, versus a more transparent process where legitimate experts could chime in and prevent any further harm. Instead we have to cross our fingers that they did it right and that the judge understood what he was agreeing to.
And I mean it's not a unique problem to this circumstance. Just kinda how institutions tend to be.
It's definitely that way with a lot of things, but it can still be super frustrating for anyone who would prefer to use their knowledge for good, but due to scar tissue, we know we can't. Scar tissue being both the laws as understandable and justifiable reactions to misuse by those with less admirable intentions, and from seeing the laws used to unjustly prosecute people trying to do the right thing in the face of authority who won't listen, and assuming wrongly that all they need is a demo to help them understand the severity of the problem.
That's what police is for. There are enough situations outside of the digital world where it's better to call the police and let them help instead of getting too involved yourself. I believe it's good that not everyone can apply their own judgement in what needs fixing but that police/law enforcement applies certain standards there (although obviously not always correctly).
> I don't know that anybody is frustrated about what they did. It's that anybody else who could have benevolently done the same thing, much faster, but without all the process, would be in massively hot water.
But the process is the whole point isn't it? If anyone was able to do this whenever they wanted and without oversight that would be a major concern. This is an example how this kind of thing can be done in a safer, legal way. Of course it won't be as fast as just letting anyone do this, but that's a price worth paying.
If, for example, part of the mechanism that allowed them to do this involved a search warrant for "all computers in the US"...that's a precedent that has obvious, chilling, future implications.
Perhaps that wasn't the mechanism, but it's such a broad action that I'm worried something like that is involved.
Imagine you were were one of the targets for this exploit, the FBI intervened in the issue, and three months from now damaging information started leaking to the press.
How would you show the source was the original hackers and not the FBI?
Interesting. They're violating their own CFAA law (accessing a computer without authorization or exceeding the access granted) to remove web shells. Legally, this is hacking. Which means that the FBI just hacked a bunch of Exchange servers to clean them.
So the message here is, if you don't clean up your act and you're on a USA network, we'll do it for you without your permission.
The beef is at the end of the article:
This operation was successful in copying and removing those web shells. However, it did not patch any Microsoft Exchange Server zero-day vulnerabilities or search for or remove any additional malware or hacking tools that hacking groups may have placed on victim networks by exploiting the web shells. The Department strongly encourages network defenders to review Microsoft’s remediation guidance and the March 10 Joint Advisory for further guidance on detection and patching.
The FBI is attempting to provide notice of the court-authorized operation to all owners or operators of the computers from which it removed the hacking group’s web shells. For those victims with publicly available contact information, the FBI will send an e-mail message from an official FBI e-mail account (@FBI.gov) notifying the victim of the search. For those victims whose contact information is not publicly available, the FBI will send an e-mail message from the same FBI e-mail account to providers (such as a victim’s ISP) who are believed to have that contact information and ask them to provide notice to the victim.
If you believe you have a compromised computer running Microsoft Exchange Server, please contact your local FBI Field Office for assistance. The FBI continues to conduct a thorough and methodical investigation into this cyber incident.
"This section does not prohibit any lawfully authorized investigative, protective, or intelligence activity of a law enforcement agency of the United States, a State, or a political subdivision of a State, or of an intelligence agency of the United States."
18 U.S.C. § 1030(f).
DOJ obtained authorization here, most likely under Fed. R. Crim. P. 41(b)(6)(B)--which, interestingly enough, cross-references the CFAA.
That is not ok. A law that purports to outlaw computer fraud and abuse should particularly prohibit the government from committing computer fraud and abuse.
That's not what it does. It clarifies that lawfully organized investigative, protective, or intelligence activities are not fraud or abuse. Which they aren't, by any normal definition.
Warrants have always allowed the bypass of physical security devices, why would digital ones be any different?
Of course not. I'm using CFAA as a moral definition of hacking, not legal. I'd be amazed if this doesn't lead to warrants issued that enable search & seizure on domestic individuals and corporations.
Edit: After re-reading my post above, I guess I did suggest they violated the CFAA in law. Not my intent to say that. I'll leave it as is.
> They're violating their own CFAA law (accessing a computer without authorization or exceeding the access granted) to remove web shells.
...
> I'm using CFAA as a moral definition of hacking, not legal.
Please don't move goalposts; it degrades the conversation. Better to own the error to let the conversation proceed normally, allowing everyone to learn together. (I doubt most of us knew about the LE carve out, for instance)
By default, we expect moral people to conform to the laws of their jurisdiction. Not because the laws are necessarily morally positive; most laws are morally kind of neutral, but because the predictability itself is a good virtue.
Of course, that default presumption can be overcome with a relatively low burden of proof.
I think it is reasonable to conclude they were referring to the bulk of the law that is administrative/procedural/etc. There are 53 titles of US code, and one of those (title 18) is about criminal code. The rest are predominately not matters of morality.
Most things that are legal aren't outlined in the law anyway, they're omitted.
> Most things that are legal aren't outlined in the law anyway, they're omitted.
So for slavery, you would expect the laws to eg deal with run-away slaves etc, not so much with slavery itself.
> I think it is reasonable to conclude they were referring to the bulk of the law that is administrative/procedural/etc. There are 53 titles of US code, and one of those (title 18) is about criminal code. The rest are predominately not matters of morality.
Yes, exactly. And I am presuming here, that there is a presumption in morality that all-else-being equal, it's more moral to stick to these neutral laws, just because it makes living in a society with other people more bearable.
Eg in a moral sense it doesn't matter whether people drive on the left side of the road or the right side. But if there's a law about driving on the books, you better follow it.
(No clue whether this is strictly speaking something about morals or more about ethics?)
> So the message here is, if you don't clean up your act and you're on a USA network, we'll do it for you without your permission.
I am not sure on any prior case law on this, but there's examples in the real world if you leave a dangerous attractive nuisance out in the public space, where it could potentially be harmful to people or animals. Local law enforcement or civic-minded citizens will remove it. It could be argued that leaving exposed outlook web access rooted systems out there for anyone to use on the public internet is not too dissimilar.
On a federal level? If you leave an abandoned ship leaking toxic chemicals anchored somewhere in a bay, don't be surprised if the USCG, a federal law enforcement agency, comes and seizes it...
Microsoft and the DOJ have several times gone to a judge to get permission to take over botnets. The reason being that to do so they take over the botnet and the result is of course that if they control the botnet they then have control of those computers. But you can't completely disassemble some botnets without taking it over.
> Just seems odd that somehow the FBI is the best server admin here?
Without arguing for or against it, I can see this new role viewed as a "sysadmin of last resort" wherein an authorized institution steps in to ensure a minimum security level among neglected systems in their jurisdiction.
> And what happens if they break something while patching the exploits?
Probably the same thing that happens when officers injure people or damage property in the course of executing a warrant (which is quite common). In short, either the victim is rich and/or outraged enough to venture a lawsuit against the relevant agency or they just file insurance claims and hope for the best.
I wouldn't be surprised if part of the reasoning for signing off on this action was that the risk of damage was considerably lower than what is routinely understood to be part and parcel of executing search warrants.
If your front door breaks leaving your house unsecured such that it is noticed and reported to authorities... you'll find that many local police will secure your building for you in the most hamfisted way possible. It's still probably better than leaving it open.
Its not just the insecure servers that are at risk. These systems can be used as jumping off points for further intrusions elsewhere. So to extend the broken front door analogy, its not just that the police are worried about your property, they are also worried that your house will be taken over by criminals and used to sell crack / store stolen property / run a brothel / insert other crime of choice.
Presumably the FBI limited this operation to "U.S. Networks". I wonder how they determined that? Based on domain registration? IP block ownership? What about a non-US company with servers outside of the US that has a Point-of-Presence IP inside the US? Seems like there's no perfect way to determine programmatically.
> The presumptively U.S.-based Microsoft Exchange Servers, corresponding to the
approximately [redacted] web shells in Attachment A appear to be located in five or more judicial districts, according to publicly available Whois records and IP address geolocation
A substantive portion of these unpatched servers end up ransomed. And if not yet, they will be. A proportion of ransom victims show up expecting the FBI to help, even if they were extremely negligent in allowing the incident to occur. Another very high proportion just pays the ransom.
The FBI here aren't just "protecting lazy admins", there are some further reaching consequences to failing to act.
Note also people are talking about "applying patches" but the order more specifically talks about removing web shells. If my experience is indicative, there are more hosts that applied patches too late and didn't remove the web shells mass scanners deployed, than hosts that never patched. I expect a lot of this disruption is about deleting a one line .aspx file.
Imo seems reasonable. There are plenty of other government agencies with far more power in their respective industries. FDA, Public Health Departments, the myriad of banking regulators.
In may of those, the respective regulators can shit the entire business down. Here, the FBI didn't even power the servers off and they got a warrant without going through a secret court
Companies have had plenty of time to address the issue on their own, at this point
I'm interested in the moral hazard this creates if this practice becomes widespread. If your servers are "too big to fail", and the FBI/NSA can reliably zero-day into your servers to patch zero-day bugs, that seems like a pretty good deal for skimping on some of your security budget.
FBI/NSA doesn't give a damn about some unpatched servers in the wild. They are probably clearing web shells in order to bait hackers into reinfecting the same servers, and try to locate/attribute the original bad actor.
Well, this is a totally awful development. In the 80s the idea of white worms being used to patch vulnerabilities was rejected for good reason, so I have to think this has little to do with security and much more to do with normalizing behavior that really shouldn't be tolerated. They didn't even patch the hole...
Before anyone tries framing it as a service to the security of the majority - understand that this is the introduction of a new attack vector: state actors hamfistedly bumbling around your network while "doing you a favor". If the threat even approached a level justifying this kind of action, the far more effective and less damaging approach would be directing upstream networks to blackhole routes to the machines.
In less than 18 months there will be a startup (likely from Israel) who will provide this kind of service for US Government, and it will normalize and move from state to commercial
It’s not a development. Court-authorized public/private patching initiatives have long existed. They’re much higher profile and news worthy theses days, though.
Care to provide an example of the USG leveraging a vulnerability to delete files on hundreds of domestic servers without the permission or knowledge of the owners? Because the closest thing I can think of involves the military targeting a foreign botnet. I know Microsoft went after a botnet with a forced Windows update mechanism, but Microsoft isn't the FBI and their update system is a known quantity.
I would like to see this type of thing become more popular with general law enforcement.
It is very frustrating to have essentially no recourse available to stop the constant vulnerability scans targeting my house.
If random people constantly walk up to every house on the street looking for pick-able locks, the police are (Setting, for a moment, aside over/under policing and other issues) available to help stop them.
But, for the digital equivalent, our collective response (especially among technical people) is typically "[shrug] Make sure your locks are unpickable and your windows unbreakable. And if you cant handle that, then just move in to the Facebook highrise"
> This operation was successful in copying and removing those web shells. However, it did not patch any Microsoft Exchange Server zero-day vulnerabilities or search for or remove any additional malware or hacking tools that hacking groups may have placed on victim networks by exploiting the web shells.
So they removed the IOC but left the hole wide open.
This kind of "help" is going to be an incentive to stop doing business with US hosting companies.
You have misunderstood me: I don't want them patching systems; I don't want them touching systems that aren't theirs at all.
It's illegal for a reason. They committed the same crime by removing the web shells as was committed by the person who placed them there. (Who can put them right back.)
Interesting precedent. Will they bill Microsoft? If not, I'm curious if this could mark the start of externalizing security and cleanup responsibilities to the federal government.
if your unpatched server is being used as a command&control server in an active offensive campaign, you can be liable for damages your server caused.
I hope that in the future there will be some fine for Server Neglect (leaving internet facing server unpatched and hosting web shells for 5 days after patch publication by vendor) and you will lose your server and all your data for such misdemeanor.
> I hope that in the future there will be some fine for Server Neglect (leaving internet facing server unpatched and hosting web shells for 5 days after patch publication by vendor) and you will lose your server and all your data for such misdemeanor.
I can see it now: "Government stole decades of family photos and videos because my Linux/Plex server was available online."
There is a constitutional basis for a warrant being permitted, upon probable cause, to execute specifically and exclusively a search.
This isn't a search, it isn't a warrant, and there's no constitutional amendment that outlines the situations in which the feds are allowed to break into my computers.
Did you read the article or just the headline? A judge issued a warrant. You can argue over where they should have issued one, but “it isn’t a warrant” is just wrong...
Here's the constitutional language: "no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized." They got a warrant, specifically describing the specific places (web servers) to be searched and the particular things to be seized. Probable cause is easy, these servers were actively attacking government computers.
This is a search. Specifically, the web shells are (1) evidence of a crime, (2) contraband, fruits of crime, or other items illegally possessed, and (3) property designed for use, intended for use, or used in committing a crime. Any one of these three would be a valid basis for a search warrant.
Wow. This is crazy unconstitutional. DOJ could seek civil injunction perhaps, but using criminal authority to break into servers without probable cause of any criminal action is crazy bad.
Another good reason to stop using proprietary binaries and instead compile your own source - even if you use code under proprietary copyright. Imagine if France also decided to "help" and bricked your server on accident.
So if a criminal is running from the police and breaks into your home to hide, what your saying is that the police shouldn’t go in and remove the criminal from your house because that would mean the police are committing a crime by entering into your home?
I think there is a balance here. State run anti virus and malware agency seems also the wrong solution. The issue with granting power in cases like this is that it’s really hard to unwind it. What happens when the FBI wants to remote to your personal laptop to remove an exploit? How do you balance all this?
The authorities actually have wide latitude... there was one case where a house was destroyed in pursuit of a fugitive, LegalEagle did a whole show on it: https://youtu.be/Dk8QO6jE5dA
Keep in mind in like 1999, you didn’t expect upgrades via package managers online for most large customers so this was an appealing release vector.