"An Amazon spokesman said the company doesn’t use confidential information that companies share with it to build competing products"
Maybe...but in the past, AWS proactively looked at traction of products hosted on its platform, built competing products, and then scraped & targeted customer list of those hosted products. In fact, I was on a team in AWS that did exactly that. Why wouldn't their investing arm do the same?
Cannot up vote this enough. During my time both at Retail and AWS it was perfectly normal to trawl production customer data and come up with ideas to launch competing products. Prices were always set lower or free offering justified as data-driven and customer obsession. I hated the gas lighting their customers and left in disgust of the company and its leadership which encourages that behavior.
I know it's hard to do when you're making good money and would be going against co-workers.
But, if you see something, say something. This crap continues because there are too many folks that are happy to help support immoral business practices for some extra scratch. This isn't all on you in particular but when google folks started raising hell about Chinese censorship the company was forced to move. We all have the power to withdraw consent over how our labour will be used and, as software developers, we've got a strong enough employment market that we have real power to help make companies behave better - power that folks working in the warehouse are absolutely deprived of.
I mean the problem is corruption begets corruption. They WANT do to these things because you're going to get a massive bonus when the product you 'invented' does well because you stole the idea from an Amazon customer.
Amazon needs to be properly taxed so that this crap doesn't happen anymore.
The idea that they shouldn't pay taxes simply because they're large should absolutely enrage everyone.
This topic has nothing to do with taxes. They will always be trying to increase their bottom line whether that line is before or after taxes makes no difference. What is needed is a whistleblower. Not just a “when I worked for Amazon we did bad stuff”. We need that person to contact startup X, whose software and customer list was compromised. And then, this is key, share knowledge and proof of these accusations. Hell, do so through an attorney where you negotiate x% of resulting litigation proceeds of you’re worried about your privacy and financial situation. I’m pretty sure this would play out badly for Amazon in court.
Here's the thing it's not just Amazon they're just the biggest fish in the manufacturing pond in NA. Who's fault is it exactly? The system is designed to MAKE SHARKS LIKE THIS.
You're a fish, eat other fish and evolve into a shark, you eat other sharks and become a whale shark, you start eating everything and then become godzilla.
A whistle blower isn't going to fix this. This is the system. The system MAKES godzilla sharks like this.
Oh yeah sure a whistle blower will do what? Get amazon fined for how much?
Then they just change tactics. Outsource. Make agreements and partnerships and farm out doing the same thing just with different proxies. I mean come on man this is a company that can buy other countries.
And let's not forget Microsoft was pulling the same shit until they got put under the same charges and then all of a sudden years later after Bill got tired of stabilizing his empire and making sure it would live without him he became a saint all of a sudden. Cuz like yeah if I was richer than 99% of the people on the planet yeah I could start being a nicer person and shit too.
The current zeitgiest is that taxes are Unamerican and tax evasion is American. Until that is fixed proposing solving problems with taxes is a pretty empty approach since people are happy to elect tax evaders to the highest office in the country and joyfully utilize services that are offered by companies that are famous for their tax evasion (Apple, Amazon, everyone honestly).
I think taxes aren't really a solution anyways - fines might be but taxes would hurt honest players just as much as dishonest ones. What they did is (AFAIK) illegal and needs to be punished, if it isn't then there is no incentive for them to correct their action.
There's specific credits/exemptions in the tax code that they are able to exploit (and perhaps they can only exploit some of them _because_ they are a big company), but it really isn't about their size.
No, I said quite clearly there are no exemptions due to their size. There's a difference between an exemption for companies with N+ employees and tax credits/exemptions you can capitalize on because you are a company that makes billions of dollars and can afford to take on different behaviors to take advantage of them. That's not the same thing at all.
I see what you are saying, but I think that if things exist in the tax code that can only be taken advantage of by large companies, it is -effectively- the same thing as a big company exemption.
He’s not talking about effectively what happens. He’s saying textually there is no “large company tax break” and if you tell legislators to repeal “large company tax breaks” they’d all give you confused looks. Textual precision is important if you want to start fixing the problem. What exactly are the test breaks that work for big companies and not small ones? Why do they exist? What was the original intent? What are the side effects?
Nobody said that there is a tax exemption for companies over N+ employees. The OP said that amazon doesn't have to pay taxes.
Can we please stop arguing like influencing is only true if it is done in the most direct way (similar to the quid pro quo debate). Obviously if big company lobbyists try to get tax law in their favour they are not pushing for "please write a law which exempts companies over N employees from taxes." They push for laws that sound innocent but only they will be able to take advantage of, just like it is at the moment. The outcome is still the same they pay less or zero taxes.
What types of AWS data would be trawled? Are we talking about data inside S3 buckets, database schemas, particular architecure styles, the fact that a product is consuming {x, y, z} amounts of cloud resources, or simply "spending $m / year" in gross?
I worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer.
We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money.
No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.
I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. They are extremely serious about that stuff since they know that will erode their customer's confidence.
But there's probably other superficial business data that's helpful to evaluate that.
That's not necessary unless SOMEONE includes computer programs.
Yes, when things go very seriously wrong, I believe AWS can have literal people override that permission, which will leave a mile long audit trail and likely accompanied by an internet scale outage.
The point I’m trying to get across is that the default viewpoint of many knowledgeable developers I know is ‘Of course AWS can’t see inside my EC2 instance because X’ — where X is some magical technology that doesn't exist.
I don’t want to devolve into audit logs and permissions and multi user key signing and wether they actually do or not.
The statement that ‘they can’t’ is 100% false, full stop. That’s all I’m trying to get across.
The technology to do it does exist likely on hardware you possess. The trusted computed platform lets you build a signed OS that encrypts its data using keys on the TPM. Using this, you could build an S3 implementation that stores customer data, but doesn’t let you access it.
It’s probably not a good idea to make a system with no human fallback, but it IS possible with current, non-magic technology.
The reality is that groups of people inside AWS have access to your stuff. A given person might only be on the S3 or EC2 team... but each of those teams can ssh to hosts in production, or has other access that could be used to compromise your data.
Amazon does take privacy and security very seriously, but these systems are run by people. Attacks like the recent Twitter attack could work for various AWS services.
Are you sure about that? Most of the aws provided S3 sdks include the option of client side encryption. Not to mention that there are plenty of third party options for that as well. AWS could I guess look at your s3 data, but it will just look like gibberish.
I think it’s pretty clear the person you are responding to is not suggesting AWS can magically break encryption, but rather that they “have access to your stuff” that is actually on AWS. There are plenty of AWS customers running data through, or storing data on, AWS that is sensitive in the form it is in on AWS. If you have an rdbms (database) actively running on AWS for example it is not e2e encrypted. If you are terminating a customer TLS connection on an ec2 hosted web server their web form upload is exposed to that machine. Etc etc.
I worked with a DO on an technical issue, and they were steadfastly against me granting them temporary access to our servers even though it would have made the issue easier to diagnose. Cloud provider that verifiably get caught doing this will quickly lose the trust of all their large customers
Reading through that second one, while the inciting incident was certainly pretty bad, their eventual response was, to my mind, all that could be hoped from a company in this day and age:
They recognized that their processes were too mechanistic and inhuman, and introduced a lot more compassion and open communication into them—and even chose to spend more money on hiring people to reduce ticket queue wait times.
I'd say that speaks volumes in DigitalOcean's favour.
The audits check that controls are in place, not that the controls are technically bulletproof or people-proof.
Source: Worked at AWS for several years including working on systems that had audit requirements for [secret project where I could not know the name of the customer because I don't have TOP SECRET security clearance].
Nobody said things were perfect or bullet proof. But that they are there, and it's not just 'trust us'. And it's not just single technical controls - the control regimes include mitigations against technical failure and requirements for ways to catch collusion and actions taken outside of authority.
And there are lots of things that many folks at the big cloud providers don't know about their internal threat management and monitoring. Source: Audited most of them for that customer you weren't allowed to know the name of. :)
Also, it is possible to build systems such that, no, there isn't a 'root' or 'unlimited permission' or whatever. Or that there is, but it's a multi-person credential.
This is one area where AWS takes things MUCH more seriously than it's competition, and they don't talk about it enough publicly.
The critical factor here is whether there are controls in place to prevent it. Sure, somebody probably could, but what to what lengths must that person go to do it, and what happens when it is discovered? Most things are not technically impossible, after all.
for its faults aws takes data privacy super serious. if you are in support you cant even see attachments customers put on cases without providing auditable justification
and you def cant see in s3 buckets or instances. hell if a customer sends you a link to an object in their s3 youre not supposed to open it
You keep making factually incorrect statements. I'm not going to go into detail to refute them, because I don't feel comfortable sharing internal design details and security mechanisms, but your comfort in confidently asserting falsehoods is disconcerting, to say the least.
I find it funny that none of the people here arguing really understand what data is important from a strategic sales point from view and what's not. The customers databases and other crap they store on the cloud. Not really important.
The raw billing information, oh motherfucking yes.
This is incorrect, at least from a logical POV and why it's hard to trust what cloud vendors say. A statement like this is either naive (most likely) or actively attempting to mislead.
Technically, its absolutely possible. Most likely you'll just need a support ticket or bug, and then you can troll around as engineer.
Also, security teams also usually have access to stuff when things get interesting.
Better to say that access is strictly on a case by case basis and monitored thoroughly.
Ideally customer is notified each time it happens - that would be cool, but likely technically not possible since data ends up in so many systems (like logs, SIEM, telemetry, debug files, backups, data scientist desktops,....)
> Ideally customer is notified each time it happens - that would be cool, but likely technically not possible
You're underestimating the investments that AWS (and Amazon at large) make in to security, confidentiality, and auditing. You're also missing a fundamental implication of building AWS on AWS primitives.
As a relevant example there is only one AWS IAM and one CloudTrail. It's a core tenant of AWS IAM to put that control and root of trust in to the customers control. That means when developer support is helping with your ticket they do so via your accounts AWSServiceRoleForSupport role. That means you can control whether that role exists, which principals can assume it, the capabilities it has, and you can see those same API calls in your CloudTrail logs. Although it would make support difficult you're welcome to delete that service linked role and prevent support.amazonaws.com from assuming said role in your account.
Yes, those are great features for compliance. But you seem to believe that your AWS instance is indeed yours. IAM is a concept built on top of lower level primitives that you do not control, but Amazon does.
I'm not talking about Amazon SSH into your EC2 instance - but of course they can do that also - at will, without you authorizing it.
Lower level disks, logs, hypervisor, telemetry, etc.. are accessible beyond your control.
> IAM is a concept built on top of lower level primitives that you do not control, but Amazon does.
Of course there are lower level primitives. And if the public documentation and observed behavior is insufficient I encourage you to inquire more about the various compliance, certification, and third party auditing programs in place https://aws.amazon.com/compliance/programs/. However at some point this approaches solipsism and I can’t prove a negative in a HN thread.
> I'm not talking about Amazon SSH into your EC2 instance - but of course they can do that also - at will, without you authorizing it.
No. Extraordinary claims need evidence. Either you have serious non public information counter to many AWS statements ... or you misunderstand some fundamentals of SSH and public key cryptography.
> Lower level disks, logs, hypervisor, telemetry, etc.. are accessible beyond your control
I would encourage you to read the AWS data privacy statements https://aws.amazon.com/compliance/data-privacy-faq/. Particularly the definitions of “customer content” and the “shared responsibility model.”
This really isn't how modern security works at most cloud companies. Even if you have root class credentials or the ability to escalate to them in some way (and that's a big if by itself), its a LOT of steps to get access to customer data, almost always involving broken glass, many protection layers, and often requires cooperation of multiple other root level people/credentials from completely different teams.
Depending on how the infrastructure is built, or what the particular service set up, it may not even be possible to gain access to specific data without extraordinary means, possibly involving replacing physical hardware.
I already corrected my statement in another reply. You're right. I said probably only a handful of people can access customer data to do their job. I personally never met one. The goal of my comment was to illustrate that in my experience handling customer data there was a big deal. It's not like something you can casually query to see if a particular customer has a good business or not.
It’s the thing they tell you the most when you work there. Like in a a obnoxious way. Most infosec training is about that.
If someone has access to customer’s data for their work they have to do a bunch of extra training and do other stuff. Potentially sign some things and there’s probably a different way to authenticate. I really don’t know because I never had to do that and nobody I knew had that type of access but I heard when you do you have to put with more things.
But then what about other commenters saying that this is exactly what their sectors of the company do? Do you think it's impossible that a massive company like Amazon that controls an ungodly amount of the Internet would break those rules? Especially when the government of their home country hasn't pursued an antitrust case in God knows how long
>But then what about other commenters saying that this is exactly what their sectors of the company do?
i don't see anybody claiming that amazon is harvesting data from inside their customer's infrastructure. amazon has a lot of data that's "amazon's data" that would tell them about businesses that are operating on AWS that might be ripe for competition.
For example, they know what your AWS bill is, and how it's been trending. If you pay a huge bandwidth bill and it goes up 50% each month, they know you've got a business model that's working and that they can undercut you on one of your big expenses.
You're right that other commenters aren't necessarily saying that they're peering into buckets and PII...but I err on the side of questioning that the company is committing wrongdoing.
However, metrics like AMI popularity is Amazon's data... and that definitely informs first-class AWS product development. Once the company identifies a business opportunity, different teams often investigate "build" and "buy" options simultaneously.
Same goes for retail - Amazon works backwards from high-margin categories to identify opportunities, then pursues investment in existing brands versus spinning up products under the company brands.
This all feels very monopolistic to me, but regardless it's worlds apart from the accusation of stealing private information through faux investment offerings.
I don't think the difference is all that large. Legally, yes. But ethically they are pretty close. After all, any product launched like that will be at the expense of those already operating in that niche including Amazon's platform users.
Yeah I don’t know. It’s possible that there’s some evil stuff happening. I’m just relating my experience as a pawn employee. They parrot this incessantly.
As I said below this is something that they will talk a about like every freaking day. They talk about customer’s data as the most important thing to take care of.
Basically is preferable to get a bullet in the head than to ever reveal or tamper with customer’s data.
I cannot answer your question about who has access or not but I’m telling you what’s the culture when it comes to customer’s data.
At the end of the day I was just another IC doing menial work so probably not a good reference, but that was my experience
Capital One Financial Corp. said data from about 100 million people in the U.S. was illegally accessed after prosecutors accused a Seattle woman identified by Amazon.com Inc. as one of its former cloud service employees of breaking into the bank’s server.
While the complaint doesn’t identify the cloud provider that stored the allegedly stolen data, the charging papers mention information stored in S3, a reference to Simple Storage Service, Amazon Web Services’ popular data storage software.
My reading of this is that the ex-employee used the knowledge about EC2 instance credentials being accessible as a path to gain unauthorized access to data. In theory anyone could have exploited this vulnerability even if they had never worked for Amazon. They never say that Amazon employees had privileged credentaials that would give them unauthorized access to customer data.
There was zero inside knowledge and they were an ex employee at all times relevant to the incident.
The EC2 instance credentials via the metadata url is public documented functionality. Its how things like the SDK “just work.”
The S3 bucket policy, instance creds, and (inferred) overly permissive IAM policy is all public documented functionality. This looks like a simple case of an initial intrusion being escalated via permissive configuration and controls. There would be no story if the suspect had not been employed by AWS in the past.
Disclaimer: Im a Principal jn AWS but have no direct or inside knowledge of this incident. Everything I know or have stated here is public record (eg the indictment) or public AWS docs.
Can speak for AWS. Only the later. Basically the usage information for cloud resources. This constitutes the foundation for billing. BTW, this is be true for any cloud, any SAAS.
There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.
I used to work for AWS and had to deep dive into IAM to build a feature.
Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it.
Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it.
Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited.
Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.
I think perhaps you misunderstand the architecture of KMS. KMS master keys are used to remotely decrypt the symmetric encryption keys for encrypted data that are stored alongside the encrypted data. KMS master keys don't ever leave the KMS servers themselves, and servers can't be accessed directly by anyone. AFAIK they don't have open ports except for handling production traffic and are hardened against opening a shell. An engineer on a different team with access to a host running a customer workload could potentially run off with a temporary customer credential being used by the customer workload, which they could then use to call KMS to decrypt encryption tokens for as long as the credential lasted. But they couldn't get at the KMS key itself or retain access past the expiration of the stolen credential, and all of the aforementioned audit logs would report all of the activity of the stolen credential.
I think you misunderstand my concern. What I'm missing in the above scenario is that a resource that should be 100% under the control of the customer and nobody else can be accessed by AWS personnel to open up a door that should be closed unless the customer permits access.
What the technical implications are is moot, the process that hands out these credentials should not be accessible to anybody but the customer. It implies that AWS personnel can impersonate customer representatives or processes run on behalf of those customers. That's a serious problem.
In all the years that I've been co-locating I do not remember a single instance where a representative of the hosting facilities that I've used gained access to our data or hardware without my very explicit permission.
As for audit logs: they are only as useful as those inspecting them, and more often than not are entirely passive until required for evidentiary purposes.
> It implies that AWS personnel can impersonate customer representatives or processes run on behalf of those customers. That's a serious problem.
Rather than being a serious problem I think it's more on an obvious fact. AWS personnel build services that specifically exist to act on the customer's behalf with delegated credentials. Any time you configure a managed service to run with an IAM role, that service assumes the role and acts with the credentials granted to the role. AWS personnel have access for emergencies to the systems running their services, and by their very nature those services are in possession of customer credential sets for the IAM roles that the service is configured to use.
For example, a Lambda Function can be configured to run with a particular role. When the Lambda service goes to run the function, it fetches the role credentials from IAM and makes them available to the running Function. It could not be otherwise, because the purpose of a managed service like Lambda is to carry out actions on behalf of the customer. The role's credential set is as much a piece of data as the code of the function to be executed.
But leaving all of this aside, of course AWS personnel can access any and all data you store in their systems. They are legally obligated to turn whatever you have stored over to the courts in response to a warrant. So not only could they gather up your data by this roundabout method of misappropriating credential sets, they must have a way to simply access all of the data directly in a way that doesn't appear in audit trails. I assume for simplicity that the IAM service simply has an endpoint accessible to the company's lawyers that will serve up forged customer credentials on demand.
I believe youre misunderstanding how KMS works and is exposed. You probably want to look at the concept of “kms grants.” Thoese regulate which principals, including service principals, can use CMK materials. The customer controls those grants. There are also substantial public docs, and more available on request, around the implementation, certification, and compliance of KMS infrastructure. If KMS is insufficient for your needs CloudHSM is availble for something even closer to “hosted HSM” than “key service.”
In short IAM controls everything, there is no “back door” or universal admin access, and KMS is used to perform sensitive operations NOT handing secrets to arbitrary (internal or external) consumers.
some1 with the right access to the kms service could change a key policy to allow access to a bad guy. in theory. bcuz some1 has to have access to key policies since customers lock themselves out of their keys all the time.
but no 1 can export the private key itself. and key policy changes are vry heavily audited by aws (and can be by the customer, too). this is all proven by the 3rd party audits aws receives
Yes, they can. However, that will leave their trails in their KMS service CloudTrail - unless they manage to exploit CloudTrail as well. That's a lot of barrier to bypass, especially because accessing all these services require you to be in the correct permission group with a hardware MFA token.
Somebody can access the key hardware but they can't extract the actual key out of that. However, I've never met anyone with that level of access - and AFAIK you have to go through various security clearance and approval before such human intervention is permitted.
There's no such thing as perfect security - but KMS is as solid as I can see with centralized key management at the moment. And customer can roll out their own key server as well that is managed in your own data center.
Plus, if there is any legitimate concern about AWS having access to KMS keys (at this point it would be that they own the servers, and that's about it), you can roll a CloudHSM and import your own keys.
KMS is very clear about it's usage and what it involves. It's obvious that with Symmetrical Encryption AWS obviously needs to know the other end of the key at some point so that it can decrypt the data.
However, as customers can't even export these keys and the whole system is based on using KMS to actually perform the decrypt operations it is a non-starter. It's a lot more secure than most infrastructure which probably encrypts locally but is stored in a broom cupboard with a $10 lock.
> It's obvious that with Symmetrical Encryption AWS obviously needs to know the other end of the key at some point so that it can decrypt the data.
Its worth noting that even symmetric keys dont imply direct access to the secret itself. You can instead use the highly controlled secret material to derive less sensitive material. For example a hash derived from a known input + the secret. A third party can use this to prove that two other parties both have/had access to the shared secret. But the third party never needs to access the secret itself.
I can tell you generally how this works in Azure, I can't speak for AWS, but unless a customer is using BYOK for encryption of their data, I can't imagine how AWS c o u l d n ' t be capable of accessing data, and even then I wouldn't gurantee they couldn't still get your data. In Azure (as of a couple years ago), in order to access a customer's tenant it required VP approval, the support engineer was granted access for a specific amount of time, and typically only to specific services, all with the customers knowledge beforehand. It may have changed since the last time I had to go through this process and was restricted to blue badge employees. I have worked support cases since then and the support engineer would not even do a log me in/WebEx, etc session as they said they were not allowed to see the portal. But it may have been that they were not a blue badge and/or bcuz the customer was a critical infrastructure customer.
In order for AWS to comply with LEO's they must have some way of accessing data, that is NOT to say they do this for business purposes.
At the end of the day there's obviously nothing other than remotely storing your keys that will keep your data opaque. Even supposing that the IAM team doesn't have a way to forge a valid credential if they need to, the confirm/deny response of their service to authorization checks is the source-of-truth for whether a credential is valid, and they could update their service endpoint to affirm bad credentials if they wanted to. Presumably for law enforcement purposes they have a way to forge a credential that doesn't show up in audit logs.
Other than the data each service actually retains themselves (i.e. the Lambda service themselves store your Lambda Functions because they need to execute them) customer data is generally stored encrypted at rest with KMS keys belonging to the customer (or sometimes managed by the storage team). It wouldn't be possible to peer into unencrypted data without persuading the KMS API to authenticate your access to the key. Presumably this capability exists, because otherwise Amazon wouldn't be able to honor warrants for customer data, but the premise that KMS is handing out decryption tokens for customer data for the benefit of Amazon Retail's business analysts is pretty silly.
And of course, you're always vulnerable to someone with access to the physical host of an EC2 instance where your workload is running. Only GCP AFAIK offers an encrypted-in-processing compute service, and it's like a week old.
Given how granular AWS billing data is, I would expect the odds to be fairly good that it alone is sufficient to make a good analysis for which third-party offerings are compelling markets. Then AWS takes their execution advantage, along with things like the lower friction that arises from first-party integration with IAM and billing, as well as not having to pay retail for the cloud resources, and it becomes very difficult to retain a moat unless you have a paradigm or perspective that is both critical to succeeding and is also incompatible with AWS culture.
aggregated api usage stats, api client headers is often enough to identify competitor products and their traction, and is non-sensitive, coupled with account id to customers.
Considering that OP created this account today and that they're admitting to what would be a felony and against Amazon's own privacy policy, I doubt this statement is true.
Even if the customer had a misconfigured S3 bucket that was exposed to the public, it would still constitute as accessing customer data you're not meant to see.
As other users have provided insight on, everything you do as an Amazon employee basically leaves a trail with your employee ID, even if you had access to private information (which you wouldn't basically because it's locked behind several layers of security). Fireable and sueable offense which Amazon would definitely not allow, let alone endorse.
> everything you do as an Amazon employee basically leaves a trail with your employee ID
That might be true in retail, but it wasn't anywhere close to true in AWS. When I left most engineers still had SSH access to the production hosts (and a not-insignificant portion of operations relied on that fact).
Definitely not defending parent here, but in this day in age many people create burner accounts specifically to avoid tying any statements back to them. It’s pretty acceptable practice to create burner accounts on HN. That said, I agree, I doubt any of these claims are true.
This frankly doesn't match my experience and I have to say I find it unlikely.
Before going into our AWS production S3 buckets, looking at our databases for customer lists AWS seems to be pretty careful to get an OK.
Now we are being told that production customer data was normal to trawl? How in the HELL are they passing all their certs with all production data so wide open. I do customer managed keys - I mean, this is a HUGE backdoor.
Either Amazon is lying about AWS security (and has fooled a bunch of others) or routinely trawling AWS customer production workloads for data is a false statement.
My understanding is that Customer Managed CMK in KMS only means that the customer has control over the key operations - like rotation, key policies, IAM policies, etc. AWS still has actual control over the KMS system and full access to the HSM.
Even under this definition how in the HELL are they "routinely" trawling our production data secured by these keys. I mean, does not one think that is rediculous?
This isn't amazon billing data etc (obviously I expect they analyze that carefully given they bring in billions from billing). To ROUTINELY go through AWS customer production datasets is beyond all reason.
I would assume the comment you're replying to means things like resource usage patterns and costs to estimate a client's profits for example. Rather than reading actual data from S3 or a database.
I want to be careful here, as I respect that you worked at AWS (that is, most likely), while I never have, and don't know what goes inside the company.
But it would be helpful if you broke that down a little more than 'trawling customer data', because at the most innocuous, if they're just looking at what's publicly selling on Amazon, what goes into sales rank, that seems acceptable, to me anyway.
I think there's a difference there, though. Retail sales and reselling are parts of what most people broadly consider the "same industry". I mean, a small seller making a deal with Amazon to resell something that they know Amazon could sell on its own is at least always aware of the competition.
In this case, tech investing and online retailing are not the same industry. Amazon is using a dominance in one to fund the other, which then it uses to either drive valuations of potential competitors down or to simply outcompete them.
And that's a plausible antitrust problem.
I'm normally not in the Amazon haters camp. Most of the time I'll defend them against the typical charges of unfair competition. Not this time. This is sketchy.
Hi former-aws: I'm one of the reporters and would like to hear more about your experience. Mind sending me an email at cara.lombardo@wsj.com so we can connect?
caralombardo: Please don't ask people to admit to felonies over email. That goes double for any FAANG employee; their employers have many options to surveil them. Your employer has a page listing better options
"perfectly normal to trawl production customer data"
It's not. And there are plenty of trainings inside of Amazon to make you aware of that. It is your fault, in the end, to not report your team. I have been on several teams at Amazon and this would always be an absolute no-go. It's already difficult to even get basic ideas about customer data, things that you would consider "essential" to improving the customer experience.
Conspiracy requires two elements: an agreement to commit a crime, and an act in furtherance of said crime. There is nothing unlawful about looking the other way. You might be a scumbag, but that's a different problem.
The elements of criminal accessory require one to harbor, conceal, or act in such a way as to help someone avoid or escape arrest or punishment (CA law here, other states may be different). Again, merely "looking the other way" is not an act. Otherwise, anyone who merely witnessed a crime could be charged with criminal accessory.
That said, corporate policy might be quite different. If I look the other way while a colleague violates customer security policies (and I'm aware of such violation), I can justifiably be fired.
*Not giving legal advice, seek licensed counsel in your jurisdiciton.
As it happens, the Congresswoman who represents the part of Seattle that contains Amazon is on the House Judiciary committee, and may also very well be your member of Congress. Seems like something her office would probably want to know about if you could substantiate the claim.
Neither "traction of products hosted on its platform" nor "customer list of those hosted products" are typically public information. They are information to which a trusted vendor might have access. There seems to be a fine line between trusting Amazon to sell and ship one's products and services without using its position to sell competing products and services, and trusting AWS to host one's confidential data without reading that data...
For web and mobile app stuff this type of competitive intelligence is very available (builtwith, datanyze, etc). Also, startups never shut up about who their clients are, social proof to land more deals.
You're right - but this isn't the first time we've heard about this exact practice. It's been reported on extensively, so if anyone was going to investigate something illegal, it would have happened.
It is however, good ground for an Anti-Trust case. Using your position as a market maker to push your own products is literally illegal anti-competitive behavior and can trigger a court order to break up the company.
That argument seems to prove too much? It's sort of an Efficient Market Hypothesis for government regulation, and it would apply just as much to e.g. FTC and DoJ with respect to anti-trust violations as to Congress (as thread parent would like) or DoJ or whomever with respect to fraud or illegal wiretapping. Maybe it would be better investigated by a class-action plaintiffs' attorney, but even the mightiest firms might hesitate to wage the discovery battle that would be required against such deep pockets.
I believe anti trust is triggered when it negatively affects consumers, with Amazon’s aggressive competition consumers usually win. At least short term...
>It definitely feels scummy, but it didn't sound like GP had access to evidence of a crime.
Violating Anti-trust statues isn't criminal...but it is still illegal. Anti-trust violations also aren't the only potential laws this would violate. It sounds like it would violate unfair trade practices as well (most states has statues/laws/codes on point).
It's not just anti-trust, it's also trade secret laws. A customer of AWS has a reasonable expectation that the information it keeps on AWS's VMs are confidential.
Isn't it more like the mall owner opening a clone of your store right next to yours while charging themselves no rent in order to gain an advantage, all the while promoting their own store they opened to steal your business on the ad boards situated around the mall?
One can violate antitrust laws without being a monopoly. Certain parts of the law (regarding collusion on price setting, for example) can be broken by very small businesses.
This is true, but those are all cases in which putative competitors collude to essentially form a cartel. Which is a distinct category of antitrust offenses from anti-competitive behavior.
As others note you don't need to be a monopoly to violate anti-trust laws. However, as it relates to being defined as a monopoly this ability to leverage your market position to stifle competition is the exact type of behavior that would support a finding of monopoly...most non-monopolies can't leverage their market position to unfairly compete
A de facto monopoly doesn't mean that other options don't exist. Microsoft had a monopoly on the PC operating system market, despite other options existing.
When I was at Google, we were encouraged by our lawyers not to worry about patents or unique parts of any product. If there ever will be a claim, they will drown the company in legal fees, so nobody is going to dare to sue us.
Patents were used, in many cases, as a form of research into a new area.
Not my google experience. They do say not worrying about patents, but that's because searching for patents could indeed make you liable as you were influenced by prior art.
Nobody at google even remotely mentioned "we will drown them in legal fees".
If anything, I have a huge respect for google legal.
Lots of anon accounts (reasonably) in this thread, so I want to back up as a non-anon former Googler that your experience matches mine. It wasn't "ignore patents", it was "don't look up patents so you aren't influenced by them".
I worked at Medtronic and currently work at Qualcomm. Both companies had policies matching this. Don't search patents so you are not influenced by them.
Since when did that matter in patent law? Patents are public domain, and ignorance of a patent is not a defense against having infringed. Since at least 2012, the US has had a first-to-file policy instead of first-to-invent.
There's no legal reason to worry about being influenced by a patent. The only concern might be boxing your creativity where you can't think of alternative solutions to a problem once you've seen one solution. That doesn't seem like a strong enough reason for a blanket policy.
What I was told is that if you research the patent and aware of its existence then you may be guilty of willful enfringement with treble the normal penalties:
More precisely, a standing policy that researching patents is forbidden is prima facie evidence that your employees couldn't have possibly known about an existing patent. That means that a plaintiff suing for willful infringement will need to find evidence that someone went out of their way to ignore the policy. That might be quite difficult.
(Of course: not a lawyer, this is not legal advice)
Wow, I'm amazed that this would hold up. That's like saying that if I drive with a blindfold on, I couldn't possibly have willfully caused an accident because, as a matter of policy, I couldn't have been aware of the other cars on the road.
My previous company was acquired by Google and I totally agree with assessment. Immense respect for Google's investing, corp dev and legal arms in as much as I interacted with them. They always treated us fairly and were ethical in their interactions.
Unfortunately the way patent law works now, make patents usually not work unless someone is ignoring the law.
Patents were created to give a reason for people to publish their "secret sauce" in a public manner, so anyone could read and copy them or create new products based on the patent.
If you DON'T want your product copied, the correct course of action instead is make it secret, for example this is what Coca-Cola does (they rarely, if ever, patent their products, and they hide the best they can their recipes and processes)
Amazon does it also with popular independent sellers. I know a motorcycle shop that was selling top quality products. Amazon representative contacted him if he would like to sell on Amazon, showed him offers how to stock and sell items. Amazon started a brand with the same name and their products were higher in search, were poor quality China made, jackets that were falling apart within a year. For the same price! Angry consumers targeted their anger to the real website leaving negative comments, not on Amazon! Shop owner had to change his brand after 18 years of being in business, as legal battle against Amazon would cost him more than the business had in stock.
There's a House of Reps. hearing on Online Platforms and Market Power next Monday with Bezos attending. If anyone has some staffer friends, could be a good line of questioning to poke them about.
I joined after the team had gotten traction already. Both the GM and senior most product person on the team told me about their tactics independently.
To be honest, I didn't think of it as anything sinister at that time. AWS had such high octane culture to move fast and innovate that I actually felt what they had done was quite smart. It was a super competitive culture and people did whatever was needed to build new things. On a day to day basis the only pressure was to build... I don't remember instances where ethical guidelines were brought up. So, in a way, the outcomes were a result of what people were rewarded on.
Only after I left AWS I started thinking it was ethically iffy. I still believe Amazon is an amazing company and my time at AWS was one of the best learning experiences.
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair.
I wish we went into this in much more detail in high school when covering economics and ethics (if the school even bothers to teach ethics). It should be a prerequisite in any capitalistic economy (but not only those, it can easily be extended to other things).
I've also worked in industries that I think don't operate very ethically. It's amazing what you can ignore as an outlier because the alternative is uncomfortable or means you have to make a large personal change.
Well, yeah. Or just having to look for a new job that may or may not pay as much. But I wasn't really going that far as saying people (myself, at one point, if you notice what I wrote) staying at a company they feel is acting unethically, but actually just noticing and accepting the company as doing unethical things instead of attributing it to an outlying situation that isn't indicative of how things are normally done.
Companies and people sometimes do shitty things. It isn't always on purpose (misunderstandings, one bad person, etc), and there isn't always a good way to fix it afterwards. I don't condemn people and companies because of this, and there's a tendency to assume this when you see something and work at the company. It can take a while before you start seeing a pattern and accept that it might just be how things are done sometimes and the management is fine with it. If you don't have a lot of options, I think there's a tendency for people to not look closer either on purpose or subconsciously because they might not like what they find, and then they've put themselves in a harder situation, where they must choose between what they believe is right and a hardship.
Sometimes ignorance is bliss, and the human mind is very complex. That's all I'm saying.
> "An Amazon spokesman said the company doesn’t use confidential information that companies share with it to build competing products"
The above statement may be "true" if you redefine what is confidential. The Amazon MNDA in past years basically said that they could use any information they remembered from the meeting. I read non-disclosures carefully. I've never seen anything like it.
This is called a residuals clause, and it’s increasingly common. Be really careful looking for these - I won’t sign a vague/broad one, unless I am out of options. (e.g., acquisition or fail)
Ah, so that has a name? It was in the middle of the document in a fat paragraph. I was delighted to find it--kind of like picking up a big seashell on a crowded beach.
We ended up signing it, but I went back and forth with their counsel to neuter this clause so that it was significantly safer:
Notwithstanding anything to the contrary contained in this Agreement, Recipient may use Residual Knowledge, subject to
Provider’s valid patents, copyrights[, trade secrets], and mask work rights. [For the avoidance of doubt, no license is granted to the Recipient for any of Provider’s Confidential Information, patents, copyrights, trade secrets, or mask work rights.] "Residual Knowledge" means any information that is retained in the unaided memories of Recipient's Representatives who have had access to Confidential Information of Provider[, without specific or intentional memorization or reference to any written or electronic information or documentation. Notwithstanding the foregoing, Residual Knowledge may only be used for internal purposes by Recipient, and Recipient may not disclose Provider’s Confidential Information to third parties under any circumstance except as outlined elsewhere in this Agreement.]
The parts in [ ] were added by me. We tried to neuter the clause as best we could; they really wanted to have one in there, for whatever reason, so my focus was on neutering it rather than arguing to remove it. There are always other concessions in a negotiation from the other side. :)
Just ... wow. This is an egregious abuse of monopoly power and is exactly the kind of thing that antitrust laws are supposed to address.
I was certainly naive when I heard about other big retailers who would refuse to allow any subcontractors to use AWS. "Surely Amazon has a Chinese wall" to prevent that kind of data sharing, I thought. Never underestimate the lack of morals in business is the right answer I guess.
> "Surely Amazon has a Chinese wall" to prevent that kind of data sharing, I thought. Never underestimate the lack of morals in business is the right answer I guess.
It’s remarkable to me how many competent programmers with years or decades experience in this industry don’t understand —- If you’re using AWS, Amazon has access to ALL of the data you put on AWS.
Not that they 'can' or 'want to', given the current state of technology they absolutely have to have access to all your data for AWS to function.
There isn’t currently a feasible technical way to work around this. And to head off all the ‘but FHE’ comments, see the ‘currently feasible’ above.
I'm not talking about not having any access in the technical sense. I'm talking about a "Chinese wall" whereby people who work for AWS supporting customers should absolutely not be able to inform any of the teams that build new Amazon services. These types of Chinese walls exist in many different industries, perhaps most famously finance, and when these walls have been "breached" in the past it has resulted in huge scandals.
I think your understanding is true, unless the claimant elaborate what those data is and how his team got it, I do not understand how it would have worked.
Access records for public services have a very detailed iam audit trail that logs people who accessed what at what time, and service teams don't get to just jump around that. Maybe they can see some metadata but certainly not actual data in an S3 bucket somewhere.
I think enclaves are a more practical near-term solution for data privacy, but they don't prevent Amazon from identifying successful businesses based on e.g. resource usage growth.
I don’t think the ‘enclaves’ concept addresses the root of the issue I was getting at, which is for there to be useful computation done on the data it must be unencrypted.
Even with ‘enclaves’, from what admittedly little I know about them, you still have to have the key to decrypt things on the machine somewhere, which means whoever is running that machine for you has access to your unencrypted data, and we’re back where we started.
Amazon does not access private S3/Ec3 data for retail competitive purposes.
The comments above indicating 'well someone has access' - yea, obviously, it's data hosting. Someone has access.
But the amount of conspiracy here is frustrating.
Amazon will play very aggressively within the bounds of the law, meaning, if they can glean public info about something, or look at their own sales data for a product, they will do that.
But to look at s3 data would risk the entire empire.
It's rational for people to be a bit skeptical, and so Walmart can say 'no data on AWS' but it's also an easy thing to do.
Now - is it possible that new retail PM, who used to be an AWS PM, and who for some reason still had access to things he shouldn't - went ahead and did that? That could happen. And maybe his boss finds out and looks the other way but calls IT and tries to have the loophole closed quietly. Etc.
As a policy are they trying to copy your product and even ask you for information and aggressively pursue customer data? Yes.
As a policy are they looking at your S3/ec2 data - no.
As a company they do, and that's how policy is set.
But individual actors are individual actors, in a company of 100 000 people, some will go astray.
They are pushing their 'white label' stuff agressively, I have no doubt the PM's have zero qualms about using Amazon.com sales data to their advantage.
But I also submit that retail PM's actually getting access to private S3/EC2 is totally rubbish, at least by any policy or scale.
They could be sued for billions in each case of that breach, and the resulting PR fallout would be impossible.
Imagine you are the VP of AWS - you make all the profit for Amazon.
Are you going to somehow allow some dirty Retail PM access to your customers data?
When your customer finds out, and tells the world, and it gets in the press, what happens?
If your ABC startup had evidence that Amazon was creeping on your data as policy, you'd have to dump them instantly.
They could say goodbye to every government contract.
If you are Bezos - would you risk the entire Brand and the cash-cow to move some low-margin pair of shoes and USB hub?
So no, I think the firewall between AWS and Retail is systematically legit.
I was on a business call with someone from AWS on a different topic, and it was pretty darn clear they opened up some sort of Account page that discussed our (limited) AWS usage, and were trying to infer a bunch about our business from that. It doesn't even really matter how deep that data goes - even just month-over-month billing #'s or something like compute/bandwidth consumption is super telling.
We mostly only do CI type stuff there, so that didn't work so well for them, but if most of our revenue & operational use was through AWS, you bet I'd be worried about what they could infer.
Shopify is way way worse than Amazon. If you think Amazon is evil, Shopify is 10 steps ahead.
It's not just my experience. Talking to startups and warehouses in Canada, the stories are all about how Shopify invites for friendly talks and then stonewalls you once they have got the required information
Wow. I understand you do this as a throwaway but if true this is very bad stuff and it would be nice to have a lot more substantiation so that it could be verified.
I wish I could reveal team and product name... but that would be a career suicide. I'm not asking you to believe what I'm saying... but I truly am sharing my experience. I'd encourage you to talk to folks you know from AWS who were there for last 8-10 years.
What I did not find from your post is in what manner are the data accessed. Is it at all publicly available? Is it metadata e.g. usage/billing? or is it production content like S3/lambda code/EC2 storage? It would be very helpful if you can clarify what kind of access it is.
Funny... I have software that combines well with something that has an online app store. They've been begging me to put my stuff in there. Nothing doing, I seen how you guys have embraced and extinguished others. They just took out their biggest app on the app store with their own version.
This doesn't matter if you don't have the millions set aside to defend yourself in court against a giant conglomerate. Giant companies breaking NDAs with scrappy startups is a story I have heard often.
If you blatantly ignore and NDA, and then make a lot of money from it, then the 'small startup' will have a ton of money because the prize is huge, i.e. a % cut to lawyers who can work pro-bono.
Imagine you have a $10B company and some bonehead PM steals info from some small startup, for some stupid small project - it puts everything at risk.
In most case, I think you have boneheaded actors, usually not acting in the best interest of the company.
This is likely disingenuous. Large corporations like Amazon systematically refuse to sign NDAs with small players, hence none of the info is “confidential”. The rationale is that large companies might have people working on the idea already and they meet so many people/companies it would be restrictive for them to agree to any confidentiality.
It is a reasonable expectation that your data is kept confidential. If my hosting provider were to so much as look at my data without my explicit permission I'd sue.
Hi throwaway_aws: I'm one of the reporters and would like to hear more about your experience. Mind sending me an email at cara.lombardo@wsj.com so we can connect?
A single comment from a throwaway account with no evidence but confirms my personal bias that Amazon is evil? I'm fully bought in before I catch myself.
The economic and reputation cost Amazon would take in ever accessing customer data to come up with some competing B-list product (say ElasticSearch as a managed service) is astronomical compared to potential profits. One thing I know about that company... they care about optimizing profit and are long term focused.
Please provide evidence for your extraordinary claim.
Whenever anyone asks for evidence I start to wonder why they need the proof. Why did you need this link? Do you have a business relationship with Amazon?
The customer data on Amazon Retail is Amazon's, not the seller's , just like the customer data when you buy shampoo from Walmart is Walmart's, not Procter&Gamble's
I'd like to ask for evidence for statement 2. This sounds more like your opinion than a substantiated claim.
1. Someone needs to pay the people who write stories. If proof is important you should not object to contributing to the people who work on your behalf.
I presume it would be a lack of history doing the exact thing amazon are accused of doing here? That and the fact that they aren’t doing an e-commerce business in the way Amazon is.
What a wonderful verb; thanks for relieving my ignorance. ISTM Amazon misses out on some aspects of the idiomatic definition, since they failed to popularize their own brand of client machines with an OS they control.
I think the original Netscape folks would disagree with your assessment of M$ sherlocking competence.
There's no way to really know how true this is but it certainly feels true if you're on the startup side - however most startups just aren't being realistic with themselves, and thinking they are special.
For example, my previous company brought our 6DOF MonoSLAM SDK, 3D model processor and OpenGL viewer to Amazon from 2014-2017 pitching the "AR View" functionality that they eventually put in 2017 [1].
Was that a result of us coming and pitching it? Probably not because that use case and stack wasn't a novel concept even back to 2010. So the concept and stack was certainly there for them to do on their own.
What we DID provide to Amazon however was a significant data point (based on our user velocity/interaction metrics and the rate of increase of 3D model generation from retailers) about whether the market was ready for that feature - and so they said ok it's probably time to do this. It just so happened that the cost of implementing it crashed to basically "trivial" in 2017 with the introduction of ARKit and so it was a no-brainer for them to roll out for a few years.
What's the takeaway? These big companies aren't dumb, your idea isn't that novel and they probably have the team and technology to do it better than you for cheaper.
> These big companies aren't dumb, your idea isn't that novel and they probably have the team and technology to do it better than you for cheaper.
This is only partially true. The thing is that even if your idea _is_ novel, giants like AWS can launch a similar product after seeing yours and thats a problem, imo.
I dunno if they can necessarily do a better or cheaper job. A lot of the engineers at these megacorps are super burned out 9-to-5ers who don't particularly care about what they're doing, since it won't have that big an impact on their lives one way or the other if it succeeds. ("Woo, 20% bonus this year" vs "Team's being reorged, gotta hit up leetcode and find another corporate job")
Why even pitch the idea to big companies then? Seems like a no brainer to just avoid discussing technology with anyone on the M&A team of any large company with an development team, unless you went to the same college as their executive staff.
Amazon announced a copycat of us pretty quickly after we went through YC. I suspect it’s not some sinister top-down thing (unless they are actually reading all YC company descriptions). Most likely, the copycats are ambitious, unoriginal PMs pitching some budget.
We facilitate subscriptions using a smart scale, and it works way better than a Dash button (Bottomless.com, YC W19). I’m actually surprised they haven’t launched yet and are taking so long.
It’s pretty wild, the hardware on their launch page is exactly like ours, only they're so slow that it’s a copy of two versions ago and hasn’t even hit the market.
I was on the team that was peripherally involved in building the Amazon Dash collection of products. The scale idea has been floating around within the company since 2015/2016. It takes long to do this at scale and a cost-effective point.
There were also other similar products that were scrapped because there was no way we could sell them at a reasonable price.
> "The scale idea has been floating around within the company since 2015/2016."
So much this.
As a former startup founder now acquired into a Top 10 valley tech company, few people understand just how many nascent projects, features and ideas a huge, successful tech company has in development. At least for our active product domains, I never saw a feature from a competitive product that we didn't already have on a list and usually in development somewhere.
When I first started, I would sometimes see a "neato" new feature publicly demoed in some competing product and bring it to the relevant PM's attention, only to be sent back a two-year-old internal video of a similar feature already working and either queued for shipping, dropped in market testing due to weak response or deferred to the "next version" queue due to resource prioritization.
Of course they’re reading the new startup descriptions. That’s the sort of market research due diligence that’s expected of any PM.
They fly in startups for presentations to get a glimpse into their inner workings, you think they aren’t taking the time to read about what everyone’s doing first?
It might not be hitting market due to lack of interest. The way I "solve" this today is to just send myself an email when I think I need to order something, then order it next time I check my email... works pretty good.
There's nothing sinister about a copycat. It just feels that way selfishly for the original creator. Capitalism works best in the long run with near perfect competition, copy cats are great for everyone except the first guy with the idea.
Hey it's me pyrrhotech, and upon some additional reflection decided that I was wrong and maybe there are some issues with copycats. Just wasn't thinking straight before.
> Capitalism works best in the long run with near perfect competition
Unfortunately someone like Amazon using their size and market power to produce copycats is the opposite of perfect competition since they can operate at a loss or at cost and starve out any newcomers.
I was a senior manager on the very first product the article talks about. I was closely involved in designing the service and presenting it to Amazon senior leadership. WSJ quotes the CEO of a startup called DefinedCrowd as accusing us of stealing their ideas from a meeting 4 years earlier.
What a bunch of conceit. I don't remember our team discussing DefinedCrowd even once. We focused on the many other more interesting players that are doing the same thing, and researching them by trying out their service etc. like anyone normally would.
I'm sure someone talked with DefinedCrowd 4 years before that. Amazon, like all other tech companies, routinely has NDA conversations with startups that never go anywhere.
I can't speak to the rest of the article, but the very first example is totally false. WSJ is looking for an angle, and this startup is probably looking for a way to blame Amazon for their own execution problems.
DefinedCrowd is pretty successful actually, just so you know this isn't sour grapes. And it isn't about a meeting 4 years ago, the very first line of the article says that Amazon was an investor (at seed stage, probably a few hundred K).
So the problem isn't that someone heard about DefinedCrowd and decided years later to make something like it. Amazon made a significant investment in an early player in this space, and then started building a direct competitor while still presumably having both access and influence over that company. Doesn't seem responsible or ethical to me.
Yeah, given the resources, I'd want to pursue all the possibilities in a promising product/market in parallel. They may have even had working prototypes for some time or even be near launch and they are just feeling out whether the startup would be competitive and/or complementary. Doubt it's anything evil. Just sounds like business minded people at work.
you're always violating some patents by IBM and the like, sometimes they don't want to compete with a product only to get you off the table... and sometimes they can do that easily
Oh, and there is the enterprise variation (option D?): talk with the costumers and demand that they stop working with the competition
I worked on the Echo Show team. The product had been in development for over a year when we invested in Nucleus. I remember thinking it was very strange that Amazon was investing in that company when we were building such a similar product internally.
I've definitely seen the same thing happen – an org that isn't sure it can build function X well might invest in a startup building X just in case.
Of course, they may or may not take advantage of that situation by misusing confidential information.
Either way the startup could lose (or just not have much negotiating power).
So yeah, sounds like this may not be a case of AMZN misbehaving. But I'm still not sure I'd want to talk to them if I were a startup, at least until I know they really need us and are willing to pay a lot.
Nothing I saw indicated that the Echo Show was in any way influenced by Nucleus.
I suspect, but don't know directly, that this investment was more about trying to jumpstart an ecosystem of third-party Echo devices rather than hedging bets.
"In 2016, a group of investors led by the Alexa Fund bought a stake in Nucleus, a small company that made a home-video communication device that integrated with the Alexa voice assistant.
Nucleus’s founders and the venture-capital funds investing alongside the Alexa Fund had reservations about collaborating with an Amazon-backed firm, according to some of the co-investors. "
"After striking the deal, the Alexa Fund got access to Nucleus’s financials, strategic plans and other proprietary information, these people said. Eight months later, Amazon announced its Echo Show device, an Alexa-enabled video-chat device that did many of the same things as Nucleus’s product.
Nucleus’s founders and other investors were furious. One of the founders held a conference call with some investors to seek advice. He said there was no way his small company could compete against Amazon in the consumer space, according to people on the phone call, and began brainstorming ways to pivot his company’s product.
An Amazon spokeswoman said that the Alexa Fund told Nucleus about its plans for an Echo with a screen before taking a stake in the company. Several people on the Nucleus side of the deal disputed that.
Before Amazon introduced its product, the Nucleus device was sold at major retailers such as Home Depot, Lowe’s and Best Buy. Once the Echo began selling, those sales declined sharply and retailers stopped placing orders, said two people involved in the deal.
Nucleus threatened to sue Amazon, which settled with Nucleus for $5 million without admitting wrongdoing, according to people familiar with the settlement. Both sides agreed not to discuss the matter.
Nucleus reoriented its product to the health-care market, where it has struggled to gain traction, some of those people said."
If I’m reading commentary correctly, Amazon would invest in other companies using its Alexa fund in order to gain access to their data and then actively operate in a way that undermines the success of the company. Sounds like they’re essentially enjoying most of the benefits of purchasing a company but for a fraction of the price. Rather than spend their own capital to do the hard legwork of building and validating a product idea, they’re effectively spending others’ and then swooping in for the bait and switch kill.
If I were a founder or board member I’d be super skeptical about ever taking money from Amazon in light of this news.
If this isn’t illegal it at least seems wildly unethical. If it’s neither of those and considered an acceptable tactic, then perhaps companies are generally undervaluing themselves otherwise it wouldn't be financially feasible?
This is a bit complicated. I’m not in Amazon but work for a (very large) org. Often we will have some product in mind and approach/maybe even partner with a vendor while at the same time having an internal effort. Occasionally we go with an internally developed solution, sometimes we go with vendor solution. I am sure some vendors feel bad that suddenly they are forced out by a “similar” internal product. However, in every case I’ve been involved it wasn’t sinister. There was a business need and the company actively invested into multiple parallel solutions, eventually an internal solution won after we used a vendor solution for some years(but sometimes it is the vendor that wins!).
Yeah, I haven't read the article (paywall) but from the first paragraph or two it sounded like AWS invested in the company and launched a competitor 4 years later.
To me, that sounds like the startup just wasn't able to perform well enough to beat an AWS-built solution, in the eyes of an executive.
Of course, if I were doing a startup that played in AWS's space, I'd view them as a pretty formidable competitor and I wouldn't expect to get a great deal from an investment or acquisition from them, since unless I have a really special sauce they can probably build whatever we've done and sell it better.
It's shitty, IMO, and probably drives startups away from a field in which they could sell to AMZN – since they know they won't get an awesome deal. Probably reduces innovation in the field overall.
Not from Amazon but this happened to me when I had an app in the top 10 of the AppStore. This a-hole from Silicon Valley flew over to London, asked me loads of stuff, saying he wanted to buy it, offering large amounts of cash and then just launched his own version and some copies of other things I’d made. I’m still not sure if it was his plan all along or not - He did try and recruit me to his company at one point but when I refused, his parting words were “if someone offers you a seat on a spaceship, don’t ask which seat”. Really glad I didn’t take that seat.
If that is the case I don’t think it’s policed at all. You can’t copyright or patent an idea so people just do what they want - at least when you’re an indie developer.
Am I the only one who thinks this is to be expected?
Stealing good ideas is literally amazon biggest strategy in retail. They monitor best selling products, then launch their own brand and crash the competition.
Why anyone expected them to change strategy with business ideas? This strategy served them well.
If nothing else, it doesn't come as a surprise because amazon consistently operates in an anticompetitive nature. I see this as building public understanding of why amazon should be broken up.
So this didnt come across in the title - but the first company they use as an example is one Amazon actually did invest in? And that was four years before the "competing" product (which is literally just providing data for AI training as far as I can tell)? I'm not sure what I'm missing, but that reads like if coke invested in an energy drink brand and four years later launched another energy drink brand. It's basically a commodity and it's 4 years after the fact- what's so controversial here?
While I cannot comment on a startup's or Amazon's perspective on this matter, I had interactions with Amazon employees that was in fact deceptive.
Short story: Amazon representative tried to trick influencer into sending their traffic without compensation.
Long story:
At the time, I commercially represented an influencer (largest in a mainstream niche with an active targeted fanbase) who had been approached by Amazon to sell access to our online course on Amazon (as in: let customers buy a coupon code for our content platform). The paperwork we received to sign did not reflect the terms we had negotiated. It included some kind of fees etc. that had never been mentioned, basically shifting percentage in favour of Amazon using fine-print. This felt dishonest.
We decided to do it anyway due to their promises of considerable sales for our program including projections. Then the representative showed us a listing of a direct competitor and told us the number of sales this competitor was able to generate. While it's nice to be on the receiving end of such information, it's unethical. Who knows how truthful the numbers were anyway.
But then the week of deals started and nothing happened. No sales.
This could have had many causes but instead of revisiting the offer or the listing or just say "bad luck", the representative kept insisting that the influencer send their traffic to Amazon which is usually a business transaction but that wasn't part of the deal. They kept insisting anyway, even a second representative.
I'm not keen on doing business with Amazon after that encounter.
There’s a big difference between stealing actual IP and executing on someone else’s idea (even if the idea was “stolen”).
The sour grapes here seem to be coming from those who thought they “owned” having an idea but failed to fully execute on it. The winner is the one who actually does it.
Well, I have been on both side (a company that wants to be acquired, and a large buyer), and yes, that's the discussion you have. It's normal and expected by all parties.
When you want to buy a company it's because you want to launch yourself into that field.
> It's like when somebody says they want to go birding with you, but really they just want to get you alone in the woods so they can take your binoculars.
I would imagine it's common. There was an article on here not too long ago about how the entire Amazon HQ2 fiasco came about because Bezos was jealous that Elon Musk was getting more govt subsidies than him. If one of the most successful people in the world can be so focused on petty grudges, I imagine there are plenty of others doing the same.
You should listen to Michael Jordans hall of speech to see how pretty he was. He is considered the greatest basketball player in the world and he spend the whole speech belittling people.
I remember as a kid there was a child's book about basketball players and it mentioned that MJ was an aggressive, confrontational player who trash talked the other team. It shaped a conception of him as a champion who is fiercely competitive to the point of pettiness (not unlike Bill Gates, funnily enough), not a champion who is known for noble sportsmanship.
Standard advice to to not keep your startup secret but to talk about openly to as many people as possible. Eg, quotes about how only 1/1000 startups die to competition, and the rest self destruct or fail to get traction.
I've seen people try to play this game. They'll pretend they want to buy you/partnership/invest/be a client and then ask pointed questions: "What dependencies are you using on your backend for XYZ ... have you found any issues with that? If you were going to rewrite it, how would you approach it?"
In the weeds questions about detailed implementation. "Do you have a detailed architecture diagram?" etc. People who haven't seen it are like "boy they are serious. Look at their diligence"
These people need to up their street smarts game: "Nah bro, they're just stealing shit."
How do you know? Check who's in the room/on the call. Ask yourself what kind of decisions these people would make. Find out who they are.
Amazon has been actually pretty light with this practice. They're pretty picky but the right people have been in the room and they haven't really gone deep into the spy questions. The real nasty violators are companies like uber and microsoft. I wouldn't be surprised if they had actual
corporate spy departments.
We had GM try to do it but they were bumblers. It was kinda cute. It also seems to be mostly American. I've dealt with a lot of Japanese and Chinese businesses and they were all genuine. If they said they wanted a partnership for reason X, then they indeed want a partnership for reason X.
If they want to steal the tech and go to other markets then there's clear nonexclusive and rights parts of the contract. It's not the "100% lies all the way down" of Uber.
Why are people surprised by this? There was a period of time in the VC world (~2001) where this was di-rigeur. You were caught between a rock and a hard place, as pitching literally meant potentially enabling a competitor.
Almost every established VC is a potential competitor as much as a potential backer, because they already have backed ventures in/near your market, you won't get approached by one without expertise in your market. Any of their other backed ventures might be mulling over a new strategic goal and doing indirect research on you via their sponsors. Before you say NDA, it's practically unenforceable, you can't stop a VC chatting about what they've seen in confidence to a trusted friend, the bar to prosecute breach is far too high.
You can't stop someone seeing what you offer, and it's hard to prevent competitors from seeing how successful you are, especially giving the lack of privacy in consumer space. In this article, we don't know if Amazon used any IP, we are told they just copied the offering, which anyone is free to try.
I bet the problem with DefinedCrowd is not so much they revealed too much, rather they revealed too little. How so? The VC dance is really about demonstrating to the uncertain backer-competitor that you are so good, it's not worth competing. That's a main point to inviting outside parties in. I imagine these guys were just so weak, they made competition more appealing that partnership.
The decision isn't made by a trillion dollar 'Amazon'. It's made by a person inside Amazon with a limited budget, whose ROI is critical to their personal financial success. At the end of the day, the decision to compete will be an ROI question, Is it better/cheap enough to roll-my-own, or is it has this outfit done a good enough job that it is better to partner? Outstanding people are a precious commodity, so if the startup is really good, why wouldn't you partner with them? And if mediocre when you are really good, why would you?
It's entirely possible they launched competing products because they know something everyone else doesn't. Did anyone bother to investigate why they didn't acquire these companies?
I've worked at several startups and you'll be surprised at why some corporations don't purchase a startup.
Some of them have awful cultures of psychological abuse, you walk in the door and you know something isn't right because the founder(s) have a cult-like environment that includes abusing his/her employees to the point where they are all afraid to say anything.
Some startups have founders and employees that are lying about everything and when you actually dig into their source code and infrastructure they aren't doing anything they say they are doing.
Some of them have the worst infrastructure imaginable and/or they have such poor software engineering practices that they will never be able to scale to meet the kind of demand a company like amazon has.
Just because they have a good idea and a company doesn't mean they have an implementation that is worth investing in.
VCs will often meet with startups to get market analysis done for them.
Maybe it's not the primary motivation of a meeting but information has to be presented to make the case for the business, so it gets recorded and used anyway. It can be used to look for alternative investment targets.
Pitches have inherent risks to them, that have to be taken into consideration.
Also I get that Amazon has much more resources but if they there able to copy it after one meeting... Was your project really that valuable? And would not be copied then you launched?
Good friend of mine said his company invented the book preview and had tons of publisher relationships due to their service for public libraries and preview snippets. They sat in a meeting with Amazon executives early 2000s - allegedly similar story. They were first offered a partnership then Amazon canceled all talks after getting a good grip on the business and launched their version shortly thereafter.
If your business is so fragile that all anybody needs to do to out compete you is the description of your business idea, then I'd say you don't really have a business that is long for this world. If not Amazon, someone else is going to clone and eat you from San Francisco, Seattle, Hyderabad, or Shanghai.
Ideas don't really mean much if you don't have excellent, industry leading execution.
Isn't this what Google did with Yelp? IIRC, google was in talks with yelp to buy them out. They either pulled out or lowballed them. Then a few months later they launched "Google Places"
At the time, this was referred to as a "brainfuck" of smaller companies for their IP.
I think what is missing here is an insider's perspective.
The volume of concepts that are being actively worked on, let alone conceived, is pretty incredible. I've been in Amazon meetings with startups before/during talks of investing. Usually it goes something like this:
There are multiple principle engineers involved. There are multiple engineers from potentially related projects involved. Most of the decision making boils down to:
- the obvious: would this investment likely be profitable?
- does investing/acquiring this company enable us to ship sooner/gain a competitive advantage?
Often times Amazon prefers to acquire companies not directly operating in the space Amazon needs them to, and then steering them to do what Amazon needs. Often times it's just talent acquisition.
For all we know, Amazon already had competing products in the works (and let's be real, given how slowly Amazon moves, this is the most likely scenario) and decided that this investment wouldn't be worth it.
I worked at a startup that made a VoIP product in the early 2000s. Microsoft was interested but declined after a technical review. Later they bought Skype instead. No hard feelings. But giving up too much technical detail was definitely a concern for us.
Have a colleague who said the same. They had come in for reasons I can't remember, and shortly after launched a product that worked strangely similar to their offering. People got laid off, including my colleague, not long after that.
I wonder how this will hurt general product innovation in the long wrong? Seems like it could have a trickle down effect of diminishing companies money for research and development to come up with new products that benefit consumers.
I'm a serial tech startup founder who has seen all sides of this up close. I've had startups I founded or co-founded IPO, be acquired by a Top 10 valley tech giant, acquire smaller startups, and (of course) go bust. I've self-funded, VC funded, angel funded and had corporate investors. In the acquisition, I ended up working for the Top 10 valley tech company for over a decade reporting directly to a key C-Suite exec and was deeply involved in company strategy, multi-billion dollar M&A, partnerships and investments.
In all that time I never saw any hint of anything even remotely unethical involving investments or M&A at my acquirer/employer. The company always played strictly by the rules, sometimes to an almost excessive degree (avoiding even the possible appearance of impropriety). This was both innately cultural, constantly preached from the BOD to CEO on down, as well as tactical since the reputational cost of negative perception was simply considered too high. Word gets around the valley fast. The invisible cost of the potential partner or startup that chooses to pass on meeting with us due to reputation could be very high.
That said, we would negotiate hard and do our due diligence, gathering public information in every legit way possible as we evaluated potential markets, product areas, acquisitions, partnerships and investments. Tons of startups were always contacting us to meet regarding investment, licensing, partnership, etc. So many that we'd actually meet with far less than 1 in 10, and even we'd usually just send a junior staffer to the mtg for a "first pass".
Those first mtgs were almost always set up as NNPI (No Non-Public Info). We'd actually have the startup sign a doc in advance stating that they wouldn't share anything with us that wasn't already public info. This was then re-iterated at the beginning of the mtg. Afterward the junior staffer would then circulate a brief memo on the mtg outlining if there were any areas of interest and recommending whether there should be any follow-up with actual business unit or tech people with domain expertise. Most such mtgs had no follow-up. In the cases where there was follow-up, before mtg again we'd internally specify what our possible interests were (acquisition, investment, partnership, etc) and if acquisition or investment we'd have at least a first-pass thesis on what our interest was, usually at the level of "interesting tech", "good talent", "cool product - might slot into XYZ product line."
Back when I was a fledgling startup founder on the other side of all this, it was sort of mysterious and I remember my first mtgs with Big Cos (including Google, Apple, Intel, Microsoft, etc). It was all very exciting until I realized that most of these 'first sniff' mtgs are with junior people and never go anywhere.
Even if my acquirer/employer was unique in being highly ethical, the reality is that any startup founder who isn't intensely aware of the risk of Big Co becoming a competitor is incredibly naive. If senior Big Co execs are taking time to meet with you (vs junior Big Co staffers), it's because they are interested in something. As a founder, your job is to figure out what why they are interested. Sometimes they are just making small investments to foster an ecosystem their primary business relies on. Or they might be interested in acqui-hiring your startup. Or they might be looking at moving into the emerging market you're in and doing a Build vs Buy analysis or even considering a roll-up of smaller firms. Or they might be looking at buying one of your competitors and doing market due diligence. As a Big Co exec, I'd usually just tell a startup founder point-blank what my interest was, as it tends to save everyone time.
Often the startup founders I'd talk to as a BigCo exec were actually too guarded, to the extent they'd hesitate to even informally have a 'get acquainted' drink at a conference or trade show. It's good to be cautious but at the same time, many of my most lucrative exits and deals began with such meetings. As a founder, I also often learned invaluable info from Big Co people at such informal mtgs. After all, Big Co folks tend to hear all the industry scuttle-butt and they actually subscribe to ALL those $10k market data reports us startup guys could never afford.
Bottom line: when engaging with Big Cos, ask good questions, rationally evaluate the benefits vs risks, plan for the worst and hope for the best.
Your particular experience does not mirror that of others.
The risk here isn't Big Co Exec. It's the aspiring PM who wants to make a name for themselves at the Big Co.
NDA's aren't worth the paper they are printed on; when it comes to that, you'll just discover that Big Co has retainers in place with all the law firms you'd want to work with.
Sure, that's why I said "Plan for the worst, hope for the best". It's always possible, even if mtg with an ethical company, the junior person you're meeting with that day could be a new-hire who's an idiot and tries to get actionable information.
NDAs aren't deterrents. At the same time, any secret that can be casually conveyed in a mtg, typically isn't all that valuable. If you have a strategic investor as an outside board member, a smart founder will ensure nothing disclosed in a board mtg or board materials is specific enough to be competitively actionable. If you do it correctly, they shouldn't gain any non-public info more specific than broad sales growth, and there are a lot of completely legal ways for interested Big Co competitors to get good intel on sales growth which are much cheaper and easier than investing.
That's why we generally passed on even evaluating most startup investments. They just weren't worth the time to manage plus if we really planned to be active in that space the legal exposure would require a formal "Chinese Wall" between our investment and our business unit, usually managed and audited by an outside law firm. If we were truly interested in the space, I'd always argue we should just acquire the startup now, buy one of their competitors or tilt up our own 'build it' version.
Guess what, we meet with potential competitors in our market and sometimes even contract with them to provide services on our behalf, and use that to gauge the market and current solutions!
You as as a startup should be fully aware that a big player can enter the space at any time they want. This is not Amazon's problem. It's your problem.
Also, it's a flawed abstraction to view large companies as a monolith: the investing arms are usually very separated from product teams.
TLDR: If a large company wants to enter a space, that's their choice. Accept this as a fact of life if you want to go work on a startup.
For the longest time (more than a decade) I ran a small company (< 50 employees) that fielded a video chat service. The whole thing revolved around a minor bug that just happened to be present in all major browsers.
All that time I thought: next week someone else is going to clue in to this, it is so obvious. But nobody ever did.
Isn't this more to do with how TCP and the socket interface is implemented (or more accurately its intended semantics) than a bug in browsers? Not draining any inbound data on a new connection prior to sending my request doesn't sound like a bug.
Yes, that's the practical reason. But a browser could of course have implemented HTTP in the way described in the RFC, instead they all took the easy and obvious way out. That's also why the bug was present in all browsers that had marketshare, it was the obvious thing to do, it is not as if they independently made the same mistake, they independently decided that it wasn't important enough to implement the protocol to the letter.
Not as sinister as it seems. You could also not be approached for investment, launch your product, then get killed by a competing product from a big company, all the same.
But you would hypothetically be accounting for that outcome in your strategy. If I'm understanding the commentary correctly, Amazon would regularly invest in companies in order to gain access to their data and technology while another team used the info to undercut and kill their product. You don’t generally assume your investment partner is actively planning your demise by pretending to support you while siphoning off your life blood.
I agree though that it’s not as sinister as it initially seems. Companies create competition at times to stimulate product development even though it’s a shitty operating mode for the ground soldiers involved. In that case though they’re spending entirely of their own resources and not leeching off other investments to actively undermine a successful exit.
Perhaps startups are undervaluing their IP if this behavior is able to manifest easily. And in light of this news I’d be super hesitant as a board to allow any money from Amazon whatsoever.
It's different when the companies are being baited with investments to spill their secrets and visions to you, and then you take the idea and throw your infinite power behind it without them.
Let’s be honest tho, what did they expect? In this case, Amazon gave them funding and a 4-year head start. The point of Alexa fund was to use Amazon money, and Alexa as a launch vehicle, to grow a product. Obviously Amazon had to gain something back.
Maybe...but in the past, AWS proactively looked at traction of products hosted on its platform, built competing products, and then scraped & targeted customer list of those hosted products. In fact, I was on a team in AWS that did exactly that. Why wouldn't their investing arm do the same?