Interesting. My initial instinct is to compare them to MongoDB which IPO'd about a year ago. They seem to be doing quite well since, but then again, who hasn't in the tech sector?
I will say I very much enjoy Elastic's suite of products, I can't say the same for MonogoDB.
There are key differences between MongoDB and Elastic.
MongoDB (the company), has been very strategic about defending themselves from Amazon. Likely due to the AGPL licensing on MongoDB (the database), Amazon doesn't offer a hosted MongoDB - you have to buy MongoDB Atlas which runs on multiple cloud providers, including AWS, GCP and Azure. Coincidentally, Atlas subscriptions are the fastest growing part of MongoDB Inc's revenue, which is why investors are bullish on the company.
Elastic (Apache licensed) has actually had their lunch eaten by Amazon to a much larger degree. Many businesses are purchasing hosted ES instances directly from Amazon via Amazon Elasticsearch service. The problem is bad enough that Elastic Inc had to write a marketing blog post about it [0].
Say what you will about MongoDB (the database), but MongoDB Inc. is the most successful open source software-based company since Red Hat. You can see it in the outperformance of $MDB compared to similarly timed open-source IPOs such as Hortonworks ($HDP) and Cloudera ($CLDR).
I can't speak for others, but I was looking to set up a quick elastic search cluster and Elastic's hosted offering beat the pants off of AWS ElasticSearch in terms of usability, polish, ease of getting started and security-by-default.
Now, I don't have a lot of data in the cluster, so we'll see how performance is when that happens, but would definitely recommend Elastic's offering.
Elastic Cloud runs in either AWS or GCP. You get to choose but it's just between those. Unless you mean you were using AWS's ElasticSearch service... which is known to not be all that great.
You could build ES clusters on AWS directly on EC2 and it would be cheaper than Cloud, unless what you're really looking for is that support contract.
I spent a few months of my life learning ES inside and out and would choose this option every time. One of the best tech investments I've ever made.
> You could build ES clusters on AWS directly on EC2 and it would be cheaper than Cloud, unless what you're really looking for is that support contract.
Or you are willing to trade money for dev time. Setting up and running the ES cluster on EC2 will take time and energy (source, I did it ). Whereas setting up the ES cluster on either the Elastic Search hosted service or AWS ElasticSearch is really just clicking some buttons.
However, I'm sure the right solution also depends on the amount of data you are using and the problem you are trying to solve.
I can largely agree with the last statement. The problem I find with the hosted services is that people just tend to throw data at it without really knowing anything about cluster/node/shard performance. This typically blows up in their face at some point and even though Elastic Cloud has nice sliders in place for spend-more-money, they still have to go through the rebalancing pains (if they didn't write-lock the cluster outright). That should never happen.
If you have strictly time-series data and are willing to make compromises around retention window, hosted elasticsearch is an easy conclusion. Invariably though, I find that where performance is a real concern, you have to carefully plan what data you're indexing and that requires operations knowledge around elasticsearch. At that point you almost might as well host it yourself anyway.
Also if X-Pack was something I _had_ to have, and tbh there are free alternatives for all of its good features, I would consider Elastic Cloud.
I'm even a customer of theirs on some clusters, but lackluster support and pushy upselling experiences have made me want to move out of it.
An issue I had with AWS was that even after they launched VPC ES clusters, they still hid the node IPs in the API output(I guess they patched ES?!). Our use case as a data lake meant that we had severely degraded performance with the hadoop/spark drivers than one would expect form it.
> Elastic (Apache licensed) has actually had their lunch eaten by Amazon to a much larger degree. Many businesses are purchasing hosted ES instances directly from Amazon via Amazon Elasticsearch service. The problem is bad enough that Elastic Inc had to write a marketing blog post about it [0].
Also the reason Redis Labs made the controversial decision to change their licensing (and they reference Elastic in it) [1].
> Cloud providers have been taking advantage of the open source community for years by selling (for hundreds of millions of dollars) cloud services based on open source code they didn’t develop (e.g. Docker, Spark, Hadoop, Redis, Elasticsearch and others). This discourages the community from investing in developing open source code, because any potential benefit goes to cloud providers rather than the code developer or their sponsor.
It's a big deal for a lot of companies to be able to pay the cloud bill in one place. Elastic used to provide the availability to pay for the Elastic cloud subscription through AWS' Marketplace. Elastic stopped this when they changed their pricing at the beginning of 2017 and we (my last employer) were negotiating a contract with them at the time. AWS billing was a hard requirement for us because it made it a lot easier on us from the finance side. We walked and I know a lot of others who did too.
It’s a business decision of course. Forge your own path or be beholden to your revenue model dictated by AWS.
Big Enterprise will not require billing through AWS (disclaimer: employed at a large enterprise, buying ~$900k software license currently). They’ll have an accounts payable department, a procurement process, and someone to cut a check or ACH to Elastic monthly/annually.
I totally get that. I don't dispute that. But if you must choose between "please sir, can I have some more?" and controlling your own revenue destiny, it is clear why some would choose the later.
Would you want to be beholden to AWS' whims? You might as well be a Walmart supplier at that point, constantly dragged out to Bentonville having to make your case for every penny of your costs. Amazon has proven its model (all about scale), how it operates (your margin = our opportunity), and you partner with them at your own peril.
I'm suggesting the thought process to shift from, "Ahhh, those Elastic guys and their bad decisions!" to "Hmm, well, tough decision to make [forgoing revenue through AWS] but I understand why they made it".
Can someone explain why a company would pay a cloud provider extra to manage an open source API for them when they could just run it in the cloud themselves?
Another thing to note, along with sibling comments: AWS Elasticsearch is limited in its capabilities. For example, you are blocked from relying on custom ranking models to rank your documents.
There are a lot of limitations with Amazon's hosted offering. Mostly you will find out about them when your cluster needs maintenance (upgrades, outages, etc.). Elastic cloud is a much better product in terms of features, support, performance, documentation, price, etc. I've used both and ran my own clusters as well. Elastic Cloud is my default option these days.
Can you say more about these limitations? Right now I'm using a pretty big custom ranking function to rank my documents in Elasticsearch (using a function_score query), but maybe I'm misunderstanding.
I setup a 3 billion plus document ES cluster using AWS Elasticsearch Service. As a whole it works really well, and it's nice not to have to manage individual EC2 instances. With that said, there are some very annoying things. Previously if you modified the access policy of the ES cluster, it would completely rebuild the entire cluster (create all new nodes and copy all data), which can take many hours.
Recently I turned off some slow query and slow index advanced logging options, again AWS completely rebuilt the entire ES cluster when these options surely should be able to be applied without copying all data.
With that said, there are some very annoying things. Previously if you modified the access policy of the ES cluster, it would completely rebuild the entire cluster (create all new nodes and copy all data), which can take many hours.
From what I can tell, it still does. I can’t comment on the length of time though. I did it before I had many documents and it was a very small Dev instance.
Really great reply - thank you for sharing. I was just researching the company more for investment. I agree also with regards to Horton/Cloudera. They had so much promise - but felt out of favor.
The results for ElasticSearch are from 2015, curious to see how relevant they are now especially given that the team has been working on addressing these issues in the interim.
Also elastic back then was used as a place where you copy your data, not your main data store. (Their positioning changed later) MongoDB was supposed to be used for your primary data store.
I will say I very much enjoy Elastic's suite of products, I can't say the same for MonogoDB.