Check out https://apify.com (disclaimer: I work there :)) - a platform focused on web automation and web scraping. You can easily build any (but mainly NodeJS or Python) project there as a Docker container and schedule it in a minute. See my 2 min. video - https://share.cleanshot.com/osSygJ
In our case (Apify.com) there was a complete outage of SQS (15mins+), most likely DNS problems + EC2 instances got restarted probably as a result of an SQS outage.
EDIT: Also AWS Lambda seems to be down and AWS EC2 APIs having a very high error rate and machines slow startup times.
Join us on our journey in building a web scraping and automation platform to help both the world’s biggest companies and upcoming startups leverage the web's full potential. Each month, we save people tons of hours of manual work by turning more than a billion web pages into data and automating over 15 million tasks on the web.
Apify runs on a highly scalable infrastructure that runs millions of web automation and web scraping jobs weekly to process way over a billion web pages every month. We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS.
Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services.
We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github https://github.com/apify. Help us make the Web more open and programmable!
Apify runs on a highly scalable infrastructure that runs millions of web automation and web scraping jobs weekly to process way over a billion web pages every month. We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS.
Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services.
We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github https://github.com/apify. Help us make the Web more open and programmable!
Apify runs on a highly scalable infrastructure that runs millions of web automation and web scraping jobs weekly to process way over a billion web pages every month. We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS.
Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services.
We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github https://github.com/apify. Help us make the Web more open and programmable!
Apify | Full-time | On-site | Prague, Czech republic
Apify runs on a highly scalable infrastructure that runs millions of jobs weekly to process way over a billion web pages every month. We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS.
Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services.
We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github https://github.com/apify. Help us make the Web more open and programmable!
Apify runs on a highly scalable infrastructure that runs millions of jobs weekly to process way over a billion web pages every month. We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS.
Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services.
We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github https://github.com/apify. Help us make the Web more open and programmable!
Apify runs on a highly scalable infrastructure that runs millions of jobs weekly to process over a billion web pages every month . We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS.
Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services.
We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github https://github.com/apify. Help us make the Web more open and programmable!
Priced: $0.25/GB-hour + data transfer and storage