I don't think something like this should be hosted/SAAS. I don't want to send my Gigabytes of logfiles (sometimes containing confidential informations ...) over the internet to some unknown entity with probably questionable security.
My small startup company already has more than 10 (virtual) servers which would cost us 500 dollars a month to monitor which is more than the servers itself cost.
We take security very seriously, but let me turn this around into a question: what would it take for you to trust an external service to manage your logs? Some of the things we're doing:
1. SSL everywhere (including internal traffic between our backend servers).
2. We add a tag to the raw representation of every string value, so that we can verify that data never leaks across accounts. (This has never detected a problem -- except in tests, because yes, we do test it.)
3. Implementing in a "safe" language (Java), to rule out low-level buffer management bugs.
4. As Greg noted, we make it trivial for you to redact sensitive data before it leaves your server.
We are sometimes asked for an on-premises installable version of our service. We don't provide that because we're using economies of scale on the backend to completely change the log management experience: when you give us a query, every CPU and spindle in our entire cluster is briefly devoted to that query. This means that you aren't limited to graphing predefined metrics; you can do ad-hoc exploration of your entire log corpus in on the fly. E.g. display a histogram of response latencies for all requests for url XXX on server group YYY in the last 48 hours, and expect a near-instantaneous response.
>what would it take for you to trust an external service to manage your logs?
I think that's a really weird question that completely fails to address the concerns that some people might have. We do logging of sales, profit margins and stuff like that. You can't have access to that because: "You're not us". If you can read our data, then we're not going to use your service and to do anything useful the logs you really do need read access.
Of cause you might have no reason to spy on our data, but the only safety is that you promise not to. We could seperate logs for different things, so webserver logs go to you, but email logs goes to an internal system, but then we would need two systems.
What's your point there, that because they are already exposed in some areas they shouldn't care being exposed in more ? Or are you looking for a "ah-ah" moment; "you're already compromising that data" ?
Because either way, that doesn't change at all the concerns he voiced regarding this particular service.
I suppose both? I think it's pretty reasonable to expect a business dealing with storing sensitive data to not look at that data, regardless of it is email or logs.
> what would it take for you to trust an external service to manage your logs?
An act of God. This implies, of course, certain further prerequisites that are themselves probably quite challenging to meet.
> economies of scale
Under your current plans, the money spent putting my entire infrastructure on Scalyr would pay 2-3+ developers (or some mix of developers and sysadmins) in Taiwan, where my employer is based. We are not a tiny company, but that would be a substantial and welcome manpower increase for the server team, and only a fraction of that manpower would need to be spent meeting even our "would be nices" for logging and monitoring.
This would only get worse as we grew. Realistically, I would expect us to be able to open an office in Silicon Valley and start hiring there at market rates for the amount of money we'd be giving you.
> what would it take for you to trust an external service to manage your logs?
Well the no-brainers are:
1 - Encrypt everything before living my infrastructure (with my key, not SSL), only decrypt it in my infrastructure when generating reports.
2 - Anything that runs on my infrastructure is open source, and widely distributed. Bonus points for a simple protocol that I can write plugins for.
3 - Make it possible for me to back everything up, and restart everything in case you go out of business.
Those are the must have "I won't even let you get through security otherwise" features. Now that I think about it, #3 alone makes anything you can offer worst than doing it in-house.
But none of those are features that'll make your system look any good in my eyes, they are just enough for your system to not look like an enemy.
Completely valid points. Here's how we're dealing with each:
1. With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers.
2. We're exploring a different pricing structure right now that would address this scenario. If this is your only hesitation, I hope you check us out anyway and contact us about pricing.
> With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers.
This is both error prone and utterly defeats the purpose. Why would I pay a bunch of money for somebody else to manage my logs when I'd just have to keep them all anyway so I can get at the unredacted versions when there's a problem?
My small startup company already has more than 10 (virtual) servers which would cost us 500 dollars a month to monitor which is more than the servers itself cost.