What's your preferred solution for observability and monitoring of tiny apps?
I'm looking for something with really compact storage, really simple deployment (preferably a single statically linked binary that does everything), and compatible with OpenTelemetry (including metrics and distributed tracing). If/when I outgrow it, I can switch to another OpenTelemetry provider (but realistically this will not happen)
I'm personally not convinced OpenTelemetry is the future. I get the desire to not be vendor-locked to a single provider, but Prometheus and Jaeger are very solid, battle-hardened, popular, well-maintained, easily self-hosted open-source projects. For small deployments you do not need to overthink things here - Grafana, Prometheus, Jaeger (with local disk storage), logging depends on how many machines you're talking about and where they're hosted (e.g. GCP Cloud Logging is fine for GCP-hosted projects, the 50 GB free tier is a lot for a small project) but as a default Loki is also just fine and much better than Elastic/OpenSearch.
OpenTelemetry is, last I looked at it, way too immature, unstable, and resource-hungry to be such a foundational part of infrastructure.
This is a lot of infrastructure, we are talking about a tiny app here. Are you sure this is warranted?
Honestly I would prefer to have observability as a library, that's not feasible because of two factors, a) I really want distributed tracing (no microservices - I just want to combine traces from frontend and backend) so I need a place to join them, and b) it could/would lead to loss of traces when the program crashes.
In any case, it makes sense for me to choose tracing and metrics libraries that can output either OpenTelemetry or Prometheus and Jaeger, in the event that OpenTelemetry is not enough.
> Loki is also just fine and much better than Elastic/OpenSearch.
I'm scratching my head a little bit on what your expectation is here. Traces and real-user-monitoring are not the same thing here. Distributed tracing is specifically a microservices thing. Maybe all you're looking for is to just attach a UUID to each request and log it? Jaeger and Tempo aren't going to help you with frontend code.
> A lot of infrastructure
> Prometheus
You need something to tell you when your tiny app isn't running, so it can't be a library embedded into the app itself.
> Grafana
You need something with dashboards to help you understand what's going on. If your thing telling you your app has crashed is outside your app, the thing that helps visualize what happened before your app crashed also needs to be outside your app.
> Jaeger
Do you really need traces? Or just to log how long request times took, and have metrics for p50/p95/p99?
> Loki
If you're running only one instance of your app, you don't need it, just read through your logfiles with hl. If you have multiple machines, sending your logs to one place may not be necessary, but it's incredibly helpful; the alternative is basically ssh multiplexing...
Thank you for that. I absolutely love that this uses tantivy.
I was previously leaning torwards VictoriaMetrics and VictoriaTraces (I will need both) but I think that OpenObserve is even simpler. Later I found Gigapipe/qryn https://github.com/metrico/gigapipe
Does OpenObserve ships something to view traces and metrics? (it appears that Gigapipe does). Or am I supposed to just use Grafana? I want to cut down on moving pieces.
I'm looking at OpenTelemetry because of broad tooling compatibility (both Rust tracing crates, tracing and emit, support it - for logs, tracing and metrics) and it seems like something that will stick around.
Also I'm not sure I will ever need actual performance out of an observability solution; it's a tiny app after all.
I'm looking for something with really compact storage, really simple deployment (preferably a single statically linked binary that does everything), and compatible with OpenTelemetry (including metrics and distributed tracing). If/when I outgrow it, I can switch to another OpenTelemetry provider (but realistically this will not happen)