Bugjail was mentioned in a thread about a xor bloomfilter alternative. Or at least that's where I saw it first mentioned. My bet is it triggered op's posting.
I think you need to make the experience more concrete.
On the visceral side: put up some example videos of use. If you can't do it to the level of polish you like, get someone else to do it for you as an unofficial one. IntelliJ is already slow, and I'm worried about event-recording debuggers being only useful for small programs -- larger ones eat up too much RAM and cpu overhead to be tractable.
On the power side: show some sophisticated use cases using real libraries and real problems to fix. What stuff can I find out directly that I can't easily do in IntelliJ?
Can I directly query the DB? Can I use those queries' results in regression tests?
When I read the page, I thought "Oh, like the valgrind family of tools for Java, nice." So perhaps some pre-baked validations using the tech you've built could be useful.
About performance impact and program size: BugJail can be used also with hardcore programs, like databases and compilers, where event based record-and-replay debuggers typically struggle. There's still significant performance hit, around "twice as slow", but it won't be orders of magnitude slower.
Also, BugJail is tested with large applications. For example one test is startup sequence of Eclipse, on 5 year old box with 32GB. The result is ~700 million rows in database, in ~30 seconds. It does consume all memory, and your workstation will scream for mercy, but it gets there :)
Direct DB query is not available in current beta, but it is partially implemented in dev version: it shows the results, but there is no navigation from the query results to CallTree, Source code view etc. So: coming, but not yet.
I hope I answered your questions at least partially? It's 7AM in the morning, after one of the hottest days in Australian history, and a surprise appearance in HN front page, so I might not be quite the sharpest tool at the moment :)
There's fair bit of compression already, but actually I haven't done that much industrial spying into time-series databases, good idea, thanks.
Also, I've tested dual machine debugging with BugJail, as part of spike towards CI/QA/SIT/UAT/Pre-Prod use. It works okey-ish but the (crappy 1GB) network became bottleneck and it was slower than single box.
My "Fair bit of compression" was massively simplified: there isn't one true canonical "compression", but dozen different places and ways where and how to compress.
BugJail does compression around storage layer, but nothing before the network hop from capturing agent to processing server.
And fully agreed of course; when BugJail graduates to CI/QA/SIT/UAT/PreProd use, it will need compression before the network hop. But that is some time away. Currently the agent and server run in same box, and compression between them has negative net effect (compression uses more CPU than what shifting around less data saves).
> If you can't do it to the level of polish you like, get someone else to do it for you as an unofficial one.
Yes, have one (or several) customer(s) write a blog (that you can help proof-read, add links to documentation, your example database, your reference guide...) on one or two features. "here's how I did it before with eclipse/intellij, how it was long and painful and manual and involved and xxxx. Now with bugjail I can do this with xxx and xxx". A blog I can read and you can polish bit by bit, asynchronously. Video can be hard to follow, copy-paste, with good and clear simple English, or you could caption but then again, lots of energy to spend!".
Maybe offer some incentive to this (those) customers of some kind (5 year free support, a number of enhancement tickets...).
The best way to demonstrate (I feel) is sometimes a happy customer with complex but satisfied needs.
- The Linux packaging will be re-done from scratch (previous beta had completely re-done packaging for Windows and Mac, re-done Linux packaging coming in next beta) and I'll make sure all these get fixed.
- The license is displayed in the launcher screen (before you click "Enter BugJail"), but i'll make it also available online, good point, thanks.
- Redistributing Oracle JRE as part of BugJail package is done exactly according to their redistribution terms, no problems there.
- The reason for redistributing JRE has more to do with Windows and Mac, where 'marking it as dependency' is not an option. I'll reconsider how to do this when the Linux packaging is rewritten.
- Would AppImage packaging work for you? What distro do you use?
AppImage would be even worse, as it's a fairly new technology and not all that well integrated. I am an ArchLinux user and generally it's easiest if the software comes as a tar.gz with installation instructions and list of dependencies. Those of us using alternative distros, i.e. not fedora, debian or ubuntu are most comfortable with getting a simple archive and adapting it to their distro. Trying to make an installable package for us only leads to frustrations for everyone.
I understand on Windows/Mac you have to package the JRE, but I'd recommend considering using the distro for that functionality on linux - it's generally frowned upon to re-package stuff in the official repos.
I think an AppImage would generally be worse, I'd rather be able to use my system's JRE and libraries. I'm on Arch, so if you provide an archive of the application we can work with that. If you really want to go above and beyond for Arch users you could make and publish an AUR package.
Thanks, I'll do as you recommend: JRE as dependency and add .tar.gz packaging. And shoot me an email if you'd like notification when done(my email is in the contact page).
This isn't quite time travel, and it's not quite too expensive, and so on. Well, it's a nice idea, for sure, and it needn't be Java-specific either -- it generalizes.
Of course, you might need enormous amounts of memory or storage if you want to observe a busy process for a long time, but presumably one would only do use this tool for brief periods.
BugJail doesn't do currently line/statement level debugging, but it already captures all the information required for full "time travel" debugging. So, coming someday.
Also, BugJail is designed so that in normal use (webapps etc) it writes fast enough to database and never runs out of memory. Though, hardcode programs like databases and compilers exceed that, and currently don't buffer to disk, but crash it hard..
If you configure BugJail to capture only your own code/packages (--primary option) the disk usage is not that bad either. If you don't, you'll burn in the magnitude of ~10MB per second.
> Next is BugJail for Android, with beta starting late 2018, hopefully.
Why does the website showcase outdated content? Personally, I will drop off at this point. You might have put this message pre 2018, but you should keep the landing page up-to-date.
Also, as one of the other commenters pointed out, you should spend some extra bucks and proof read the landing page. Sentences like "And cheap perpetual license after that" does not go well for most customers who are willing to shell money for a product. If your product is worth it, people will pay for it.
Tech is just a part of the equation. If you don't take care of other stuffs like messaging you are sending wrong signals to your potential users.
I think JS should be higher on the priority list, considering the demand to effort ratio (I'm thinking it's probably easier to implement for JS than Java). I would guess that it would also be easier to host it and load JS on the fly to debug online. Just my 2c, will try the Java one anyway to test it out.
I totally agree on the demand, but on the effort side the dynamic nature of JS makes it really tricky to do right. Thanks for your comment & let me know how it goes!
Take a look at GraalJS/Truffle instrumentation. It lets you easily run JS and even NodeJS on the JVM with per statement instrumentation handled at the engine layer. So it should make it very easy to add the code you need and the compatibility with modern js is great.
The critical problem is not the instrumentation, but the way how BugJail works: it doesn't dumb the raw events to disk, instead it reconstructs relational model of the program execution. This requires resolving raw low level events against schema of types/methods/fields/etc. With JS the schema is dynamic, so now the already hard 'reverse-engineering' has moving target. Not impossible, just hard.
And thanks for the pointer to GraalJS/Truffle, I'll have look and perhaps that provides great way for doing the instrumentation side.
Hi, I'm interested in program execution state and the different ways it can be represented/modelled. The "relational model" you describe sounds very interesting. Do you have a technical whitepaper on this topic, or perhaps some third-party literature that has guided you?
I share the same interest as parent, a white paper or detailed blog describing the vision and how to use it would be amazing. But then it might be a bit 'secret sauce'... At one time or another you need to explain or demonstrate your killer features in a way that people will react : 'hey I've always wanted that but didn't know how to ask for it!'.
Good luck, it seems with all the critics with the copywriting on your landing page, you've piqued the interest of a lot of people. I hope you'll feel encouraged by this.
Alright, I'll add writing blog post about the implementation to my TODO list. And my feeling is closer to exhausted than encouraged, I was just going to bed, after day of 45c heat, when the half-finished landing page hit HN front page, and it's now 7AM :)
I would suggest something along the lines of YourKit pricing which seems to be industry standard for the most part.
Also their personal license which isn't advertised heavily is $99/yr which allows for commercial use but must be paid for by the individual rather than a business entity. I like this license because I like to take my tools with me and doesn't make me beholden to purchasing/procurement departments of the companies I do work for.
>> All feedback is highly appreciated, no matter how brutal.
You should really get a native English speaker to proofread and correct the website. The content itself is fine but there are so many spelling & grammar errors in the text that it really starts to detract, and it honestly affected my perception of the quality of your software in a bad way.
Also not a big fan of the long-scroll style landing page in general, but that seems to be the flavor of the day now. I think some structure (index, menu) would make it a lot easier to navigate to the relevant information, right now reading top to bottom I had no idea what kind of other information could/would/should have been lower down the page.
>>I hope you can also brutally criticize everything else beyond the surface! :)
I recommend removing the word "cheap" from your landing page. I understand the sentiment (and love it), but what may be "cheap" in one part of the world is likely "expensive" in another. It also cheapens (no pun intended) your value proposition from an enterprise perspective. Be confident and drop the word.
Also, perhaps in some fine print (or even better - a clearly visible and simple explanation) defining the scope of the perpetual license -- limited to +K (where K ≤ 0) major software versions or all future releases (K = ∞)?
"cheap" in English means both "low cost" and also "low quality". To convey what I think you want to convey you could use the word "inexpensive" or "low cost" (but I also agree with the parent comment's advice).
You need to do a market analysis yourself as developer tools are not my area of market expertise. As a general strategy, given this is a new tool you might want to start with just two dimensions/factors: introductory vs. long-term (to drive adoption), and consumer vs. enterprise.
For the purposes of answering your question in the short-term: you might end up with a perpetual license single-developer price knowing that this initial set of customers will be innovators and an "investment" that will be on your books for quite some time. You could even time or quota-limit if you're looking for a specific growth trajectory and initial burst of revenue.
Given that this is perpetual licensing (and again - you need to decide whether this will be for all versions forever, or limited to +K major versions), you need to balance 12-18month runway (cost-feature) planning against any margin you're hoping to generate (maybe none if you're aiming for ramen profitability).
Feel free to reach out if you want to discuss further.
Of course I'm not expecting to be spoon-fed with a ready made pricing model, but was just trying to get overall sentiment..
You make a lot of great points, and this is one area where I definitely need help, so thank you for your offer. I'll get in touch after I've survived today :)
Somebody else suggested this already, but I’d probably do something like the following:
$100 - personal license, must be bought by an individual (not company), but can be used for commercial purposes. Best effort only support, pushing folks to use a support forum.
$200/seat - business license version of above, but including email support, just without an SLA.
$400/seat or $15/seat/month - enterprise license, includes priority support.
Based on experience, providing support is the most “expensive” and time consuming portion, so make sure you are pricing high enough to cover that, especially with perpetual licenses.
Keep in mind what this tool replaces. Sure, most obviously, this tool might "be" a debugger, but if by using it you suddenly don't need separate static analysis/reverse-engineering tools in your debugging workflow, then it's in competition with those tools, too. Imagine everything you'd have to buy+install in order to get the same effect as this tool, and then charge more than that (because probably there's at least a few use-cases where, without this exact tool, you just can't do it at all.)
Thanks, really great point. BugJail actually started life as internal tool for dynamic analysis, not for debugging. The longer term vision is merging of dynamic and static analysis, so not exactly, or only, "better debugger".
Thanks. My current plan is pretty much exactly that: perpetual license for $ X including 1 year of updates, then $ 0.5X for additional year of updates/maintenance.
The main thing I'm uncertain about is the value of X :)
Many legal/finance teams hate licensing costs based on revenue numbers. I’d suggest avoiding that and instead tying something of value to larger companies to the higher priced license, like priority support.
Tech that disappears because it isn't profitable enough (or at all) is even worse. A successful business model is in everyone's interest. Don't unilaterally rule anything out, look at it rationally.
If the goal is to help as many developers as possible, then being profitable and/or well funded makes achieving that goal easier, through hiring more employees and developing better product faster. Which in retrospective sounds obvious.
I'll still keep perpetual license as option, but stop being arrogantly non-commercial.
Very sensible viewpoint, thank you. Perhaps perpetual licenses and subscriptions could co-exist as alternatives, and customers can choose whichever they prefer.
He should really remove the political content too. Yes, many of us think Trump is an asshat, but why alienate a potential customer who disagrees? I think it's an attempt to be edgy/cool (along with "Frigging") but I think it risks hurting more than it helps.
It probably doesn't matter, but the third screenshot wasn't any kind of planned attempt of being edgy/cool. Just another 16h coding day ending in bit more lighthearted way.
Database-backed debugging is great! It means that debugging can be done as a data science exercise, which is very useful when, for example, a rare confluence of external factors make the program behave badly.
Earlier examples: "Debugging in the (very) large", which describes how Microsoft used Windows telemetry, especially for debugging weird slow-down glitches. In the networking domain, where there is AppDynamics for trying to debug the interactions between network and distributed services.
I'd love to see data-science style debugging much more accessible (especially in Python). And it's great to see that you're making a serious effort to get the user experience right -- though I hope that the underlying data will also be accessible from more capable data science platforms.
BugJail writes its data files in open source format, Apache ORC.
You can load the data files directly to Spark, Hive or anything else that reads ORC, and then use Python to query, visualize or get statistics from the captured program execution.
I think it might be really useful to run BugJail as an (optional) CI step and offer the user to download, integrate with the IDE, or have historical data. But I didn't fully understand the license regarding such usecases.
Absolutely. The long term plan is to do separate edition "BugJail for CI/QA/SIT/UAT/PreProd/etc", so that when things-are-not-fine on server, you don't need to repro but just load the capture in BugJail and see what happened [edit: that in addition to your "post CI build step" use case; both uses seem compelling].
But the current beta version is not robust enough and doesn't handle long captures well enough. I want to make sure that BugJail doesn't waste people's time, hence the license allows only capturing from local workstation.
I wonder how they are able to trace and capture variable and stack state at a rate of 20M invocations — per second!?
How is this possible on the JVM?
(I’m aware of the instrumentation API that profilers use but I thought they had to do sampling to limit interference with program performance and they are not capturing all the data req’d for debugging either.)
Pure speculation: You don't need to record everything, only enough information to reconstruct what happened later (e.g. outcomes of branches, results of system calls, etc). You can then run a post-processing phase after execution (or in parallel on another thread) to fill in the extra detail that is needed for efficiently presenting what happened.
This is really neat and I wish I had this around 10 years ago when I had to migrate Java apps.
However, these days I rarely need to debug programs while developing - I usually use some combination of TDD with unit/integration testing and it gets the job done with the added bonus of being easy to maintain down the road.
Of course if you're on a project where time to market is critical and you need to sacrifice tests, I suppose a debugger is required. But then again, you're probably not using Java either.
Since "database" and "relational model" is mentioned, I assume the collected data is indeed being written to an RDBMS. Would be interesting to be able to poke around in that with some SQL, even if not available as an official API.
I tried using it and my first impression is that it's pretty difficult to use for a webapp (or maybe just specifically a Spring application since there are a ton of added proxy classes) without the ability to search. You need to know exactly which thread accepted your http request and then dig your way down the call stack to find your method invocation. You should be able to search for your method invocation and then go back up the call stack from there.
Other than that, seeing exactly how long every single method call took is awesome. I think there's a lot of potential, but without the ability to search I can't see myself reaching for this again
The search functionality is currently under development, and it's by far the most critical 'missing' feature.
As a workaround for webapps: using the 'Capturing Assistant' and configuring BugJail to capture only your own packages helps a lot. That way you usually see clearly which thread handled the request, and also have less app-server / framework level crap that you need to expand through before you get to your own code.
If you'd like to get notified when the search functionality is available, just shoot me an email (my email is in the contact page).
Thanks for providing the feedback, and sorry that the initial experience wasn't better.
Thanks for your reply. I actually did specify my packages in the agent -javaagent:/Users/aa/.BugJail/agent.jar=--snapshot:.--primary:io.mycompany. but it didn't seem to help too much. I'll shoot you an email to get notified - is there an estimated timeline?
Hmm, if setting --primary didn't help, then it might not have worked (aka Bug in BugJail..). I'd love to check that further tomorrow, if you have couple of minutes?
As for the search/query timeline, I'm afraid to say anything more exact than "month or two". The core search/query is working on dev version, but it needs a lot of work.
I tried playing around with this and I can't see it being useful for me currently. If the source and tracing information were more tightly integrated it would be a lot more useful. Offering a more traditional debugging experience with the ability to rewind time to the last time a value is set would also make it valuable.
A specific issue I ran into was that multi dimensional arrays do get logged as changing when a value is set in the bottom layer. This meant for my application the were a bunch of init calls setting that value, but none of the actual logical changes were tracked.
The source and tracing sides will definitely get fully integrated. This is only the second beta that even has the source side at all, so it's just "early days".
The line/statement level traditional debugging is possible to implement with currently captured information, but the implementation is seriously hard, likely in 2021+ timeframe.
About the specific issue: I'd really like to get that fixed, do you think you could shoot me an email so that I can followup in case that i cannot repro the bug (my email is in the contact page)?
Many thanks for the feedback & apologies that the initial experience wasn't so great
Shouldn't take long. Google "screencast software". Record yourself: "Hi, this is Henri. Let show you how find bugs with BugJail..." Do a 3-minute video with some simple examples. Put it on YouTube and embed it in your site.
If I were doing it, I'd budget 4-8 hours of work and close to zero cost.
The execution logs (or "Captures", in our Jail terminology..) do get fairly large if you capture something hardcore, like database or compiler.
For normal web/business/enterprise applications you can configure BugJail to only capture your own packages, which massively reduces the file size (and performance impact).
To give concrete example of hardcore scenario, capturing startup of Elasticsearch results in about 85MB file.
There is no way to enable capturing dynamically, and this is likely never going to be supported, due to the way how BugJail works (it reconstructs the program execution, so if beginning of the execution is missing, "things are not fine").
As a constructive feedback on just the presentation of the landing page I think the screenshots are showing some code with theme colors that are kind of ugly.
I understand this is a landing page but if you care about Developer Experience DX I'm sure you can find a theme that is better looking or inspiring.
Looks like a pretty nice way to debug Java programs! I look forward to testing it.
One quick note though, you might want to consider removing the politics from your screenshot before really releasing this product. You've got a great product here, but there's no need to inject politics - even if it's amusing to some. It takes away from the seriousness of your project and makes it feel more like a personal side project instead of something I should pay for and build into my workflow, in my opinion. Just my two-cents.
Great! I've shared your link with a few other java devs and the consensus so far is this looks like an amazing tool. I can't wait to take it for a spin! Keep up the great work!
This doesn't work if source code is not available. I can get a trace of the function calls but no arguments, or return values or anything. This makes it entirely useless as a reverse engineering tool, something i was very excited to use it as due to it essentially being a tracer. It also NEEDS modern java support.
Actually, it does work without source code (or, should work)
The left side 'CallTree' is fully functional, and shows owner instances, arguments and return values. If you click those, then the Object/Array is displayed at bottom and is expandable just like in normal debugger (+slider to change debug time). So only the right side "Source" view is missing if you don't have source.
Have a question to rr or any replay debuging tool, does it still work if the program has database access while the replaying machine doesn't have or not have the same database?
Yes, it records all syscall return values, so the replay doesn't actually need to perform them. This means that IO (GUIs, network accesses, etc.) shouldn't happen on replay, in general. (The replay does show stdout/stderr data, so perhaps those are special-cased?)
