Now everyone can know the joys and horrors of Bundler! ;) I kid. This announcement is very exciting for people tinkering with Rust like myself.
I really like Go, but the lack of a solid package management solution that I like using is a downer. Before you rage-comment: I know how Go packages work. I know you think they're better than anything that's ever been invented. I know there are solutions out there for some portions of what I want. But nothing has been created that really works for me (yet). So, to see this development in another one of my "tinker with it but not build a ton of production quality code just yet" languages is exciting!
> I really like Go, but the lack of a solid package management solution that I like using is a downer. Before you rage-comment: I know how Go packages work. I know you think they're better than anything that's ever been invented. I know there are solutions out there for some portions of what I want. But nothing has been created that really works for me (yet).
What is it that you do want in a Go "package manager", then?
You say that you know how Go packages work, so I'm assuming it's not one of the typical misunderstandings of newcomers from the Python/Ruby/Node.js world (all 1.x code is forwards-compatible with no modifications, static linking means makes specifying versions of packages less relevant, the package/filesystem layout parallel means that vendoring project-specific forks of packages is relatively straightforward once you know how, etc.)
I put "package manager" in scare quotation marks because one of the design goals of Go is essentially to render most of the functionality of such package managers irrelevant.
I say this as someone who has been writing Go for work on a daily basis for over a year and half: while Go's package system isn't perfect, it's pretty damn good, and I haven't felt the need for a package manager at all ever since I learned the way things worked.
I put "package manager" in scare quotation marks because one of the design goals of Go is essentially to render most of the functionality of such package managers irrelevant.
Golang does have a de-facto package manager in go get (with its own conventions on file system layout etc), it just doesn't handle versioning. The Go packaging system is pretty good, but it isn't perfect, and I'm not convinced vendoring is a viable solution if you're going to be sharing code and collaborating on several open source projects - it's a recipe for lots of incompatible versions with security and other bugs unfixed floating around in different projects, and a lot of confusion from newcomers about the required version of a library because it is not specified anywhere.
The current approach works perfectly if you are working in-house, willing to do your own dependency resolution, and are happy to vendor your own dependencies - it's completely reasonable within one company or org to do this. However it fails in more collaborative or open-source environments like libraries shared on github depending on other libraries shared on github. I do think they need version pinning for that (in fact they have version pinning for the language in go get, just not for individual packages), it wouldn't be a large change.
For example, in the recent Stripe CTF, people ended up downloading and building different packages locally than those available on the server builds, because there was no clarity on which version was included. That was kind of annoying. It could be solved by vendoring, but then you have other issues in the long-term with relying on other people to keep their pkg dependencies up to date or you can be including two conflicting versions of a third pkg.
They've taken an interesting approach in trying to eschew versions (which avoids some problems with dependency resolution), but I'm not sure it is sustainable long-term. Didn't they also try to make golang itself version free initially?
Well, Rust also uses static linking by default and crates are also laid out based on the filesystem [1]. But we still feel the need for a package manager...
[1]: You can nest subdirectories within the package directory if you wish, to provide more fine-grained namespacing, but all source files belonging to a Rust crate must be descendants of one directory (in contrast to Go, in which the source files must be children of one directory).
I remember reading http://golang.org/cmd/go/ (specifically, the part about the <meta> tag) a while back and thinking "well, that'd be pretty straightforward to get working" and then (sadly) never having the time for it :(.
Thanks for the link. Just tried it out, nice, and I love the use of semver. So you can do this to import a specific version of say yaml
import "gopkg.in/v1/yaml"
or for an arbitrary url on github
https://gopkg.in/username/v1.0.0/pkg
Weird that versions are before the pkg name though, that doesn't seem right...
Here's hoping this sort of versioning scheme makes it into the go toolchain eventually. The only downside of this solution is it redirects only to github and guesses urls, I quite like that the go toolchain is indifferent as to where the sources are hosted. This would be very easy to add to the toolchain though; all they'd need to do is recognise imports like this (personally I prefer versions at the end of current urls, not in the middle):
import "bitbucket.org/user/pkg/v1.1.2"
in go get and download that tag to that path, no other changes in the toolchain would be required, and those who ignore versions could continue to do so, while people who wanted them use the version specific imports. I think the objection of the go team was that this doesn't solve any of the complex dependency issues that plague package managers, but it'd be nice just to be able to specify which version is expected for future users and have predictable builds when sharing code outside one org.
I actually haven't even used gopkg.in myself. Gustavo Niemeyer just released it and has been pushing it as one way to promote API stability. I agree with him and plan to migrate to the scheme soon, as there are a few people using my libraries.
With that said, I doubt very much it will make it into the official toolchain. There's really no point if it can be offered just as well by a third party. For example, godoc.org has proved invaluable but isn't official.
Also, keep in mind that gopkg.in is still pretty new, and I believe Gustavo has said that adding support for other web sites or revision control systems is perfectly doable. Just wanted to get something started for the lowest common denominator, I suspect. :-)
The advantage of having it in go get would be that no one has to rely on a single point of failure like gopkg.in, nor would they have to get new urls recognised there, and finally that it could become an accepted way to version dependencies instead of the versionless future golang lives in right now. They did add versioning to the lang after a while so hopefully this will be similar. It's not a huge deal anyway right now, it'll become more important as the pkg universe grows.
> static linking means makes specifying versions of packages less relevant
I'd really like to know why you think this is the case. I don't specify versions of dependencies because of dynamic linking, I specify versions of dependencies (wherever possible) so I know I can still build the thing later.
An old binary of a go program is no use to me as a developer.
I think he's referring to the fact that you won't try to run some executable that depends on Ruby 1.9.3 and SomeGem 2.9, but you're on your rbenv that is Ruby 2.0 and SomeGem 3.0.1.
I'm not a Ruby developer, but I know this scenario has caused me considerable trouble just trying to use things built with Ruby.
This doesn't work when you have more complex dependencies, because they in turn have dependencies, which may clash. You have no way of knowing what pkg c version packages a and b expect when you include a and b.
If the dependencies clash, it doesn't matter what system you use. There isn't a package manager good enough to fix the issue of having conflicting dependencies.
Vendoring or cloning (with explicit upstream merging) is again your best bet for dealing with this as it the only way you have complete control of the dependencies involved.
If you are saying something like having a package X in your vendor lib that depends on packages Y and Z which you don't have vendored... then you haven't done it right. You vendor all dependencies.
I agree vendoring is essential in some situations, mostly in deployment to production or keeping tight control of libraries used in a project, but I disagree that versioning is not useful in a packaging system.
If you are saying something like having a package X in your vendor lib that depends on packages Y and Z which you don't have vendored... then you haven't done it right. You vendor all dependencies.
No, I'm saying imagine you have an app, which depends on pkgs a and b. a depends on c1.1, b depends on c1.5, but this is documented nowhere in the code because the import statements simply say import "c". This is the default and encouraged behaviour currently with golang and go get. go get a fetches the latest of c.
You then vendor a,b,c and happily compile your app, which doesn't work because the version of c you got when you vendored was 2.1, which neither a nor b was written against. Maybe your app will fail to build and you fix it, maybe your app will build but be wrong in subtle ways (say an enum value changed), without explicit versioning you have no idea really.
If a and b had a vendored c, you have an even more messy situation as security bugs exist in c1.1 and c1.5, but you don't even know which version you have if you pull in the ac and bc and compile against them, or if you can safely use two versions of c at once, your code uses c2.1 too, etc, etc.
Being explicit about versions is helpful for resolving dependency conflicts because it makes it explicit which version was expected at the time of writing, it doesn't magically solve all conflicts, no-one has this completely worked out, but it helps to be as explicit as possible when including other projects, and it is helpful in keeping up to date libraries whilst easily including them, even if you vendor.
If you have a large, open ecosystem of constantly evolving libraries, which I think golang should aspire to, it is useful to version them (just as golang itself is versioned).
Dependencies clashing is not always a binary thing. If they're well-specified, there may be a way to meet everything's version specifications.
Anyways, this isn't just academic. Tools to enable you to specify dependencies without vendoring them have been developed and used in several different environments, and at no point in using any of them have I gone "Gosh I wish I could go back to cluttering up my version control with other people's code!"
> If the dependencies clash, it doesn't matter what system you use. There isn't a package manager good enough to fix the issue of having conflicting dependencies.
With NPM, it's fine as long as the two modules aren't calling into each other with improper data as a result of the dependency clash (because each dependency is responsible for installing its own subdependencies, rather than relying on a global namespace/install location). So, you'd have a problem if A relies on C1, and B relies on C2, and you got data in format c1 in A and called into B which expected format c2. Otherwise, you're fine.
There are some people trying to remedy this issue. While the Golang designers don't feel the need to include it in the language, others may resolve this. Id recommend reading the goals doc, as they seem to have a good starting point.
Handling packages in Go is always something that stops me from going any further when I look at it. Every time I realize I won't be able to properly specify dependencies with versions or pin them (like what Bundler does) it drives me away screaming. It's only the last few years dependency hell has been resolved in other environments (like Ruby and Node, to a lesser and sometimes weirder extent Python), I don't really want to go back in time any more.
Coming from Ruby, when I last looked at Go about 5 months ago, I was dismayed at the package management state. My specific concern was around version pinning. I'll be excited to try out Cargo when it launches.
I really like Go, but the lack of a solid package management solution that I like using is a downer. Before you rage-comment: I know how Go packages work. I know you think they're better than anything that's ever been invented. I know there are solutions out there for some portions of what I want. But nothing has been created that really works for me (yet). So, to see this development in another one of my "tinker with it but not build a ton of production quality code just yet" languages is exciting!