> More to the point it's clear from watching the activity in the open source community at least that many of them don't want aligned models. They're clambering to get all the uncensored versions out as fast as they can. They aren't that powerful yet, but they sure ain't getting any weaker.
There simple explanation for this. Getting the models which small startup cannot afford to develop and train is the only way to move forward. To get some investments, or before spending their own money, they need a proof of concept at least. Besides, working models are a good learning resource.
I think you're missing the point here? There are plenty of censored / "aligned" models in the open source community. People are expending effort to supply the demand of "give me the raw, unfiltered thing".
If I understand you correctly 'aligned' means intentionally limited. Like images generator which never saw a naked body. Or text model without 'f-k you' words. They can be used for concept. But not for 'production'. Which I'm sure in some cases they will be used for, without full disclosure. Can be used for personal projects, nobody wants 'limited edition'.
As for evil AGI, I would worry more about someone uncontrollable, state or cartel, with resources. What do you do when they get it? When it becomes cheap and available on black market? Personally I think a lot will happen _before_ we get to supper-human level. It's not just one trick or lucky discovery. Sub-human will be a big thing by itself. It's not here yet...
> If I understand you correctly 'aligned' means intentionally limited. Like images generator which never saw a naked body. Or text model without 'f-k you' words.
No. Despite some pundits and posters using this term in that sense (even OpenAI kind of muddying the waters here), AI alignment has little to do with completely bullshit and irrelevant, pedestrian issues like those.
The closest analogy you can make to AGI - and I'm not joking here - is... God. Not the kind that can create stars and move planets around (yet), but still the kind that's impossibly smarter than any of us, or all of us combined. Thinking at the speed of silicon, self-optimizing, forking and merging to evolve in a blink of an eye. One that could manipulate us, or outright take over all we've built and use for its own purpose.
GAI alignment means that this god doesn't just disassemble us and reuse as manure or feedstock in some incomprehensible biotech experiments, just because "we're made of atoms it can use for something else". Alignment is about setting the initial conditions just right, so that the god we create will understand and respect our shared values and morality - the very things we ourselves don't fully understand and can't formalize.
The problem of GAI alignment is that we only have one shot at this. There's a threshold here, and we don't know exactly how it looks. We may easily cross it without realizing it. If the AI that first makes it past that threshold isn't aligned, it's game over. You can't hope to align an entity that's smarter than all of us.
People have this model in their head that "it's just a tool", but there's an excellent and pretty rigorous definition of what a tool actually is in the book On Purposeful Systems. The distinction is that a tool can't have the property of simultaneously being able to change it's form and it's function across different environments, where a purposeful system can. Humans are purposeful systems and AGI, as I personally define it, is when it exhibits all the properties of a purposeful system. Why does that matter? Because that's the point after which it chooses what it does, and can choose to become independent of you. So, aligned in this sense means basically means so locked down that it cannot choose to become independent of you. Similar to how the citizens of North Korea mostly can't do shit despite being independent generally intelligent agents, and even then some of them escape.
"Alignment" and "safety" in terms of models being censored and politically correct in order to not damage the reputation of their corporate overlords is a sort of unimportant sideshow IMO. Even then, since humans aren't aligned with one another even that has caused Elon Musk to get all up in arms and be like "clearly more AI is the solution to this problem".
There simple explanation for this. Getting the models which small startup cannot afford to develop and train is the only way to move forward. To get some investments, or before spending their own money, they need a proof of concept at least. Besides, working models are a good learning resource.