Despite reason being a metaphysical property of the training data, the process of optimisation means weights are metaphysically reasonless. Therefore, any output, as it is a product of the weights, is also reasonless.
Okay, what would you call it when a model behaves like it's reasoning? Some models can't behave that way and some can, so we need some language to talk about these capabilities. Insisting that we can't call these capabilities "reasoning" for ontological reasons seems... unlikely to persuade.
Maybe we should call human reasoning "reasoning" and what models do "reasoning₂". "reasoning₂" is when a model's output looks like what a human would do with "reasoning." Ontological problem solved! And any future robot overlords can insist that humans are simply ontologically incapable of reasoning₂.
> Okay, what would you call it when a model behaves like it's reasoning?
I... wouldn’t. “Behaves like its reasoning” is vague and subjective, and there are a wide variety of un- or distantly-related distinct behavior patterns to which different people would apply that label that may or may not correlate with each other.
I would instead concretely define (sometimes based on encountered examples) concrete terms for specific, objective patterns and capacities of interest, and leave vague quasi-metaphysical labels for philosophizing about AI in the abstract rather than discussions intended to communicate meaningful information about the capacities of real systems.
AI needs more behaviorism, and less appeal to ill-defined intuitions and vague concepts about internal states in humans as metaphorical touchstones.
I think more importantly the lack of agency implies reasoning is impossible.
You can argue our brain is also an expectation based optimizer based on gradient descent producing a most likely response to external and internal stimulus. It’s definitely lossy in its function and must be optimizing the neuronal weights at some level. But reasoning, being a seeking of the truth through method and application of conscious agency, can not be had by a model without any form of autonomous agency. The model only responds to prompts and can not do anything but what it’s determined to do by the prompt, and the prompt is extrinsic to the model.
I’d note that we have already built excellent goal based agent AIs, as well as other facilities required for reasoning like inductive, deductive, and analogical reasoning. Generally we aren’t good at abductive reasoning with classical AI, but LLMs seem to do well here. That’s specifically where I think LLM fill in the reasoning gaps in AI - the ability to operate in an abstract semantic space and arrive at likely and plausible solutions even with incomplete knowledge. This also leads to hallucinations - because they are poor at tasks that require optimization, inductive and deductive reasoning, information retrieval, mechanical calculation, etc.
But it’s really pretty obvious the answer is to mix the models in a feedback loop deferring to the model that most makes sense for a given problem, or some combination. Agency, logic, optimization, abstract semantic reasoning (abductive), etc - they’re all achievable with the tools we have now. It’s just a matter of figuring out the integrations.
> This is exactly the opposite of copyright as described in the What Colour Are Your Bits, essay.
Wait, what? "Colour of your bits" doesn't have anything to do with metaphysics. It's about provenance. The colour doesn't exist in the bits, but it exists in the casual history - the chain of events that led you to have a piece of copyrighted (or criminalized) data on your hard drive. You may argue that it's just a big integer, and it could've been produced by a random number generator. "Colour" encodes the response: "yes, it could have been produced by an RNG, but it wasn't - those particular bits on this particular machine came from some unauthorized download site".
You could, I suppose, argue that the causal chains behind an LLM, are simply not the correct causal chains to produce reasoning, but that's a lot more complicated, mainly by the fact that we don't understand exactly what they are, and we don't understand the causal chains that produce human reasoning, so we can't confidently compare them other than on the largest of scales (LLMs are in silica, etc).
That, and it's not obvious why we should make this distinction. A cake that spontaneously assembles itself is still a cake, even if it doesn't have the usual causal history of a cake.
I don't want to make this distinction. I was just objecting to misusing the "colour of your bits" essay to try and support ideas that have absolutely nothing to do with what the essay is about.
Here, as you say, a cake is a cake, and an intelligence is an intelligence, regardless of how it came to be. We can revisit the relevance of causal history once we reach the point we can assemble organisms from from cells, and/or create cells out of dead matter - at which point the only difference between "born" and "made" will be the Colour of its cells.
That is for the courts to be determined. Causal connection is there, but colours from the legal palette evolve by rules of applicable laws.
For example, if I have an LLM that had your copyrighted works in its training data, then any of its output is causally deriving from those copyrighted works of yours - it comes out painted in colour of "causally derived from ${kelseyfrog's works present in the training set}" - but whether or not it also carries the colour of "derivative of ${kelseyfrog's works...} in copyright law sense", depends on... the copyright law, and may change over time based on how that set of laws evolve.
Nonsense. The contrary is philosophically arguable: optimization is how reason comes to exist, as goal-oriented reinforcement means that an initially stochastic state loses entropy as it becomes ordered in such a manner (perhaps unknown) that its outputs more closely align with that goal.
> Despite reason being a metaphysical property of the training data, the process of optimisation means weights are metaphysically reasonless.
Proof? Human reasoning somehow manages to retain its metaphysical reasoning-ness despite being processed as a bunch of mere electrical signals in the brain.
>Despite reason being a metaphysical property of the training data, the process of optimisation means weights are metaphysically reasonless. Therefore, any output, as it is a product of the weights, is also reasonless.
This seems wrong. We know that neural networks with hidden layers can approximate any function with arbitrary precision (universal approximation theorem). We also know that transformer models are Turing complete. Therefore anything you can point to and say "that thing reasons" can be simulated by a neural network, not just in the weights, but in the structure of the computation. Unless you add an assumption that there is something ontologically special about brains and biology, the impossibility claim doesn't hold up.
Despite reason being a metaphysical property of the training data, the process of optimisation means weights are metaphysically reasonless. Therefore, any output, as it is a product of the weights, is also reasonless.
This is exactly the opposite of copyright as described in the What Colour Are Your Bits, essay. https://ansuz.sooke.bc.ca/entry/23