Interesting read. Was surprised to learn how much damage can be done to a model's parameters without making any discernible difference in its quality of output.
I didn't see any mention of dropout in the article, during training parameters or whole layers are removed in different places which helps force it into a distributed representation.
I think this will be the inevitable outcome. I wrote a post [1] about this a few months back explaining how (I think) the long shot of AI adoption is an extreme dumbing down to the point where humans no longer know how to organically build or maintain what they need without AI.
This general direction of things is quite disheartening. The move away from small to large orgs dominating is exactly why modern life feels like war. Corporate, impersonal, manufactured, dead.
I don't see a move back to a "smaller" world any time soon, but I'm glad people are talking about this (and the downsides of your only options rapidly being conglomerates or big institutions).
Because we all sold out to the man. Culturally, we have chosen the lavish life promised under the man's umbrella, to doing the work of trying to go our own way. We now reap what we've sown.
The centralization of power also means the leaders of those large organizations have disproportionate power. Everyone is looking for the singular strongman at the head of an organization with nation-level power to save them from current turmoil.
> I don't see a move back to a "smaller" world any time soon
I do! Unironically: AI assisted software development – and please, we can call that anything else, we do not need to confuse it with Serious software development.
Just the amount of super simple software (Apps Script, Office Script) that baseline tech savy people can now/soon build to enhance what they think their business needs are, without the impossible constraint of having to pay a dev to find it out for/with them (because that is really not how you can find that out, while you find out everything else about your super small business) gives me a lot of hope here.
There is a chance that local models get good enough and efficient enough that we won't need the large companies, so much as a reasonable graphics card.
I recently moved away from as much big tech as possible. Canceled Spotify, won't order anything from Amazon, deleted Instagram, trying not to watch as much YouTube Videos etc. Sadly cant move away from WhatsApp and Google yet...
Instead, I am sitting here right now working on a blogging engine so I can create personal blogs to let my friends keep up to date with my shenanigans. Basically give them a chance to participate in my life without enabling them to doom scroll.
I really hope its not only me growing tired of all these addictive unhealthy apps and subscriptions that sneaked into most peoples everyday life. I can only recommend boycotting big tech with CEOs only caring about their own enrichment.
Its only the internet part of life, but this is where I spend most of my time. In real life I try to buy from the local stores as much as possible. However, I do not participate in many other smaller organizations...
Really enjoyed this! Thank you for sharing your approach and notes on process.
This framework makes a lot of sense. I think the general approach of eschewing one-shot prompts in favor of letting the LLM craft its own prompts (per your standards) is the missing piece in a lot of workflows.
This is a good thing. That shift in expectations will hopefully prevent a total collapse of systems as inexperienced developers would have yeeted god knows what code into production.
The best part: all of the materials that were available to now-seniors are available to new juniors. The added advantage of having an LLM to explain or clarify things? Seniors didn't have that. We had to suffer.
Sorry, but the outcome is fair. Seniors had to suffer to learn and can now benefit from the speed up of AI. Juniors don't have to suffer to learn, but they do have to learn to leverage AI to help them catch up to the existing crop of seniors.
Not impossible. Not impractical. Just a different form of hard work, which, no matter how much anyone kicks and screams will always be the thing standing in the way of where you want to be.
1. Assume that any model will start to lose focus beyond 50K-100K tokens (even with a huge context window).
2. Be gluttonous with chats. At the first sign of confusion or mistakes, tell it to generate a new prompt and move to a new chat.
3. Write detailed prompts with clear expectations (from how the code should be written to the specific implementation that's required). Combine these with context like docs to get a fairly consistent hit rate.
4. Use tools like Cline that let you switch between an "Act" and "Plan" mode. This saves a ton of tokens but also avoids the LLM getting stuck on a loop when it's debugging.
The above approach helped me to implement a full-blown database wrapper around LMDB for Node.js in ~2 weeks of slow back-and-forth (link to code in post for those who are curious).
Don't. Motivation external to the self will always be ephemeral and fleeting, like a drug (hence the popularity of "motivational" content on YouTube). If you don't have a genuine desire to do the thing, don't do it. Not because you can't do it, but because you lack the internal drive and inspiration to do it (and that may just be right now, not permanently).
Yes, extrinsic vs intrinsic motivation. Sometimes, you MUST do a task (taxes, take a driving test, etc).
And, yes, the extent that tasks start from intrinsic motivation, you're ahead of the game. For me, this is making habits (personal example: eating less). Habit frees you (mostly) of the need for any sort of 'motivation' - it's just Habit.
reply