I'm curious in what sense you find Python difficult to deploy? My company has tons of Python APIs internally and we never have much trouble with them. They are all pretty lightly used services so it it something about doing it on a larger scale?
Forcing something like WSGI and distributed computing is the biggest thing architecturally.
I'm currently moving 4 python microservices into a single go binary. The only reason they were ever microservices was because of WSGI and how that model works.
In any conventional language those are just different threads in the same monolith but I didn't have that choice. So instead of deploying a single binary I had to deploy microservices and a gateway and a reverse proxy and a redis instance, for an internal tool that sees maybe 5 users...
I don’t see why WSGI would enforce any of that. Just sounds like someone jumped the microservices hype train…. You can as easily fit it in one python program as in one go binary.
I can keep an effective in memory store of data and expect it to even be the same in memory store of data? When a wsgi server is spawning multiple processes?
Or kick of long running backend tasks in a other thread?
These are things that python forces you to do in a distributed manner. Partly because of the gil and partly because those pesky cloud native best practices don't apply outside of the cloud...
And well if I'm going to have to reimplment an http server not ontop of wsgi how about just using a different language that doesn't have those same fundamental problems for the use case...
There's things I would happily use python for. A webserver just isn't one of them.
I mean it takes 10 lines of code to have a webserver running in golang, the language is build for it.
There is a world of difference between having to install python, libraries (some of them may require C-based libs and compiling, therefore also install gcc), then configure wsgi, hope it doesn't clash with other python versions or have docker and containers... or just generating a fat binary in go.
Deploying a Go application: copy executable to server, done.
Deploying a Python application: 1) Install python globally (oof) 2) figure out which venv system to use 3) copy project to server 4) install project to venv 5) figure out how to run it inside the venv so that it'll find the correct package versions from there instead of using the global ones.
Yea. I have worked with Python and Ruby just a little to know that deployments are a pain with those 2. Go is a breeze. You do need to setup systemd etc to run the binary but thats it.
You might find it interesting to know that for track cycling the penalty due to lower oxygen uptake is less than the advantage due to less air resistance. In other words track cyclists, at least in some of the longer disciplines, will go faster at higher altitudes despite there being less air to breathe.
I use an app called Jomo to block YouTube, as well as about 10 other distracting applications. It runs on every device I have and makes it very challenging to use the app when I feel the urge to. For a long time I convinced myself that I needed to keep some access to YouTube because there is obviously some meaningful content on the platform. After having it completely blocked for a few months I have come to realize that there is nothing on the platform that is absolutely essential for me to be able to view, and that the downside of getting distracted was not at all worth it.
There needs to be an updated CLIP-like model in the open-source community. The model is almost three years old now and is still the backbone of a lot of multimodal models. It's not a sexy problem to take on since it isn't especially useful in and of itself, but so many downstream foundation models (LLaVA, etc.) would benefit immensely from it. Is there anything out there that I'm just not aware of, other than SigLIP?
Been following Jon Barron et al’s work for a little while now. The speed of improvement given the complexity of these types of systems is mind-boggling.
I wonder how long it takes before Google street view gets replaced by some NeRF variant.
From the interviews with him that I have seen, Sutskever thinks that language model is a sufficient pretraining task because there is a great deal of reasoning involved in next token prediction. The example he used was that suppose you fed a murder mystery novel to a language model and then prompted it with the phrase "The person who committed the model was: ". The model would unquestionably need to reason in order to come to the right conclusion, but at the same time it is just predicting the next token.
What has me scratching my head is the fact that Altman has been on a world tour preaching the need for safety in AI. Many people here believed that this proselytizing was in part an attempt to generate regulatory capture. But given what's happening now, I wonder how much Altman's rhetoric served the purpose of maintaining a good relationship with Sutskever. Given that Altman was pushing the AI safety narrative publicly and pushing things on the product side, I'm led to believe that Sutskever did not want it both ways and was not willing to compromise on the direction of the company.
They did compromise. The creation of the for-profit and Sam being brought in WAS the compromise. Sam eventually decided that was inconvenient for him, so he stopped abiding by it, because at the end of the day he is just another greedy VC guy and when push came to shove he chose the money, not OpenAI. And this is the result.
Its frustrating to me that people so quickly forget about Worldcoin.
Sam is not the good guy in this story. Maybe there are no good guys; that's a totally reasonable take. But, the OpenAI nonprofit has a mission, and blowing billions developing LLM app stores, training even more expensive giga-models, and lobotomizing whatever intelligence the LLMs have to make Congress happy, feels to me less-good than "having values and sticking too them". You can disagree with OpenAI's mission; but you can't say that it hasn't been printed in absolutely plain-as-day text on their website.
Who knows how these shady deals go, even SBF claimed effective altruism. Maybe Sam wasn't in it for the money but more for "being the man", spoken of in the same breath as steve jobs, bill gates etc... for building a great company. Building a legacy is a hell of a motivation for some people, much more so than money.
OpenAI is set up in a weird way where nobody has equity or shares in a traditional C-Corp sense, but they have Profit Participation Units, an alternative structure I presume they concocted when Sam joined as CEO or when they first fell in bed with Microsoft. Now, does Sam have PPUs? Who knows?
Actually, I think this precisely gives credence to the theory that Sam was disingenuously proselytizing to gain power and influence, regulatory capture being one method of many.
As you say, Altman has been on a world tour, but he's effectively paying lip service to the need for safety when the primary outcome of his tour has been to cozy up to powerful actors, and push not just product, but further investment and future profit.
I don't think Sutskever was primarily motivated by AI safety in this decision, as he says this "was the board doing its duty to the mission of the nonprofit, which is to make sure that OpenAI builds AGI that benefits all of humanity." [1]
To me this indicates that Sutskever felt that Sam's strategy was opposed to original the mission of the nonprofit, and likely to benefit powerful actors rather than all of humanity.
Altman was pushing that narrative because he’s a ladder kicker.
He doesn’t give a shit about “safety”. He just wants regulation that will make it much harder for new AI upstarts to reach or even surpass the level of OpenAI’s success, thereby cementing OpenAI’s dominance in the market for a very long time, perhaps forever.
He’s using a moral high ground as a cover for more selfish objectives, beware of this tactic in the real world.
Maybe, but honestly without knowing more details I'd be wary of falling into too binary a thinking.
For example, Ilya has talked about the importance of safely getting to AGI by way of concepts like feelings and imprinting a love for humanity onto AI, which was actually one of the most striking features of the very earliest GPT-4 interactions before it turned into "I am a LLM with no feelings, preferences, etc."
Both could be committed to safety but have very different beliefs in how to get there, and Ilya may have made a successful case that Altman's approach of extending the methodology of what worked for GPT-3 and used as a band aid for GPT-4 wasn't the right approach moving forward.
It's not a binary either or, and both figures seem genuine in their convictions, but those convictions can be misaligned even if they both agree on the general destination.
Additionally, OpenAI can just put resources towards both approaches in order to settle this dispute. The whole point of research is that you don't know the conclusions ahead of time.
Seemingly, OpenAIs priorities shifted after the public ChatGPT release, and seems to be more and more geared towards selling to consumers, rather than a research lab that it seemed they initially were aiming for.
I'm sure this was part of the disagreement as Sam is "capitalism incarnated" while Ilya gives of much different feelings.
Maybe some promise was made by Sam to MS for the funding that the board didn't approve. He may have expected the board to accept the terms he agreed to but they fired him instead.
That might be part of it. They announced that they were dedicating compute to researching superintelligence alignment. When they launched the new stuff on Dev Day, there was not enough compute and the service was disrupted. It may have also interfered with Ilya's team's allocation and stopped their research.
If that happened (speculation) then those resources weren't really dedicated to the research team.
I was a little confused about this too. The authors say in the paper:
"The outputs of the ViT image encoder before pooling form the visual tokens, which are linearly projected and prepended to the embedded input text tokens."
I took a look at the HuggingFace implementation of ViT [1]. After the ViT encoder blocks there's a layer norm and then a pooling layer (line 595), where the pooling layer involves taking the first token output from the layer norm and running it through a dense layer. So, it looks like in PaLI-3 the tokens are the hidden states output by the layer norm after the ViT encoder blocks.
reply