People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code.

So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns.

I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch.

Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.

I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).

You'd likely get a model with very similar behaviour but the weights would be different.

Please someone correct me if I'm wrong.

It's mostly ad hoc scripts and some pretty horrible hacks being run by a hundred engineers trying to improve a thousand different things at once with a hundred thousand GPUs.

I'm sure once we understand the tech better, the training process will look like running a program.

No, it's the preferred way because that's literally how you train it. Contrary to popular misconceptions, you don't "compile" data into weights. You initialize a model (based on architecture, config, etc) and then you modify it via training. But crucially they (i.e. model creators) modify it the same way (technically speaking) as you would. That's what the license grants you. Nothing less, nothing more. The "how" as in knowhow has never been something covered by a license.

Open-weight is something dreamt up by people misunderstanding the basics of model creation and training, and having ideological biases against AI and/or LLMs. It is what it is, but you should know that you are technically wrong. A model released under an open source license (Apache, MIT, etc) is an open source model. Because the weights are the source of the models. Training is not "compilation". Training is the "how" as in knowhow to modify the model. Training deals with values. Source deals with everything, including values.

In the past, if someone would have released a piece of software (say a PID controller algo) w/ hardcoded values, no-one would bat an eye. LLMs are just that, with billions of hardcoded values. Nothing less nothing more.

I do prefer open weights as being more precise (like, is it even really software that has source code in the first place?) but I feel like at this point the ship has sailed somewhat (though if this is something you're willing to spend your time arguing then... moral support I guess?)

Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).