Carding Forum
Professional
- Messages
- 2,788
- Reaction score
- 1,187
- Points
- 113
Researchers from the University of Nijmegen (the Netherlands) have prepared an openness rating for 40 large language models and 7 models for generating images based on text descriptions, which are declared open by manufacturers. Due to the fact that the criteria for openness of machine learning models are still being formed, there is currently a situation where models that have a license that restricts the scope of use are distributed under the guise of open ones (for example, many models prohibit use in commercial projects). Also, manufacturers often do not provide access to the data used in training, do not disclose implementation details, or do not fully open the accompanying code.
Most models that are positioned as "open" should actually be interpreted as "open weighting factors" or, more precisely, "available weighting factors", since they are distributed under restrictive licenses that prohibit use in commercial products. Third-party researchers can experiment with such models, but do not have the opportunity to adapt the model to their needs or inspect the implementation. More than half of the models do not provide detailed information about the data used for training, nor do they publish information about the internal structure and architecture.
The most open-source models are BloomZ, AmberChat, OLMo, Open Assistant, and Stable Diffusion, which are published under open licenses along with the source data, code, and API implementation. Models from Google (Gemma 7B), Microsoft (Orca 2) and Meta (Llama 3), positioned by manufacturers as open-source, were closer to the end of the rating, as they do not provide access to source data, do not disclose technical implementation details, and distribute model weights under licenses that limit the scope of use. The popular Mistral 7B model was approximately in the middle of the rating, as it is delivered under an open license, but is only partially documented, does not disclose the data used in training, and does not have fully open accompanying code.
The researchers proposed 14 criteria for the openness of AI models, covering the conditions for code distribution, training data, weights, data variants, and coefficients optimized using reinforcement learning (RL), as well as the availability of ready-to-use packages, APIs, documentation, and a detailed description of the implementation.
In accordance with the draft definition of open AI proposed by the OSI (Open Source Initiative), the main criteria for the openness of an AI system are to provide opportunities for use for any purpose without the need for separate permission; to study the operation of the system and inspect its components; to make changes for any purpose; to transfer to others both the original version and editorial offices after making changes.
To enable changes to be made, the AI system must include:
* Detailed information about the data used in training and the training methodology. This information should be sufficient for a professional developer to recreate an equivalent AI system on their own, using the same or similar data for training.
* The presence of source code that allows you to both launch the AI system and complete its training process (in the table discussed above, in the "code" column, many models indicate"~", which implies partial availability of code, when code is available to launch the model, but there is no code for training or creating the model). The code should also cover areas such as preprocessing, data validation, and tokenization. In addition, a detailed description of the model architecture must be provided.
* Model parameters (weight coefficients), which imply the presence of a ready-to-use state slice after training or the presence of the final optimized version of the model.
Most models that are positioned as "open" should actually be interpreted as "open weighting factors" or, more precisely, "available weighting factors", since they are distributed under restrictive licenses that prohibit use in commercial products. Third-party researchers can experiment with such models, but do not have the opportunity to adapt the model to their needs or inspect the implementation. More than half of the models do not provide detailed information about the data used for training, nor do they publish information about the internal structure and architecture.
The most open-source models are BloomZ, AmberChat, OLMo, Open Assistant, and Stable Diffusion, which are published under open licenses along with the source data, code, and API implementation. Models from Google (Gemma 7B), Microsoft (Orca 2) and Meta (Llama 3), positioned by manufacturers as open-source, were closer to the end of the rating, as they do not provide access to source data, do not disclose technical implementation details, and distribute model weights under licenses that limit the scope of use. The popular Mistral 7B model was approximately in the middle of the rating, as it is delivered under an open license, but is only partially documented, does not disclose the data used in training, and does not have fully open accompanying code.
The researchers proposed 14 criteria for the openness of AI models, covering the conditions for code distribution, training data, weights, data variants, and coefficients optimized using reinforcement learning (RL), as well as the availability of ready-to-use packages, APIs, documentation, and a detailed description of the implementation.
In accordance with the draft definition of open AI proposed by the OSI (Open Source Initiative), the main criteria for the openness of an AI system are to provide opportunities for use for any purpose without the need for separate permission; to study the operation of the system and inspect its components; to make changes for any purpose; to transfer to others both the original version and editorial offices after making changes.
To enable changes to be made, the AI system must include:
* Detailed information about the data used in training and the training methodology. This information should be sufficient for a professional developer to recreate an equivalent AI system on their own, using the same or similar data for training.
* The presence of source code that allows you to both launch the AI system and complete its training process (in the table discussed above, in the "code" column, many models indicate"~", which implies partial availability of code, when code is available to launch the model, but there is no code for training or creating the model). The code should also cover areas such as preprocessing, data validation, and tokenization. In addition, a detailed description of the model architecture must be provided.
* Model parameters (weight coefficients), which imply the presence of a ready-to-use state slice after training or the presence of the final optimized version of the model.
