Releasing glaive-coder-7B and the Code Models Arena

We are releasing a new open-source code model glaive-coder-7b which is a 7B parameter code model trained on a dataset of ~140k programming related problems and solutions generated from Glaive’s synthetic data generation platform.

The model is trained to act as a code assistant, and can do both single instruction following and multi-turn conversations unlocking a complete code assistant experience which users can run locally.

We also release the data used to train the model which is licensed as apache-2.0 allowing the community to further improve and build on top of the data.

Code Arena and Benchmarks

The model achieves a 63.1% pass@1 on HumanEval and a 45.2% pass@1 on MBPP, however it is evident that these benchmarks are not representative of real-world usage of code models so we are launching the Code Models Arena to let users vote on model outputs so we can have a better understanding of user preference on code models and come up with new and better benchmarks. We plan to release the Arena results as soon as we have a sufficient amount of data.

Join the Glaive discord for improvement suggestions, bug-reports or if you'd like to use Glaive to get a model for your own use-case.