Japanese StableLM is a 7 billion-parameter general-purpose language model. It stands as the top-performing publicly available Japanese language model, according to a benchmark suite against four sets of other Japanese LMs.
Japanese StableLM Base Alpha 7B will be released under the commercially available Apache License 2.0. Japanese StableLM Instruct Alpha 7B is a model created for research purposes and is released exclusively for research use. For details, please refer to the Hugging Face Hub page.
“We are proud of our first big step towards contributing to the Japanese generative AI ecosystem,” said Meng Lee, Project Lead of Japanese StableLM. ”We look forward to continuing to create models across several modalities, built specifically to reflect Japanese culture, language and aesthetics”.
Japanese StableLM Base Alpha 7B
Japanese StableLM Base Alpha 7B is trained for text generation using large-scale data sourced mainly from the Web. The training data is predominantly composed of Japanese and English text, with the remaining 2 percent of material in the form of source code.
In addition to open datasets, the training data includes datasets created by Stability AI Japan and datasets created with the cooperation of the Japanese team of the EleutherAI Polyglot project, along with members of Stability AI Japan's community.
For training, we used software that is an extension of EleutherAI's GPT-NeoX. For example, the model architecture incorporates new technologies such as SwiGLU and xPos. A cumulative total of 750 billion tokens were processed across epochs.
Japanese StableLM Instruct Alpha 7B
The Japanese StableLM Instruct Alpha 7B model is a language model that is additionally tuned to follow user instructions.
Supervised Fine-tuning (SFT) was employed for the additional training, and multiple open datasets were used. As discussed below, SFT also significantly improves the performance evaluation score by lm-evaluation-harness.