Red Hat Blog: IBM's Granite foundation model: A detailed look at its training data

The Granite 13b model is the base model from which all other variants of Granite were fine-tuned for specific tasks. However, version 2 of the 13b model, granite.13b.v2, had an additional pretraining on 1.5T new tokens that were deemed usable after going through the data processing pipeline seen below. Upon adding these tokens to version one of the model we now have 2.5T Tokens used in the training of version 2. …