Mistral just announced at
@SHACK15sfthat they will release a new model today: Mistral 7B v0.2 Base Model - 32k instead of 8k context window - Rope Theta = 1e6 - No sliding window
@pi's explanation if you need it:
Excited for Mixtral-8x7B to drop the sliding window attention and have an extended context length when it’s inevitably trained on this model!
