Understanding Context Windows
rkayg.comI think degraded performance as the context window increases is one of the largest issues models are facing today. Do you think we've reached the end of any "RAM" based context performance and that external storage/ retrieval mechanisms is the only way forward?
I think we're in the early innings of RAM implementations. I also think RAM and external storage / retrieval are kind of the same thing in that efficient RAM implementation works best w/ another storage system. I hope to do a deeper dive here, but in the mean time check out https://arxiv.org/abs/2310.08560