Amidst a discovery dispute, OpenAI filed an interesting declaration from Nick Ryder, VP of Research, Foundations, OpenAI. In it, Ryder reveals the frequency that researchers at OpenAI are developing models that are never commercially released.
Since it was founded in 2015, OpenAI has created hundreds of thousands of artifacts that might each be called “models.” OpenAI, for example, has employed many hundreds of AI researchers (many of whom are no longer OpenAI employees), each of whom has conducted many machine learning experiments. Collecting useful information, including training data and
documentation, about each of these research artifacts would be an incredibly complicated and burdensome exercise and would take months if not years of work, if even possible.
He also reveals that 2 Internet-based books corpora were used to train GPT 3 and 3.5, but not any subsequent OpenAI models.