-
Research: Meta paper on The Art of Scaling Reinforcement Learning Compute for LLMs
Read more: Research: Meta paper on The Art of Scaling Reinforcement Learning Compute for LLMsResearchers from Meta with university researchers from UT Austin, UCL, UC Berkeley, Harvard, along with Periodic Labs, posted on arXiv a paper on scaling in reinforcement learning (instead of pre-training). One of their key findings is that “(3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs.” ABSTRACT EXCERPT Related Stories I…
-
Google DeepMind research paper on Generative Data Refinement to cleanse undesirable content
Read more: Google DeepMind research paper on Generative Data Refinement to cleanse undesirable contentGoogle DeepMind posed a preprint paper on “Generative Data Refinement: Just Ask for Better Data.“ Abstract: For a fixed parameter size, the capabilities of large models are primarily determined by the quality and quantity of its training data. Consequently, training datasets now grow faster than the rate at which new data is indexed on the…
-
Paper on The Illusion of Diminishing Returns [of scaling]
Read more: Paper on The Illusion of Diminishing Returns [of scaling]Fascinating research paper refuting the notion that scaling of datasets has diminishing returns in the performance of large language models. While that might be true for simple “single-step” tasks, it is not for more complex “long horizon” tasks, according to the following researchers (Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, Jonas Geiping), who posted…