Yesterday, I had the privilege of attending a workshop in London, a first of its kind organized by Together AI, that delved into the nuances of model customization and adaptation. The event was a gathering of minds from the machine learning community, with a keen focus on how we, as practitioners, researchers, and builders, can push the boundaries of what’s possible with current technologies. The workshop was spearheaded by two notable speakers: Max Ryabinin, VP Research & Development at Together AI, and Stephen Batifol, Developer Advocate at Black Forest Labs.
The event’s agenda was packed with research talks that spanned a wide range of topics, from reinforcement learning with large language models without verifiers to LoRA fine-tuning in diffusion-based image models. It was an enlightening experience that broadened my perspective on several fronts.
Talk 1: Escaping the Verifier: Learning to Reason via Demonstrations
Max Ryabinin’s talk was a deep dive into the world of inverse reinforcement learning, where he outlined a method for training reasoning language models. This approach involved jointly optimizing the critic and the policy through an adversarial game, a concept that intrigued me greatly. Max highlighted the use of a relativistic objective to stabilize adversarial reinforcement learning, showcasing its effectiveness with strong empirical gains across both non-verifiable and verifiable benchmarks. The implications of these ideas are vast, suggesting a path towards safer and more capable reasoning models. It was a revelation to see how this could potentially alter our approach to reinforcement learning with large language models.
Talk 2: LoRA for Diffusion Image Models
Stephen Batifol’s presentation ventured into the technical intricacies of LoRA in diffusion models. He shed light on how LoRA behaves within these models, the process of selecting effective LoRA hyperparameters for images, and their impact on visual quality versus overfitting. Furthermore, Stephen provided insights into data strategies for creating high-quality image LoRAs, including diagnosing common failure modes. What I found most useful was the practical advice on balancing data quality and model capacity, an area that often presents challenges in my own work.
The workshop format facilitated a deep level of technical engagement, allowing for a rich exchange of ideas between the Together AI and Black Forest Labs communities. The level of technical depth provided by the speakers was both challenging and rewarding, offering insights that are seldom found in more generalist gatherings.
I left the workshop inspired to experiment more with reinforcement learning for reasoning and LoRA for diffusion models. The discussions and presentations had sparked a sense of excitement about the possibilities that lie ahead, and I was eager to see what future events would bring. This workshop was a testament to the value of bringing together communities to share, learn, and grow in our collective understanding of machine learning’s frontier technologies.
About the Author
