In partnership with

Hello readers, in this issue we cover

• Google creates a model that can generate games in real-time by predicting the next frame

• In-depth review of music generation models

• Nvidia creates a method to generate high fidelity image with less parameters

• Meta creates a single model to generate both text and images

• Cohere introduces the concept of Multilingual Arbitrage that generates superior synthetic data

• Google introduces statistical framework to understand how retrieval augmented models behave

🕹️ Generating Games in Real-Time with Diffusion Models

GameNGen, developed by Google Research, is the first game engine powered entirely by a neural model, capable of real-time interaction with complex environments, such as simulating the classic game DOOM at over 20 frames per second on a single TPU. Its next frame prediction achieves a quality comparable to lossy JPEG compression. Human raters find it difficult to distinguish between real and simulated game clips. GameNGen is trained in two phases: first, an RL agent learns to play the game and records the sessions; second, a diffusion model is trained to generate the next frame based on past frames and actions.

Read the paper

🎵 Generating music with Music Foundation Models

This paper examines SOTA pre-trained models in music, covering areas such as representation learning and generative learning, and discusses the potential of these models in music understanding and generation while also exploring important topics like instruction tuning and long-sequence modeling.

If you’re looking to understand the current state of music generation, this paper is a good starting point.

Read the paper

🌅 Nvidia creates new image generation model that requires less parameters

As the cost of models and parameters increase when creating models, researchers are searching for efficiencies to improve scalability while improving the model’s capabilities.

In this paper, Nvidia created a new model called DiffiT using a new methodology for generating high-fidelity images with better parameter efficiency. It shows state-of-the-art performance while using up to 20% less parameters.

Read the paper

🏞️ Transfusion: Meta’s new technique for single models to generate both text and images

Transfusion, created by Meta researchers, is a new recipe for training a multi-modal models for both text and image generation. Previously, these were two separate models.

Transfusion models with up to 7B parameters were trained on a mixture of text and image data. These models scale much better and perform on par with similar scale diffusion and language models.

Read the paper

🗣️ Cohere introduces Multilingual Arbitrage - a method to generate superior synthetic data

Multilingual Arbitrage, a technique created by Cohere, capitalizes on performance variations between multiple models for a given language to generate a superior synthetic data training set. This technique strategically routes samples through a pool of different language models, each with unique strengths in different languages.

These techniques allow for spectacular gains compared to state-of-the-art models, with up to 56% improvement in win rates.

Read the paper

🐕 Google introduces framework to understand retrieval augmented models

Modern machine learning models often use extra information to improve their predictions. However, the best way to design and train these retrieval-augmented models isn't fully understood. This paper introduces a new framework to explore these models, which have two parts: one that finds relevant information, called the retriever, and another that uses it to make better predictions, called a predictor. It also presents a new method for training both parts together and examines their impact on model performance.

Read the paper or watch the YouTube video

❤️ Our Sponsors

Our work is supported 1440 Media. It’s a publication that prides itself on objective, fact-based news. No bias or misreporting like other publications.

The Daily Newsletter for Intellectually Curious Readers

• We scour 100+ sources daily

• Read by CEOs, scientists, business owners and more

• 3.5 million subscribers

Sign up today!

🤯 Today I Learned

Every issue, we highlight new AI concepts and terminology to help educate our readers. This issue we learned about:

Ensemble

In machine learning, an ensemble is a method that combines multiple models to improve the overall performance. By using a group of models rather than a single one, ensembles can reduce errors, increase accuracy, and make more reliable predictions. Common ensemble techniques include bagging (like Random Forests), boosting (like Gradient Boosting Machines), and stacking. The idea is that a collection of diverse models will perform better than any individual model.

Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) are a type of neural network inspired by a mathematical theorem called the Kolmogorov-Arnold representation theorem. KANs are presented to be an alternative to Multi-Layer Perceptrons.

KANs have learnable activation functions, compared to the fixed activation functions of multi-layer perceptrons

Retrieval Augmented Models

Retrieval-augmented models combine traditional machine learning or deep learning methods with a retrieval mechanism. This model retrieves relevant external information (like documents, facts, or text snippets) from a large dataset to augment its input data before making predictions or generating responses. This approach improves accuracy, relevance, and context-awareness by leveraging external knowledge not contained within the model itself.