The DALL-E neural network allows you to create photorealistic images by recognizing the context of the user’s request. RBC Trends found out where the neural network can be applied and how not to compose requests
DALL-E is one of many neural networks capable of generating images based on text descriptions. However, unlike Midjourney and other competitors, it also takes into account the context of the description, which sometimes makes the AI’s results frighteningly realistic.
What is Dall-E
DALL-E is a machine learning model from OpenAI that generates images based on text descriptions.
The neural network is based on OpenAI’s developments related to text generators. In 2019, the company created a model called GPT-2, which could predict the next word in a text. It recognized 1.5 billion parameters and was trained on 8 million web pages. A year later, an improved model, GPT-3, was released, which became the basis for the creation of DALL-E. Essentially, the new neural network is a version of GPT-3 with 12 billion parameters, trained to generate anthropomorphic animals and people, objects, as well as plausibly combine unrelated concepts and transform existing images.
In March 2023, OpenAI introduced an even more advanced model, GPT-4, which recognizes not only text descriptions but also images. However, it has not yet been implemented as the basis for image generators.
The name of the neural network is a combination of the name of the artist Salvador Dali and the name of the robot WALL-E from the Pixar cartoon. The developers explain that it reflects the fusion of art and digital animation using artificial intelligence.
The first version of DALL-E was introduced in 2021. A year later, OpenAI presented an improved version, DALL-E 2, which offers higher image quality and new conversion capabilities. In addition, it supports queries in 107 languages, including Russian. DALL-E uses 12 billion parameters, while DALL-E 2 works with 3.5 billion and an additional 1.5 billion parameters to improve resolution.
In July 2022, DALL-E 2 was released for beta testing. In September, OpenAI opened access to the neural network to everyone. In November, the company made the neural network software available to application developers. At that time, it reported that DALL-E was already being used by more than 3 million people, and the neural network was generating more than 4 million images per day.
How Dall-E works
DALL-E uses a transformer-based neural network to generate images. This is a type of machine learning that understands context and processes sequences to create new images based on text prompts. The model is constantly learning from new data.
In total, DALL-E consists of three neural networks: CLIP (Contrastive Language–Image Pre-training), GLIDE, and a neural network for increasing image resolution. The first recognizes text and creates a sketch of the future image, the second converts it into a final low-resolution image, and the third scales the image and adds details.
Here’s how it works step by step:
CLIP translates the text query into a set of numbers that are linked by vectors. The vectors show how closely related the categories described by the user are to each other.
CLIP converts this set of numbers into a table, which serves as a draft image.
The table is passed to GLIDE, which converts the text into an image. The second neural network compares the initial set of numbers and the CLIP table, combining the data from them. It then creates the final image using a diffuse model. First, a gray square consisting of pixel noise appears, and then the noise is gradually removed from it until an image with the desired content emerges.
Capabilities of the Dall-E neural network
The OpenAI model can not only generate images on demand. It is also capable of:
- creating complex images by mixing different concepts;
- creating images similar to the original;
- mixing two images to get a third;
- This makes it possible to supplement well-known paintings and develop new plots based on them;
- change the composition, shadows, and texture of an image, add and remove objects;
- create photorealistic images;
- edit photos.
How to use Dall-E
DALL-E has potential applications in education, graphic design, media and marketing, architecture, and even research illustration.
The neural network is already being used in practice. Microsoft has released the Designer app for graphic designers with DALL-E integration, as well as the Image Creator extension for the Edge browser, which allows you to create images directly in it. The stock image service Shutterstock has started selling images created by DALL-E 2. And authors use it to generate images for texts on social networks and blogs. Dall-E has also been tested in solving non-standard tasks, such as generating game locations.