About the Animations
Return to Main Rimbaud Page
The AI animations which accompany the performance of Rimbaud: This Fugitive Soul are an experiment: Is it possible to use the text of the supertitles to generate a real-time visualization to complement the music? The answer seems to be yes, but we’re crossing our fingers that it works! For those who are interested, here are some details.
The supertitle lines are defined in a text file. A program running on a souped-up Mac Studio uses this text file as input. When the operator presses the right arrow key, the next supertitle is displayed. Simultaneously, the supertitle passed as the new prompt to a continuously-running image generator using the Stable Diffusion platform (model version 1.5, at a resolution of 512x512 — the only size that runs fast enough). This generator outputs images as fast as it can. Since this isn’t actually all that fast, the images are then blended together to create intermediate frames that make the animation smoother. The operator can also press another key to turn off the current supertitle (in instrumental passages, for instance). But the program continues to use the last supertitle for image generation until a new one appears.
The style of the images is set by asking Stable Diffusion to create everything in the style Odilon Redon, a French artist famous for his otherworldly works and a slightly older contemporary of Rimbaud. When the text is in French, we use that rather than the English. It wasn’t clear whether Stable Diffusion’s language model would understand French, but it seems to work fine. Perhaps its training gave it familiarity with Rimbaud’s poetry.
Although in general we are just passing the supertitles to the machine and seeing what happens, there is one tweak to improve the results. Most of the time, successive lines simply modify the image that came before, resulting in smoothly varying animation. But every so often, the AI is instructed to start afresh with a new image. You can see when this happens; the animations fade to an entirely new look. There are technical reasons (having to do with color balance) to restart every so often, but we chose to restart on supertitle lines that seemed most likely to generate compelling imagery.
Because the images are being generated in real-time, we don’t actually know what will come out. So if something inappropriate comes out of the mind of the AI, we sincerely apologize! We will be too busy singing to notice, so let us know what you see.
The supertitle lines are defined in a text file. A program running on a souped-up Mac Studio uses this text file as input. When the operator presses the right arrow key, the next supertitle is displayed. Simultaneously, the supertitle passed as the new prompt to a continuously-running image generator using the Stable Diffusion platform (model version 1.5, at a resolution of 512x512 — the only size that runs fast enough). This generator outputs images as fast as it can. Since this isn’t actually all that fast, the images are then blended together to create intermediate frames that make the animation smoother. The operator can also press another key to turn off the current supertitle (in instrumental passages, for instance). But the program continues to use the last supertitle for image generation until a new one appears.
The style of the images is set by asking Stable Diffusion to create everything in the style Odilon Redon, a French artist famous for his otherworldly works and a slightly older contemporary of Rimbaud. When the text is in French, we use that rather than the English. It wasn’t clear whether Stable Diffusion’s language model would understand French, but it seems to work fine. Perhaps its training gave it familiarity with Rimbaud’s poetry.
Although in general we are just passing the supertitles to the machine and seeing what happens, there is one tweak to improve the results. Most of the time, successive lines simply modify the image that came before, resulting in smoothly varying animation. But every so often, the AI is instructed to start afresh with a new image. You can see when this happens; the animations fade to an entirely new look. There are technical reasons (having to do with color balance) to restart every so often, but we chose to restart on supertitle lines that seemed most likely to generate compelling imagery.
Because the images are being generated in real-time, we don’t actually know what will come out. So if something inappropriate comes out of the mind of the AI, we sincerely apologize! We will be too busy singing to notice, so let us know what you see.
