Current diffusion models for AI image generation do a great job of creating images by gradually reversing noise. They start with a noisy image and progressively clean it up, eventually generating high-quality visuals. But what if, instead of generating pixel-based images, the model could generate shader code—code that itself generates the image procedurally? This could open up a whole new way of thinking about AI-generated visuals. We don’t know if it would work, but it’s a direction worth exploring.

The Problem

While diffusion models can produce impressive visuals, they come with some issues:

• The process is hard to understand: When a model generates an image, it’s difficult to see or interpret what’s happening inside the neural network. We end up with pixels, but not much insight into how the image is constructed.

• Pixel-based generation lacks flexibility: Once an AI model generates an image, it’s fixed. It doesn’t give us the flexibility to dynamically manipulate or reuse the process that created it.

• Complexity of images isn’t handled well: Neural networks seem to encode some kind of program-like behavior in their latent space, but it’s not explicit or easy to extract or understand.

So, the question is: could we instead generate shader code—compact programs that generate images on the fly? Shaders are already used in graphics, gaming, and simulations for procedural content, and they offer a lot of flexibility and efficiency.

A Possible Solution

The key idea is to use large language models (LLMs), which are already good at generating code, to produce shader programs rather than pixels. Here’s how this might work:

1. Dual Loss Function: You’d train a model to generate shader code, optimizing for two things: (1) minimizing the visual difference between the shader-generated image and a target image (image loss) and (2) minimizing the complexity of the shader program (complexity loss). This would encourage the model to create simpler, more efficient shaders that still generate the correct image.

2. Leveraging LLMs for Shader Generation: Large language models like GPT-4 already understand code and can generate complex, structured logic. Fine-tuning them on shader code could enable them to output shader programs that create procedural images, not just raw pixels.

3. Using Multimodal Models: Models like CLIP, which already work with both images and text, could potentially be adapted for this. These models might help map visual features to shader logic, allowing for shader generation from text descriptions, visual inputs, or a combination of both.

Why This Approach Might Be Valuable

This approach isn’t just about generating the image—it’s about understanding how the image is constructed. If the model produces shader code, we get a tangible, reusable program rather than just a static image. The shader can then be modified, extended, or used in other contexts, making it far more flexible than pixel-based outputs.

It could also help with compression. A shader program is typically much smaller than a fully rendered image, but can generate equally complex visuals. So, in a way, this approach could offer a kind of efficient, procedural compression.

And if we think about what’s happening in neural networks today, we could argue that these models are already generating something like implicit “programs” in their hidden layers. We just can’t easily extract them. By generating explicit shader code, we might be able to lift this hidden knowledge out and make it more useful and understandable.