These image generation models can convert a simple image description like

“A car on a road, sci-fi style”

into beautiful photo-realistic photographs like this

In this article, we are going to take a deep dive into how the stable diffusion (which is the current state-of-the-art image generation AI) model works.

How do diffusion models work?

Diffusion Models are a type of generative model which means they are built to generate outputs similar to ones they have been trained on.

There are two different types of diffusion processes:

Forward Diffusion

This process progressively adds noise to the image until it is converted into uncharacteristic noise. Uncharacteristic noise means you can’t tell whether the original image was a dog, a cat or maybe even a car. This is a very important step in the process.

Reverse Diffusion

This is the fun part! The reverse diffusion process tries to reconstruct the original training image from the noisy image we got from Forward Diffusion.

The reconstructed image isn’t always the same as the original as there is randomness that comes into play here. Now we know what the model needs to do. But the question is HOW 🤔

Training Process

To reconstruct the diffusion process, we need to find out how much noise was added to the image. To achieve this we train a neural network to predict the noise that was added. It is called a Noise Predictor which is a U-Net Model.

The steps are as follows:

• Pick a random training image

• Generate some noise

• Add this noise to the training image for a certain number of steps

• Teach the noise predictor how much noise was added

🔥 Now we have a fully trained noise predictor

Inference

To use this noise predictor, we generate a new noisy image. The noise predictor estimates how much noise was added and then removes the noise from the image. We repeat this process for a specified number of sampling steps.

The above process we discussed is running in pixel space (holds data for the red, green and blue channels for every single pixel). A single image of resolution 512x512 has 786,432 dimensions to it. Running these computations in pixel space is very, very slow and requires several GPUs at a minimum to run 🤯. But, Stable Diffusion has a trick up its sleeve 👇

Stable Diffusion

Latent Space

To overcome the computational speed issues, we have what are called Latent Diffusion Models such as the Stable Diffusion family of models which compress the high dimensional space from pixel space into something called latent space.

Latent Space is 48x smaller than pixel space which makes it exponentially faster to run which unlocks the ability to run inference on a single GPU with decent speeds.

Variational Autoencoder (VAE)

To perform the compression we use a technique called the variational autoencoder (VAE). It consists of two parts:

1. Encoder - It handles compressing the image from pixel space to latent space

2. Decoder - It handles converting the image from latent space back to pixel space

   

The image is compressed into latent space without any loss of information. This is possible since natural images are not random. For example, faces follow a certain spatial relationship between eyes, nose and other features. You can read more about this here → Manifold Hypothesis.

Inference in Latent Space

The inference in latent space is mostly the same as the one in pixel space except, a random latent space matrix is used instead of the generated noise image. An additional VAE Decoder step is also added after inference completes to convert the latent matrix back into a regular image (pixel space) which is our final generated image.

Conditioning

In the steps we discussed above, we never specified what we wanted the model to generate. Telling or “conditioning” the model to generate a certain kind of desired result is known as conditioning.

There are several types of conditioning such as:

• Text conditioning (aka prompting)

• inpainting

• outpainting

• controlnets

• and more….

In this article, we will only look into text conditioning which is the most widely used conditioning method and it is also used in several other conditioning methods.

Text Conditioning

Here is a high-level view of how the text prompts are processed and fed into the noise predictor (U-NET). This might look familiar to some who know about the transformer model architecture for LLMs.

Tokenizer

The text prompt is tokenized using the CLIP tokenizer. Tokenization allows the model to understand the prompt without having to understand “words”. Each word doesn’t always correspond to a single token. One word may consist of multiple tokens as well.

Embedding Model

The embedding model converts these tokens into vectors. Each vector has a unique fixed vector embedding which is learned by the embedding model when it was trained. Vector embedding allows computers to understand how semantically similar two tokens using the distance between any two vectors. Stable Diffusion used OpenAI’s ViT-L/14 CLIP Model.

Text Transformer

The Text Transformer is the final step in the pipeline for processing the text prompt. It serves as an adapter for other conditioning methods. The inputs to the transformer are not limited to text, it can include images, depth maps and a variety of other conditioning inputs.

Cross-Attention Mechanism

The Noise Predictor (U-NET) ingests the output of the text transformer via a cross-attention mechanism. It has two parts to it:

(a) Self Attention (within the prompt)

Assume the text prompt is as follows:

“A blue car on the road”

The self-attention mechanism pairs up “blue” and “car” so the model generates images with a “blue car” and not a “blue road”. For an in-depth look into this, read the Attention is all you need paper.

(b) Cross Attention (between prompt and image)

The model then uses the information from (a) to guide the reverse diffusion process to generate the images containing blue cards.

This is a very important part of the conditioning pipeline, so much so that modifying its functionality can change the style of the generated images. Modifying these to fine-tune model outputs is known as Hypernetworks, you can read about them here.

End-to-End Pipeline Overview

Based on everything we have discussed above, here is what the finished pipeline looks like.

and this is a visualization of how the noise is converted into an image.

If you made it till here, 👏. Thank you for reading the article. I hope you found the information useful. Please post any feedback you have in the comments below. Peace ✌️.

Step 1: Create OpenAI API Key

⚠️ If you don’t have an OpenAI account, create one and then follow the steps below.

1) Open Dashboard

Dashboard Link → https://platform.openai.com/account/api-keys

You should see this:

2) Create a new secret key

Enter OpenDex as the key name and create it

3) Save the key

Copy the key it gives you and save it on notepad for now, you will need it later!

Click on Done and you can close the tab

✅ You now have your API Key

Step 2: Find a database of all the pokemon and their attributes

I found a pokemon dataset on kaggle which had pokemons up to the 8th Generation.

https://www.kaggle.com/datasets/mariotormo/complete-pokemon-dataset-updated-090420?select=pokedex_(Update_04.21).csv

You can use other pokemon databases/datasets if you wish to do so.

⚠️ Download this dataset into a folder called datasets in your working directory so your folder looks like this.

Here dataset_file_name.csv is a placeholder name

.
└── datasets/
    └── dataset_file_name.csv

Step 3: Setup the backend

Step 1: Define the environment variables

Create a file .env in your working directory so the file tree looks like this

.
├── datasets/
│   └── dataset_file_name.csv
└── .env

and paste the following into the file

OPENAI_APK_KEY="<key>"

Replace <key> with your secret key you noted down earlier and save the file

Step 2: Install python dependencies

Create a file requirements.txt in your working directory so you file tree looks like this

.
├── datasets/
│   └── dataset_file_name.csv
├── .env
└── requirements.txt

Add the dependencies to the file

flask
langchain
openai
python-dotenv
flask-cors
flask-restful
pandas

Save the file.

Install the dependencies using pip using: pip install -r requirements.txt

Step 3: Building the flask server

Create a new file app.py in the working directory so you file tree looks like this:

.
├── datasets/
│   └── dataset_file_name.csv
├── .env
├── requirements.txt
└── app.py

We’ll start by importing all the modules we will be using

# Flask API
from flask import Flask, render_template, request
from flask_restful import Api, Resource
from flask_cors import CORS

# Langchain SDK
from langchain.llms import OpenAI
from langchain.agents import create_csv_agent

# Environment Variables
from dotenv import load_dotenv

Now we’ll load in the environment variables

load_dotenv()

To work with our dataset we’ll be using the CSV Agent included with the langchain SDK

agent = create_csv_agent(OpenAI(temperature=0.1),
                         'datasets/dataset_file_name.csv', verbose=True)

make sure you replace dataset_file_name.csv with the filename for your dataset

Now, create the flask and add CORS middleware

# Create Flask A
app = Flask(__name__)
# Add CORS middleware
CORS(app)
api = Api(app)

We’ll return our html document at the / path

# return index.html at / path
@app.route('/')
def index():
        return render_template('index.html')

Now the core section which is the api endpoint which takes a prompt as input and returns the response from the LLM

class OpenDexApi(Resource):
    def post(self):
                # data from the frontend
        jsonData = request.get_json(force=True)
                # user's prompt
        prompt = jsonData['prompt']

                # return an error if the prompt is empty
                if (prompt == None):
            return {'error': 'prompt is required'}, 400
        else:
            try:
                # execute the prompt using langchain csv agent
                                op = agent.run(prompt)
                                # respond with the response from the agent
                return {'prompt': prompt, 'output': op}, 200
            except Exception as e:
                # return error data if there is an issue in execution
                return {'error': str(e)}, 500

Map the resource we created to the /api path

api.add_resource(OpenDexApi, '/api')

Run the server

if __name__ == '__main__':
    app.run(debug=True)

Here is all the code in one code block

from flask import Flask, render_template, request
from flask_restful import Api, Resource
from flask_cors import CORS
from langchain.llms import OpenAI
from langchain.agents import create_csv_agent
from dotenv import load_dotenv
load_dotenv()
agent = create_csv_agent(OpenAI(temperature=0.1),
                         'datasets/dataset_file_name.csv', verbose=True)
app = Flask(__name__)
CORS(app)
api = Api(app)

@app.route('/')
def index():
    return render_template('index.html')

class OpenDexApi(Resource):
    def post(self):
        jsonData = request.get_json(force=True)
        prompt = jsonData['prompt']
        if (prompt == None):
            return {'error': 'prompt is required'}, 400
        else:
            try:
                op = agent.run(prompt)
                return {'prompt': prompt, 'output': op}, 200
            except Exception as e:
                return {'error': str(e)}, 500

api.add_resource(OpenDexApi, '/api')

if __name__ == '__main__':
    app.run(debug=True)

Step 4: Add the files for the frontend

1) HTML Template

Create a folder templates and create a file index.html inside it

<!DOCTYPE html>
<html>

<head>
    <title>OpenDex</title>
    <link rel="stylesheet" href="/static/style.css">
</head>

<body>
    <div class="logo">

        <span>OpenDex</span>
    </div>
    <div class="form-container">
        <form>
            <label for="prompt">Question</label>
            <input type="text" id="prompt" name="prompt">

            <button id="submit" type="submit">Ask OpenDex</button>
            <div id="loader" class="loader"></div>
        </form>
        <!-- add a section to display the response -->
        <div id="response-div">
            <p id="response"></p>

        </div>

    </div>
    <script src="/static/index.js"></script>
</body>

</html>

2) Stylesheet and Javascript

Create a folder static and create a file index.js

// references to html elements
const submitButton = document.getElementById('submit');
const loader = document.getElementById('loader');
const responseDiv = document.getElementById('response');

// add click callback for the submit button
submitButton.addEventListener('click', (event) => {
        // turn on loading animation
    loader.style.display = 'block';
        // hide submit button
    submitButton.style.display = 'none';
    // clear previous references
    responseDiv.innerHTML = '';
    console.log('submit button clicked');
        // prevent default form submission
        event.preventDefault();
    // get user's prompt from the input element
    const prompt = document.getElementById('prompt').value;
    // create request
    const xhr = new XMLHttpRequest();
    xhr.open('POST', '/api');
    xhr.responseType = 'json';
    xhr.setRequestHeader('Content-Type', 'application/json');
        // define callback for the api request
    xhr.onload = () => {
        const response = xhr.response;

        if (xhr.status === 200) { // if request was completed without any issues
            console.log(response);
                        // display the response from the backend to the user
            responseDiv.innerHTML = response.output;
        } else { // if there was an issue while processing the request
            responseDiv.innerHTML = 'Error\n' + response.error;
            console.error(response);
        }
        // stop the loading animation
        loader.style.display = 'none';
        // show the submit button
        submitButton.style.display = 'block';
    };
    // send the api request
    xhr.send(JSON.stringify({ prompt: prompt }));
});

Inside the same static folder create another file style.css

@import url('https://fonts.googleapis.com/css2?family=Lexend+Deca:wght@400;500;700&display=swap');

* {
    font-family: 'Lexend Deca', sans-serif;
}

.logo {
    font-size: 32px;
    font-weight: bold;
    color: #333;
    margin: 20px;
    text-align: center;
}

.logo img {
    max-width: 100%;
    height: auto;
}

.form-container {
    max-width: 500px;
    margin: 0 auto;
    padding: 20px;
    border: 1px solid #ccc;
    border-radius: 5px;
    box-shadow: 0 0 10px #ccc;
}

.form-container input[type="text"] {
    width: 100%;
    padding: 10px;
    margin-bottom: 10px;
    border-radius: 5px;
    border: 1px solid #ccc;
    font-size: 16px;
    box-sizing: border-box;
}

.form-container button[type="submit"] {
    background-color: #333;
    color: #fff;
    padding: 10px 20px;
    border: none;
    border-radius: 5px;
    font-size: 16px;
    cursor: pointer;
}

.form-container button[type="submit"]:hover {
    background-color: #555;
}

/* add some styling for the dropdown */
.dropdown {
    position: relative;
    display: inline-block;
}

.form-container select {
    width: 100%;
    padding: 10px;
    margin-bottom: 10px;
    border-radius: 5px;
    border: 1px solid #ccc;
    font-size: 16px;
    box-sizing: border-box;
}

.loader {
    border: 5px solid #f3f3f3;
    /* visibility: hidden; */
    display: none;
    /* Light grey */
    border-top: 5px solid #ed7777;
    /* Blue */
    border-radius: 50%;
    width: 30px;
    height: 30px;
    animation: spin 1s linear infinite;
}

@keyframes spin {
    0% {
        transform: rotate(0deg);
    }

    100% {
        transform: rotate(360deg);
    }
}

Now your file tree should end up looking like this

.
├── datasets/
│   └── dataset_file_name.csv
├── templates/
│   └── index.html
├── static/
│   ├── style.css
│   └── index.js
├── .env
├── requirements.txt
└── app.py

Step 5: Run the app

python app.py

Step 6: Open the Website

When you run the server you should see output similar to this

Open the IP address which was logged in your web browser and you should be able to see the OpenDex Page.

Source Code

The source code for this project is available on Github at the link below 👇

https://github.com/kry0sc0pic/OpenDex

Thanks for reading!

If you enjoy content like this, consider following so you don’t miss out.

In today's fast-paced and competitive tech industry, time is of the essence, and every minute counts. This is especially true when it comes to developer onboarding, where the time it takes to get new hires up to speed can have a significant impact on project timelines and deadlines. Team leads are faced with tough choices - make the deadline or spend time on documentation. The choices determine the ability of the team to scale up. Team leads will most often sacrifice documentation to meet project deadlines. This results in higher onboarding times of new developers down the line which delays projects to which the developer may be assigned. Documentation is an investment that pays itself back many times over.

After struggling with a lengthy and frustrating onboarding process that took up to 40 hours across 4 days, I decided to take matters into my own hands and find ways to streamline the process. In this article, I will share the strategies that helped reduce this time to just under 2 hours.

The Problem

For a bit of background, I joined the team as a new developer and decided to take the time to document this process as I went through it. The team was geographically spread across various time zones as well. This amplified the impact of the problem as well.

The existing documentation/wikis:

1. Were not comprehensive

2. Were not up to date

3. Did not serve as a knowledge base

Problem Impact

1. Lower developer productivity

2. Increased development costs

3. Detrimental to team morale and motivation

4. Delayed project and product timelines

Solution Approach

Some of the important considerations while designing the new onboarding process were:

1. The documentation should be experience and skill-agnostic

2. Minimize the need for communication with other developers during the onboarding process

The Solution

1. Having complete, in-depth, and up-to-date documentation that caters to developers with any experience level. The documentation should at a minimum include:

   • Environment setup guide

   • A brief overview of the codebase and repositories

   • Additional information such as commit formats, standards

   • A document owner early on in the process

2. Update documents frequently

   • Review and update just after each release

   • Review and update after migrations or upgrades (a lot of issues tend to pop up post and during these processes so it is essential to do this)

   • Ensure all developers are contributing ideas and information to the document

3. Build and maintain a well-structured knowledge base. This is should ideally include

   • Solutions to frequent issues faced by team members

   • Traps, pitfalls, and quirks of the tools and frameworks being used

   • Any other relevant details and resources

It is also very effective to have a blog-like log section for developers to quickly note down any issues they may encounter or notice as well as solutions to those. These tidbits of information can be collated and refined to further expand the knowledge base.

I would suggest adding walkthrough videos to complement relevant text guides to further improve productivity.

One of the most important things to keep in mind is to keep the documentation up to date with solutions to issues and new documentation as and when needed.

It is a common occurrence where the documentation is left untouched after the initial version or a few updates and eventually becomes outdated.

TLDR;

• Create comprehensive and up-to-date documentation for the onboarding process that includes an environment setup guide, codebase and repository overview, solutions to frequently faced issues, and additional information such as commit formats and standards

• Review and update documentation regularly

• Establish a document owner and ensure all developers contribute to the document.

• Build and maintain a well-structured knowledge base that includes solutions to frequent issues faced by team members, traps, pitfalls, and quirks of the tools and frameworks being used

Bonus → Tools

I used these tools in building the documentation and still use them regularly.

1. Video Recording → Loom

   Loom is a simple and powerful tool to record videos. I used it to record the video guides for the onboarding process. I also regularly use it to demonstrate features and share ideas or concepts.

2. Building Guides → Scribe

   Scribe is a powerful tool that makes building detailed step-by-step guides a walk in the park for anyone. It’s just as simple as hitting record, doing all the steps you would like to detail, and hitting save. That’s it.

3. Wikis/Docs → Notion

   Notion is one of the best and most versatile productivity tools out there and is excellent for building wikis, guides, and managing projects.

I hope these strategies and tools are helpful for you and your team in streamlining your onboarding process. Remember, documentation, and communication are key to ensuring new developers can quickly get up to speed and start contributing to your projects as soon as possible.