Illustrious, a text-to-image design based on Stable Diffusion XL, has become so strong in the AI craft group that Civitai, the largest hub for AI art models, had to create a separate category simply to solve its enormous ecosystem of resources.
And it all happened in three decades. The key behind its accomplishment? A twist on the classics and a transfer.
While older models like SD 3. 5 and Flux rely on extensive organic language descriptions, Onoma AI, the Illustrious creators, used Danbooru tags to aid in the understanding of concepts without resorting to elaborate captioning and modeling.
The woman’s training on Danbooru’s large collection of tagged genre images gives it an advantage in understanding visible concepts.
Each label in the Danbooru method represents certain elements like character features, clothing items, offers, or origins, allowing for accurate control over the generated graphics without wasting precious tokens on long explanations.
These keywords have been around for a while and have evolved into a sort of image classification common among those who enjoy art and movies.
When it comes to understanding the features of a photo, the concept is very accurate and effective.
” It’s like having an designer who understands exactly what you want without having to reveal it in sections”, Vishnu, a Discord part who participates in a site focused on NSFW AI information, told . You simply need to be aware of the appropriate keywords.
Illustrious’s fundamental approach utilizes the good old SDXL infrastructure with a powerful dual-encoder system that combines Picture ViT-L and OpenCLIP ViT-bigG to understand phrases and associate them with their physical equivalent.
The model is capable of processing and generating images at an impressive 1536×1536 resolution, with the capability to stretch up to 2048×2048 and even 3744×3744 without significant quality loss.
For context, the original SDXL handled full HD resolutions ( 1024×1024 ).
Deep dive
The process of making Illustrious was deliberate and methodical. The initial training phase, which produced version 0.1, processed 7.5M images at 1024×1024 resolution with a batch size of 192 images per batch.
The team carefully balanced learning rates, running for 20 epochs ( the process in which AI studies 100 % of its dataset ) to establish a solid foundation. Once the outcomes were satisfactory enough, the team moved on to increase the size and resolution of the dataset for the following iterations.
In the advanced training phase, Illustrious truly began to shine. The dataset was increased to 10 M images in version 1 and the resolution was increased to 1536 1536.
Although they simplified the batch from 128, they also introduced sophisticated tag manipulation techniques and register tokens, fundamental changes that define the model’s exceptional performance.
The final refinement phase of version 2.0 made things a little more complicated. The team incorporated a multi-caption method that significantly improved text-image correspondence by using 20M images at the same high resolution but with a larger batch size of 512.
The result was the best waifu generator known to man, with good finetuning capabilities, prompt adherence, decent aesthetics, and high-quality outputs.
For the more tech-savvy, the Illustrious devs also introduced a lot of interesting techniques like a” No Dropout Token” approach, ensuring that specific tokens would never be excluded during training, the implementation of Quasi-Register Tokens, for the model to be capable of handling unknown or weird concepts, a Cosine Annealing Scheduler, for the learning rate, a Multi-Level Dropout system and Input Perturbation Noise Augmentation, to turn a simple AI model into a powerhouse.
How to use Illustrious
Illustrious doesn’t need any additional steps to run.
The SDXL Model installation procedure is the same as it is for any other model. Depending on the user interface you use, you can download the checkpoint and place it in the appropriate folder.
Windows and Linux
- For ComfyUI, the route is modelscheckpoints.
- For A1111/Forge, the route is /models/Stable-diffusion.
- For Fooocus, the route is also modelscheckpoints.
MacOS
Mac users have similar routes. However, some popular macOS-oriented UIs require additional steps.
- Draw Things users will have to click on” Models”, go to” Customize”, and then click on” Import Model”.
- If they downloaded the model and saved it to their local drives, they can either enter the URL to directly download Illustrious or click” Import Custom Model” to select the file.
- Users of Diffusion Bee must click on the hamburger icon in the top right corner, then click on” Settings”, and then click on” Add new model”, and select their locally downloaded illustrious checkpoint.
Once the model is loaded, there are three things to consider.
- Do not use natural language. For better results, you should rely on Danbooru tags and adhere to the old SDXL prompting format.
- Do not use Pony LoRas. Since the model uses different approaches, it is better to use Illustrious Loras for best results.
- Choose some of the most well-known finetunes over the original Illustrious model. The base model of the original Illustrious model is ideal for fine tunings that concentrate on the desired outcomes. It’s the same as SDXL, Pony or Flux. Finetunes tend to yield better results.
What are the best Illustrious models to select from?
There are many models to choose from, all focusing on different styles, aesthetics, and characteristics.
There are even general models like those from Noob AI, which fine-tuners are using as a base for their models.
However, here are our top pics for different needs. These are great at prompt understanding, output quality, and ease of use. All the samples are from the Civit AI community and are copyright-free.
Best for Versatility: Mistoon_Anime
Link: Mistoon_Anime- v1.0 Illustrious | Illustrious Checkpoint | Civitai
Best for 2.5d: Smooth Mix- Illustrious — Warning! Very NSFW oriented
Link: Smooth Mix- Illustrious | Pony- Illustrious | Illustrious Checkpoint | Civitai
Best for Art and Illustrations: NTR Mix
Link: NTR MIX | illustrious-XL | Noob-XL- XIII | Illustrious Checkpoint | Civitai
Best for Realism: THRILLustrious
Link: THRILLustrious- v5.0 THRILLed | Illustrious Checkpoint | Civitai
edited by Josh Quittner and Sebastian Sinclair
Generally Intelligent Newsletter
A generative AI model’s generative AI model, Gen, tells a weekly AI journey.
