Models are usually trained at a fairly low resolution, something like 512×512. At this resolution they produce the best output. Therefore my SD workflow, probably your’s as well, is to upscale images to a resolution where you can use them as backdrops, posters, calendar images, as all the stuff you want to use photos for. Now SD (I’m using automatic1111) has a huge number of upscaling options, going from blindingly fast to gruesomely slow. Which one to pick? Do the slow ones justify the time investment?
I mention the time to wait for an upscaler to do its job, so here’s some context: my computer is a laptop using the NVIDIA Geforce 3080TI. Not totally high end anymore but fairly good (it outperforms my Mac’s M1 graphic card 10:1 or so).
My workflow for nice backdrop images is: render at 640×400, upscale 4x to the native resolution of my mac – 2560×1600. This is what I’m going to do here.
Extras – Upscaling
Automatic1111’s UI offers an “extras” tab that contains an option to upscale any image with a good number of upscalers.
Let’s have a look what they do.
Here is a 100×100 part of the original image, scaled with gimp, with a scaler that does absolutely nothing but turn 1 pixel into a little square of four pixels with its color.
Not very beautiful.
Now, as a starting point, let’s look at GIMPs traditional cubic scaler.
I must confess I never tried this, so now I’m shocked; the cubic scaler produces a worse image. It doesn’t look as low-res, instead it looks blurry and adds weird artifacts – look closely at the corners of the building.
Off to the AI upscalers.
NEAREST
Blindingly fast it is, took me no second to render the image. Faster than GIMPs scalers, actually. Here’s its output.
Doesn’t look much better, does it? A bit of a mixture of the two above – certainly less blurry than cubic, but plenty of artifacts and generally a mess you can’t use for e.g. a poster-size photo.
ESRGAN-4x
One of the scalers that are recommended
ESRGAN produce a good upscale, you see that instead of a blurry mass you get good structure in the building. It produces quite some artifacts though.
R-ESRGAN-4x+
The scaler with the unpronouncible name is my favorite one that I’ve been using constantly. Supposedly it works similar to ESRGAN but produces a smoother output (with less details).
… and it does. Very cool output, in my opinion beating all the others by a high margin. Nearly no artifacts, well scaled structures with detail and little noise. Yes, the structure probably are a bit too clean, but this is great.
Both ESRGAN scalers run in a few seconds (5?) on my machine, so they can be well used to upscale many images.
SwinIR 4x
I never tried this one. Not recommended by “stable diffusion art”, at the end of the dropdown as well 🙂
And undeservedly so. Produces a fairly clean and fairly artifact-free render. I had to look hard for differences to the previous one. In reality it probably doesn’t matter if you use this one or the R-ESRGAN.
LDSR
This upscaler is SLOW. Supposedly (stable diffusion art – a great site – explains this) it came as the scaler with SD 1.4, and has its own neural network model (that it downloads – 1 gig or so – the first time you use it). It’s terribly slow, something like 20 times slower than the others.
And the result is not that much better. It preserves a bit more detail than the two above, at the expense of looking less smooth.
And finally:
LANCZOS
Frankly, I don’t like this one. Compared to the three above, it looks blurry and has more artifacts.
A first conclusion
Judging only from this one crop, the upscalers of choice are R-ESRGAN-4x+ and SwinIR 4+, and maybe LDSR if you have time to waste.
However, in my experiments I found that overall the LDSR images look better. It seems to recover details in dark places and find a nicer overall tone of the image. Here’s two 640×480 crops from another image:
R-ESRGAN-4x+ does a fine job here, great tree detail, no artifacts, wonderful.
But LDSR clearly outperforms it with much more detail fo the person in front and an overall slightly better coloring. This is just perfect.
So overall, if you’re in a hurry and/or want to create many upscales, go for R-ESRGAN-4x+. For a picture you want to have perfect, use R-ESRGAN-4x+ and LDSR, and compare the two.
And what to do with Hires.fix?
Apart from this extra upscaler, Automatic1111 also offers “hires.fix” as part of txt2img. I’m struggling a lot with this option, not using it often, because it pretty much kills my workflow: when I’m creating something with stable diffusion, I start with a good prompt, let it create a batch of 6 images (which takes me some 10 secs), and refine until I’m happy with the output. Then I create 15 batches of 6 images and look for the few best images that I then inpaint and then upscale. I’m fairly happy with this workflow.
Hires.fix supposedly adds more detail and is better than a “post mortem upscaler”. However, it is SLOW and a real MEMORY HOG. I can’t upscale to my final 2560×1600 due to out-of-memory situations. So the best I can do is go for a 2x scale with Hires.fix and another 2x scale with an extras scaler afterwards.
Also, this is terribly slow. If I can generate 90 images in half an hour with my normal workflow (with 6 image batches), it takes me hours this way (with 2 image batches and a SLOW upscaler).
So let’s try this, with R-ESRGAN-4x+as a Hires.fix to go from 640×400 to 1280×800 and LDSR for the final scaling to 2560×1600.
Now this is a different image. The overall building is way more broken (yes, the prompt included post-apocalyptic, so no surprise there). But other than the “extras” upscalers, the upscaling process of stable diffusion with Hires.fix significantly alters details of the image.
(and yes, I double checked – I reused prompt and seed, and if I turn off Hires.fix I get exactly the same image as before).
The quality is excellent, though, no artifacts, no noise, no blurriness.
Conclusion (again)
Hires.fix is NOT an upscaler. It is a way to get a good image at a higher resolution by sacrificing a large amount of time in the workflow. The image is significantly different to the image you would have got without it. I’m not going to use it because I like having 100 images to choose from for my prompt, and the extras upscalers do their job well.