{"id":565,"date":"2023-04-02T07:00:40","date_gmt":"2023-04-02T07:00:40","guid":{"rendered":"https:\/\/gammonrants.org\/?page_id=565"},"modified":"2023-04-02T07:00:41","modified_gmt":"2023-04-02T07:00:41","slug":"upscaling-in-stable-diffusion-compared-in-detail","status":"publish","type":"page","link":"https:\/\/gammonrants.org\/index.php\/upscaling-in-stable-diffusion-compared-in-detail\/","title":{"rendered":"Upscaling in Stable Diffusion compared. In Detail."},"content":{"rendered":"\n

Models are usually trained at a fairly low resolution, something like 512×512. At this resolution they produce the best output. Therefore my SD workflow, probably your’s as well, is to upscale images to a resolution where you can use them as backdrops, posters, calendar images, as all the stuff you want to use photos for. Now SD (I’m using automatic1111) has a huge number of upscaling options, going from blindingly fast to gruesomely slow. Which one to pick? Do the slow ones justify the time investment?<\/p>\n\n\n\n

I mention the time to wait for an upscaler to do its job, so here’s some context: my computer is a laptop using the NVIDIA Geforce 3080TI. Not totally high end anymore but fairly good (it outperforms my Mac’s M1 graphic card 10:1 or so).<\/p>\n\n\n\n

My workflow for nice backdrop images is: render at 640×400, upscale 4x to the native resolution of my mac – 2560×1600. This is what I’m going to do here.<\/p>\n\n\n\n

Extras – Upscaling<\/h2>\n\n\n\n

Automatic1111’s UI offers an “extras” tab that contains an option to upscale any image with a good number of upscalers.<\/p>\n\n\n\n

Let’s have a look what they do.<\/p>\n\n\n\n

Here is a 100×100 part of the original image, scaled with gimp, with a scaler that does absolutely nothing but turn 1 pixel into a little square of four pixels with its color.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Not very beautiful. <\/p>\n\n\n\n

Now, as a starting point, let’s look at GIMPs traditional cubic scaler.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

I must confess I never tried this, so now I’m shocked; the cubic scaler produces a worse image. It doesn’t look as low-res, instead it looks blurry and adds weird artifacts – look closely at the corners of the building.<\/p>\n\n\n\n

Off to the AI upscalers.<\/p>\n\n\n\n

NEAREST<\/h2>\n\n\n\n

Blindingly fast it is, took me no second to render the image. Faster than GIMPs scalers, actually. Here’s its output.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Doesn’t look much better, does it? A bit of a mixture of the two above – certainly less blurry than cubic, but plenty of artifacts and generally a mess you can’t use for e.g. a poster-size photo.<\/p>\n\n\n\n

ESRGAN-4x<\/h2>\n\n\n\n

One of the scalers that are recommended<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

ESRGAN produce a good upscale, you see that instead of a blurry mass you get good structure in the building. It produces quite some artifacts though.<\/p>\n\n\n\n

R-ESRGAN-4x+<\/h2>\n\n\n\n

The scaler with the unpronouncible name is my favorite one that I’ve been using constantly. Supposedly it works similar to ESRGAN but produces a smoother output (with less details).<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

… and it does. Very cool output, in my opinion beating all the others by a high margin. Nearly no artifacts, well scaled structures with detail and little noise. Yes, the structure probably are a bit too clean, but this is great.<\/p>\n\n\n\n

Both ESRGAN scalers run in a few seconds (5?) on my machine, so they can be well used to upscale many images.<\/p>\n\n\n\n

SwinIR 4x<\/h2>\n\n\n\n

I never tried this one. Not recommended by “stable diffusion art”, at the end of the dropdown as well \ud83d\ude42<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

And undeservedly so. Produces a fairly clean and fairly artifact-free render. I had to look hard for differences to the previous one. In reality it probably doesn’t matter if you use this one or the R-ESRGAN.<\/p>\n\n\n\n

LDSR<\/h2>\n\n\n\n

This upscaler is SLOW. Supposedly (stable diffusion art – a great site – explains this) it came as the scaler with SD 1.4, and has its own neural network model (that it downloads – 1 gig or so – the first time you use it). It’s terribly slow, something like 20 times slower than the others.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

And the result is not that much better. It preserves a bit more detail than the two above, at the expense of looking less smooth.<\/p>\n\n\n\n

And finally:<\/p>\n\n\n\n

LANCZOS<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Frankly, I don’t like this one. Compared to the three above, it looks blurry and has more artifacts.<\/p>\n\n\n\n

A first conclusion<\/h2>\n\n\n\n

Judging only from this one crop, the upscalers of choice are R-ESRGAN-4x+ and SwinIR 4+, and maybe LDSR if you have time to waste.<\/p>\n\n\n\n

However, in my experiments I found that overall the LDSR images look better. It seems to recover details in dark places and find a nicer overall tone of the image. Here’s two 640×480 crops from another image:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

R-ESRGAN-4x+ does a fine job here, great tree detail, no artifacts, wonderful. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

But LDSR clearly outperforms it with much more detail fo the person in front and an overall slightly better coloring. This is just perfect.<\/p>\n\n\n\n

So overall, if you’re in a hurry and\/or want to create many upscales, go for R-ESRGAN-4x+. For a picture you want to have perfect, use R-ESRGAN-4x+ and LDSR, and compare the two.<\/p>\n\n\n\n

And what to do with Hires.fix?<\/h1>\n\n\n\n

Apart from this extra upscaler, Automatic1111 also offers “hires.fix” as part of txt2img. I’m struggling a lot with this option, not using it often, because it pretty much kills my workflow: when I’m creating something with stable diffusion, I start with a good prompt, let it create a batch of 6 images (which takes me some 10 secs), and refine until I’m happy with the output. Then I create 15 batches of 6 images and look for the few best images that I then inpaint and then upscale. I’m fairly happy with this workflow.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Hires.fix supposedly adds more detail and is better than a “post mortem upscaler”. However, it is SLOW and a real MEMORY HOG. I can’t upscale to my final 2560×1600 due to out-of-memory situations. So the best I can do is go for a 2x scale with Hires.fix and another 2x scale with an extras scaler afterwards.<\/p>\n\n\n\n

Also, this is terribly slow. If I can generate 90 images in half an hour with my normal workflow (with 6 image batches), it takes me hours this way (with 2 image batches and a SLOW upscaler).<\/p>\n\n\n\n

So let’s try this, with R-ESRGAN-4x+as a Hires.fix to go from 640×400 to 1280×800 and LDSR for the final scaling to 2560×1600.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Now this is a different image. The overall building is way more broken (yes, the prompt included post-apocalyptic, so no surprise there). But other than the “extras” upscalers, the upscaling process of stable diffusion with Hires.fix significantly alters details of the image.<\/p>\n\n\n\n

(and yes, I double checked – I reused prompt and seed, and if I turn off Hires.fix I get exactly the same image as before).<\/p>\n\n\n\n

The quality is excellent, though, no artifacts, no noise, no blurriness.<\/p>\n\n\n\n

Conclusion (again)<\/h2>\n\n\n\n

Hires.fix is NOT an upscaler. It is a way to get a good image at a higher resolution by sacrificing a large amount of time in the workflow. The image is significantly different to the image you would have got without it. I’m not going to use it because I like having 100 images to choose from for my prompt, and the extras upscalers do their job well.<\/p>\n","protected":false},"excerpt":{"rendered":"

Models are usually trained at a fairly low resolution, something like 512×512. At this resolution they produce the best output. Therefore my SD workflow, probably your’s as well, is to upscale images to a resolution where you can use them as backdrops, posters, calendar images, as all the stuff you want to use photos for. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"class_list":["post-565","page","type-page","status-publish","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/pages\/565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/comments?post=565"}],"version-history":[{"count":1,"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/pages\/565\/revisions"}],"predecessor-version":[{"id":578,"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/pages\/565\/revisions\/578"}],"wp:attachment":[{"href":"https:\/\/gammonrants.org\/index.php\/wp-json\/wp\/v2\/media?parent=565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}