Image Upscaling Algorithms Explained: ESRGAN, Real-ESRGAN, SwinIR (2027)

If you upscale images in 2027 and want to know why some tools produce sharper results than others, you need to understand the algorithms underneath. This is a readable, no-PhD-needed walkthrough of ESRGAN, Real-ESRGAN, SwinIR, HAT, and the new wave of diffusion-based upscalers (including the engine behind enhance.hjlabs.in). It is written for developers, engineers, and curious power users.

What "super-resolution" means

Super-resolution (SR) is the task of generating a high-resolution (HR) image from a low-resolution (LR) input. Mathematically, it is an ill-posed problem — many possible HR images could downsample to the same LR. The algorithm has to choose a plausible HR among infinitely many candidates.

Pre-AI methods (bicubic, Lanczos) make a single deterministic choice based on neighbour pixels. AI methods learn from millions of LR/HR pairs and choose based on what HR images "tend to look like" — effectively transferring knowledge of texture, faces, edges, and structure from the training set into your image.

Generation 1: SRCNN (2014)

The first deep-learning SR network. Three convolutional layers. Outperformed bicubic. Tiny by modern standards (~57k parameters). Historical interest only — you would not use it today.

Generation 2: SRGAN and ESRGAN (2017-2018)

SRGAN introduced GAN-based training: a generator network produces upscaled images, a discriminator network learns to distinguish "real HR" from "generated HR". The adversarial training pushes the generator to produce perceptually realistic outputs — not just pixel-accurate, but believable.

ESRGAN (Enhanced SRGAN) refined the architecture: deeper residual blocks, removed batch normalisation (which caused artifacts), introduced Residual-in-Residual Dense Blocks (RRDB). For natural images, ESRGAN was the de-facto state-of-the-art for years.

Strengths: fine texture detail, perceptually pleasing edges. Weaknesses: trained on synthetic LR (bicubic-downsampled HR), so it failed on "real-world" LR images that have unknown blur kernels, JPEG compression, sensor noise.

Generation 3: Real-ESRGAN (2021) — the practical workhorse

Real-ESRGAN's key insight: train on complex synthetic degradations instead of clean bicubic downsampling. The training pipeline applies random combinations of:

• Various blur kernels (Gaussian, motion, defocus)
• Random downsampling (bicubic, area, nearest)
• Noise (Gaussian, Poisson, JPEG)
• JPEG compression at random qualities
• Second-order degradation (degrade twice, simulate real-world chains)

The result: a network that handles realistic LR images — phone photos, web screenshots, social-media-compressed images. Real-ESRGAN became the gold standard for general-purpose 4x upscaling and is the engine inside Upscayl, many web demos, and many commercial tools.

Architecture variants:

• realesrgan-x4plus — general photo content
• realesrgan-x4plus-anime — tuned for anime/manga (lines, flat shading)
• realesr-animevideov3 — for animated video frames

Generation 4: Transformer-based (SwinIR, HAT)

SwinIR (Swin Transformer for Image Restoration, 2021) replaced convolutional backbones with the Swin Transformer architecture borrowed from NLP. Self-attention captures long-range dependencies that CNNs struggle with. Result: sharper edges, better fine textures, and better generalisation across image types.

HAT (Hybrid Attention Transformer, 2022) combined channel attention and self-attention with overlapping cross-attention windows. Marginally better than SwinIR on benchmarks; computationally more expensive.

When to use Transformer-based: high-quality content where the marginal gain matters (architectural photography, fashion, art reproduction). When to skip: real-time pipelines where the additional latency hurts.

Generation 5: Diffusion-based (2023-2025)

Stable Diffusion Upscaler, SD-SR, and StableSR apply latent diffusion to SR: the LR image is encoded into latent space, noise is added, and a diffusion model denoises into a HR latent. Strengths:

• Highly realistic textures (best perceptual quality)
• Can be guided by text prompts ("more detailed grass", "sharper eye")
• Handles severely degraded inputs better than GAN-based

Weaknesses:

• Slow (10-30x slower than Real-ESRGAN)
• Stochastic — same input produces different outputs each run
• "Hallucinates" detail that may not be faithful to the original

Generation 6: Multimodal foundation models (2025-2027)

Gemini 3 Pro Image, GPT-5 Image, Claude 4 Image, and similar foundation models treat upscaling as one specific instance of image-to-image generation. They:

• Take an image + optional text prompt
• Generate a higher-resolution, optionally restyled output
• Combine SR with deblurring, colorisation, restoration in one pass

Strengths: best-in-class for "I want a better version of this photo" use cases. Weaknesses: not pure SR — the output may be creatively re-rendered, not pixel-faithful. For archival work, prefer Real-ESRGAN.

This is the architecture behind enhance.hjlabs.in's 5 creativity levels — level 1 is closest to pure SR; levels 4-5 lean into creative reimagining.

Comparison table

Picking the right algorithm for the task

"I want a 4x upscale of a phone photo, free, no install"

Real-ESRGAN via enhance.hjlabs.in level 1, or Real-ESRGAN locally via Upscayl.

"I want a creative restyling, not just upscaling"

Gemini-based (enhance.hjlabs.in levels 2-4) or Stable Diffusion img2img.

"I'm a wedding photographer, batch processing, offline"

Topaz Gigapixel (uses a tuned variant of Real-ESRGAN + face-specific models).

"I'm restoring 100-year-old photos with severe damage"

Stable Diffusion SR or Gemini-based (creativity level 3-4) for the inpainting; Real-ESRGAN for clean-input upscaling.

"I need API access, programmatic, integrated in my workflow"

Replicate / Hugging Face for hosted Real-ESRGAN; Gemini API directly for Gemini-based; or build on Cloudflare Workers + the Gemini API (this is what powers enhance.hjlabs.in).

"I'm doing scientific imaging or forensics"

None. AI upscaling is not appropriate for evidentiary or scientific work because the algorithms hallucinate detail.

Common metrics: PSNR, SSIM, LPIPS

Algorithm comparisons cite three numbers:

• PSNR — Peak Signal-to-Noise Ratio. Higher is more pixel-accurate. Bicubic often wins PSNR while looking worse to humans.
• SSIM — Structural Similarity Index. Closer to human perception of similarity.
• LPIPS — Learned Perceptual Image Patch Similarity. Lowest is most perceptually similar to ground truth. Best of the three for "looks right to humans".

In 2027, LPIPS is the metric to watch — it correlates best with human rankings.

Open-source projects worth knowing

• Real-ESRGAN — the canonical implementation
• Upscayl — cross-platform GUI bundling Real-ESRGAN
• SwinIR — SwinIR reference implementation
• StableSR — diffusion-based SR

Building your own

If you want to ship an SR product on Cloudflare Workers (free tier), the practical 2027 path:

1. Use Gemini 3 Pro Image API for the heavy lifting (no GPU on Workers)
2. Cloudflare Worker proxies user input to Gemini, returns the result
3. Add rate limiting via KV (we use this on enhance.hjlabs.in — 3/day free, unlimited with user's own Gemini key)
4. Add R2 for caching common outputs
5. Total cost: ₹0 for under ~10k requests/month

For dev tools to support this kind of stack, see fmt.hjlabs.in (JSON formatters and validators), or the OG image API at og.hjlabs.in.

Bottom line

In 2027, image upscaling is not one algorithm — it is a family of tools each suited to a different problem. For the 90% case (phone photos, social media, casual restoration), Real-ESRGAN via Upscayl or a Gemini-based hosted tool like enhance.hjlabs.in is correct. For the 10% specialist cases, pick from the table above. Either way, the era of "you need a PhD and a GPU cluster to upscale images" is over.