Repository navigation
Not getting the desired result as Stable Diffusion Web UI #11585
Description
Describe the bug
While using the Stable Diffusion Web UI and Hugging face Diffuser's Stable Diffusion for inpainting task with ControlNet, I am not getting the desired result in HF diffusers. But I want to get the same result as Stable Diffusion Web UI.
Reproduction
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/control_v11p_sd15_inpaint", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet = controlnet, torch_dtype = torch.float16)
pipe.to("cuda")
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
generate image
output = pipe(prompt = pos_prompt, negative_prompt = neg_prompt, num_inference_steps = 45,
generator = generator, image = init_image, mask_image = mask_image,
control_image = ref_image, strength = 0.5, guidance_scale = 8.0,
controlnet_conditioning_scale = 1.0, padding_mask_crop = None)
Logs
The code runs successfully, but the results are not up to the markSystem Info
- 🤗 Diffusers version: 0.33.1
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Running on Google Colab?: Yes
- Python version: 3.11.12
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.6 (gpu)
- Jax version: 0.5.2
- JaxLib version: 0.5.1
- Huggingface_hub version: 0.31.2
- Transformers version: 4.51.3
- Accelerate version: 1.6.0
- PEFT version: 0.15.2
- Bitsandbytes version: not installed
- Safetensors version: 0.5.3
- xFormers version: not installed
- Accelerator: Tesla T4, 15360 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
Who can help?
No response
Activity
asomoza commentedon May 22, 2025
Hi @aniket-professional2025, for me is not clear which result is which and why are you using inpainting with a controlnet that also does inpainting? which parts are you inpainting? are you also doing the same with auto1111, meaning using an inpainting controlnet with another inpainting?
If I try to guess I would say that the big vertical images are done with auto1111 and the square ones with diffusers but if you're comparing them, why not use the same exact image with the same mask on both?
Also are you using a long prompt? because diffusers by default doesn't use long prompts or the weights in the prompt that are commonly used with auto1111, meaning the parts that are like
(word:0.5)aniket-professional2025 commentedon May 22, 2025
asomoza commentedon May 22, 2025
Maybe I expressed myself wrong, so I'll try to be more specific:
I don't know which one (if there is one) that corresponds to automatic1111 and which one to diffusers since you presented 4 images, two are vertical and two are squared, are there ones that are good here? Since you're using SD 1.5 I can't really tell, all of them seem really bad to me, but the last is the worst.
Maybe don't jump to SD 3 from SD 1.5, you can try SDXL.
Even with SD3, SDXL and other models, you can use
enable_model_cpu_offloadwhich will make them run in google colab, you can also try quantization.It's really hard to help you with this if you don't provide the values and images you're using with auto1111.
aniket-professional2025 commentedon May 23, 2025
asomoza commentedon May 23, 2025
We don't see the names in github, using the name of the filenames in github does not work also it makes it harder for us to see, you should just add the description of each one.
which one is the source and which one is the control image?
Why are you using a
guidance_scaleof 9 with diffusers? are you using the same steps?also a
controlnet_conditioning_scaleof 0.05 is really low (almost 0), what are you expecting with this? You're also using it in a really low number of steps and not at the start which is really bad.The same with the strength of the img2img, you're using 0.1 while in auto1111 you're using 0.75, with auto1111 you're make the image really noisy but with diffusers you're practically using the original image.
I told you this before, the reason why this is being that hard to help you is because you're not providing all the information we need, I recommend to update your initial post with all the information and the correct description of the images, so I can see the source images, the input parameters and what you're expecting from diffusers.
github-actions commentedon Jan 9, 2026
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.