Hi everyone, I’m looking for open-source (free license) models to remove objects and empty rooms from indoor photos using masks. SDXL inpainting generates irrelevant objects instead of filling in background textures, while Qwen Image-Edit works great but is too VRAM-heavy for my setup. Are there lightweight models (like LaMa or optimized ControlNet inpainting setups) or lower-VRAM alternatives that can seamlessly erase objects and complete wall/floor patterns?
There are lighter alternatives to Qwen-Image-Edit, but after trying them, it looks like each has its own quirks:
Yes — for this particular job I would probably not start by trying to squeeze Qwen-Image-Edit into less VRAM.
For masked object removal / room emptying, I would try the following order instead:
- LaMa as the cheap/default eraser.
- RORem when LaMa is too conservative or leaves an implausible hole.
- OSOR when the object has shadows/reflections or the supplied mask does not cover all of the object’s visible effects.
- If you have depth / normals / a mesh / multiple views, treat that as a separate structural-reconstruction problem and use those signals to guide inpainting rather than asking an RGB-only model to guess everything.
The important distinction is that removing an object and emptying a furnished room gradually become two different problems. For a chair in front of a wall, an inpainting model can often infer a plausible continuation of the wall. For a large cabinet hiding the wall/floor boundary, there may simply be no RGB evidence for the true geometry behind it. At that point the model is generating a plausible completion, not recovering hidden ground truth.
So my default practical pipeline would be:
mask
↓
exact mask + slightly expanded-mask variant
↓
LaMa
↓ if too conservative / poor semantic completion
RORem
↓ if shadow/reflection/residual footprint is the problem
OSOR
↓
keep the least-invasive successful result
I would test the mask before spending much time changing models. A very cheap A/B test is:
A: exact object mask
B: object mask + small dilation
Furniture often leaves contact shadows, anti-aliased edges, reflected color, etc. outside the segmentation mask. On the other hand, making the mask too large gives a generative model more freedom to redesign the room, so I would not blindly use one large dilation value for every image.
Why these three would be my main candidates
1. LaMa: first thing I would try
LaMa is still a very useful baseline for this kind of task.
Its original goal is resolution-robust large-mask inpainting, and the authors specifically report good generalization to resolutions much higher than its training resolution and to structures such as repeating patterns.
That maps surprisingly well to many indoor-erasing cases:
- walls,
- floors,
- tiles,
- relatively regular texture,
- furniture-sized holes,
- situations where you want the untouched area to remain untouched.
It is also conceptually much simpler than a large text-conditioned image-editing model. You give it an image and a binary mask; it fills the hole.
That simplicity is also one of its limitations: it is not really an instruction-following “understand this furniture and remove all consequences of it” system. If the mask misses a shadow, reflection, or contact region, LaMa has no special reason to decide that those pixels belong to the removed object.
Still, for a low-VRAM pipeline I would absolutely keep it as the first-pass model.
The original repository is Apache-2.0 licensed.
If you mainly want an easy local UI/API rather than the research repository itself, IOPaint is also worth knowing about; it exposes LaMa and several other inpainting models through a self-hosted workflow.
2. RORem: probably the most interesting middle ground here
RORem is particularly relevant because its problem statement is very close to the failure mode you described.
A generic diffusion inpainting model has an ambiguity:
Should I reconstruct the masked object, replace it with another plausible object, or reconstruct the background that should exist after removal?
RORem is specifically trained as an object remover, rather than merely as a generic inpainting model.
The authors built more than 200K removal training triplets through a human-in-the-loop data-generation process, and the released model is based on SDXL-Inpainting. The repository includes:
- the regular RORem checkpoint,
- a mixed-resolution variant,
- an LCM / accelerated variant,
- Diffusers inference code,
- training code and dataset links.
The official Diffusers example is straightforward:
import torch
from diffusers import AutoPipelineForInpainting
pipe = AutoPipelineForInpainting.from_pretrained(
"LetsThink/RORem",
torch_dtype=torch.float16,
variant="fp16",
).to("cuda")
The authors’ example uses 512×512 inference, exposes mask dilation, uses strength=0.99, and notes that roughly 15–30 denoising steps can work well:
The Hugging Face checkpoint is published in Diffusers format.
Conceptually I would put RORem between LaMa and a general image-editing model:
LaMa
more deterministic / conservative background continuation
↓
RORem
generative, but specifically trained to remove objects
↓
general diffusion/image-edit model
much more freedom to reinterpret the masked region
That makes it quite an appealing candidate when ordinary SDXL inpainting keeps putting something else where the furniture used to be.
The RORem repository and HF model card are marked Apache-2.0. Since the model is derived from SDXL-Inpainting, I would still check all applicable base-model terms for your exact redistribution/commercial use rather than treating the repository license as the only license involved.
3. OSOR: interesting when the mask is not the whole problem
OSOR (One-Step Diffusion Inpainting for Effect-Aware Object Removal) is a newer and slightly different direction.
Its target is not only:
remove pixels inside this mask
but more like:
remove the target and the visual effects associated with it
The project explicitly discusses:
- shadows,
- reflections,
- residual traces,
- incomplete / inaccurate masks.
Its Phase-II design predicts an editing region that can extend beyond a conservative object mask. That is useful for something like:
sofa mask
+
shadow extending outside sofa mask
+
color/reflection contamination around sofa
where simply feeding the exact segmentation mask to a conventional inpainter may leave obvious evidence that the sofa used to be there.
OSOR also uses a one-step diffusion formulation, so it is aimed at avoiding the many-denoising-step cost of conventional diffusion removal.
There are both:
- OSOR-SDXL-Inpainting
- OSOR-FLUX-Fill
variants in the released repository/model collection.
For a lower-VRAM/open workflow I would look at the SDXL-Inpainting branch first, rather than assuming the FLUX version is preferable.
There is also an important license distinction here:
| Component | Published license |
|---|---|
| OSOR source code | MIT |
| OSOR-SDXL-Inpainting checkpoints | CreativeML Open RAIL+±M |
| OSOR-FLUX-Fill checkpoints | FLUX.1-dev Non-Commercial License |
| CORNE dataset | Apache-2.0, with upstream terms also relevant |
The repository documents this explicitly in its license section.
So if “free license” means commercial deployment as well as freely downloadable code, I would check the specific checkpoint, not just the GitHub repository license.
Why SDXL inpainting can generate another object instead of an empty background
I would not necessarily interpret your SDXL result as a bug.
The basic Diffusers inpainting contract is that white pixels in the mask are repainted. It is a conditional generation problem, not a classical texture-copy operation.
See the Diffusers inpainting documentation.
In particular, strength controls how far the generation can move from the source image:
- lower strength retains more source-image information,
- higher strength introduces more noise and gives generation more freedom,
- at
strength=1, the masked content is effectively maximally noised.
That creates a basic tension for object removal:
too little generative freedom
→ remnants / poor completion
too much generative freedom
→ new furniture / windows / decorations / geometry
A model trained specifically for object removal can reduce that ambiguity because the training objective says, in effect, “the correct answer is the scene after removal.”
That is one reason I would test RORem before spending a lot of time prompt-engineering generic SDXL inpainting.
Prompting can still help, but prompts such as:
empty room
clean wall
wooden floor
also tell a powerful generative model that it is allowed to synthesize an “empty room”, which is not necessarily the same thing as preserving the exact original room and only reconstructing the occluded background.
Room emptying is eventually a geometry problem, not just an inpainting problem
There is a useful line of work from Matterport that is almost exactly about this problem.
Their CVPR 2024 work, An Empty Room is All We Want: Automatic Defurnishing of Indoor Panoramas, treats defurnishing as a pipeline rather than as a single magic model:
furniture segmentation
→ context preparation
→ specialized inpainting
→ super-resolution / blending
They explicitly fine-tune their inpainting component to tolerate inaccurate masks and remnant shadows and to reduce the tendency of Stable Diffusion to hallucinate objects.
That is a good hint for architecture: masking, removal, and final compositing do not have to be solved by the same model.
Their later work, Defurnishing with X-Ray Vision, goes one step further.
When a 3D mesh is available, they first remove the furniture from the mesh, extend room planes and fill holes, then render depth/normal information from that simplified geometry. Canny edges derived from those signals guide a ControlNet inpainting stage.
That distinction matters a lot.
Suppose a cabinet covers this region:
+---------------- wall ----------------+
| |
| +--------------------+ |
| | | |
| | cabinet | |
| | | |
|-------+--------------------+---------|
| floor |
+--------------------------------------+
From a single RGB image, the model does not actually know exactly where the hidden wall/floor boundary runs behind the cabinet.
It can infer:
probably a straight continuation here
but that is a prior, not recovered evidence.
If you have:
- another camera view,
- depth,
- room-layout estimation,
- normals,
- a mesh / point cloud,
then you can turn that missing information into an explicit conditioning signal.
That is the situation where I would seriously consider ControlNet.
About ControlNet: useful, but I would not treat it as the lightweight replacement by itself
ControlNet is very useful, but I would separate two questions:
- How do I reduce VRAM?
- How do I tell the model what hidden structure should be reconstructed?
ControlNet mainly addresses #2.
The Diffusers ControlNet documentation describes it as an adapter conditioned on additional visual structure such as:
- Canny edges,
- depth,
- pose,
- other structural controls.
There is also a dedicated SDXL ControlNet inpainting pipeline.
But adding ControlNet does not automatically make an SDXL workflow smaller. You still have the base diffusion model plus the additional control network.
So I would use it when you actually possess a useful structural signal.
For example:
single RGB + mask only
→ LaMa / RORem / OSOR first
RGB + reliable depth
→ depth-guided ControlNet becomes interesting
panorama + room layout
→ layout/edge-guided inpainting becomes interesting
multiview / mesh
→ reconstruct geometry first, then use it to control inpainting
Using Canny edges from the original RGB image is also different from using edges describing the desired empty geometry. The original image contains the furniture edges you are trying to remove.
That is why the Matterport “X-Ray Vision” pipeline derives its control signal from a defurnished structural representation rather than simply applying edge detection to the furnished photograph.
A few cheap things I would test before adding another model
A. Exact mask vs small dilation
This is probably the highest-information low-cost test.
Try at least:
mask A = exact segmentation
mask B = slightly expanded segmentation
Expansion can absorb:
- antialiased boundaries,
- contact shadows,
- small reflected regions,
- segmentation under-coverage.
But increasing it indefinitely is not better. A large generative hole gives the model permission to invent more of the room.
RORem’s own example exposes a dilate_size parameter, so mask expansion is explicitly part of its intended workflow.
B. Preserve everything outside the required edit
If your goal is removal rather than general editing, I would make this an explicit contract:
outside edit region → preserve source pixels
inside edit region → use generated completion
boundary → narrow feather/blend
This is useful even when the diffusion pipeline already conditions on the original image.
It prevents tiny color/texture changes across the whole frame from becoming part of the output contract.
The Matterport defurnishing pipeline similarly uses post-processing/blending to retain high-frequency source detail.
C. Give the model enough context, but not the whole world
For a small object in a 4K image, resizing the entire photograph to 512×512 can throw away exactly the wall/floor detail that needs to be continued.
A useful compromise is:
find mask bounding box
→ expand crop substantially for context
→ inpaint crop
→ composite result back
Diffusers even exposes padding_mask_crop for this kind of mask-centered crop in its inpainting API.
For indoor scenes the context margin matters because the model may need to see:
- wall/floor boundaries,
- neighboring floorboards,
- tile repetition,
- baseboards,
- perspective lines.
D. Evaluate “successful removal”, not only image similarity
For object removal, a low pixel error is not necessarily the same as success.
Two particularly useful checks are:
1. Is the original target still recognizable?
2. Did the model invent a new salient object in its place?
Then separately check:
3. Is the wall/floor continuation plausible?
4. Did pixels outside the intended region change?
5. Are shadows/reflections left behind?
That gives you a much more informative comparison than just asking which result “looks best” globally.
VRAM reduction if you still want an SDXL / Diffusers route
If an SDXL-family remover is otherwise giving the best result, Diffusers already has several useful memory controls.
The current Reduce memory usage guide covers the main ones.
The first thing I would try is FP16/BF16 plus model CPU offload:
pipe = AutoPipelineForInpainting.from_pretrained(
model_id,
torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()
enable_model_cpu_offload() moves whole pipeline components between CPU and GPU and usually gives a much better speed/memory compromise than sequential offload.
If VRAM is still the hard limit:
pipe.enable_sequential_cpu_offload()
can go lower, but it can become much slower because submodules are repeatedly transferred.
For larger images:
pipe.vae.enable_tiling()
can reduce VAE peak memory.
One current Diffusers detail worth noting: if you already have PyTorch SDPA or xFormers memory-efficient attention active, the docs recommend not stacking old-style attention slicing on top, because that can make inference much slower rather than helping.
So I would optimize in roughly this order:
FP16/BF16
→ 512px or mask-centered crop
→ model CPU offload if needed
→ VAE tiling for large decode/encode
→ sequential offload only if VRAM is still the blocker
rather than turning on every memory option simultaneously.
My practical decision tree
If I were wiring this into an application, I would probably make the branches explicit rather than searching for one universal model.
START
|
Do you have a mask?
|
yes
|
Is cheap/simple removal enough?
/ \
yes no
| |
LaMa Does generic diffusion
| invent replacement objects?
| / \
acceptable? yes no
/ \ | |
yes no RORem keep current
| |
DONE RORem
|
Are shadow/reflection/
mask-outside remnants
still visible?
/ \
yes no
| |
OSOR-style DONE
effect-aware
removal
|
Is the hidden geometry
itself the main failure?
|
yes
|
Do you have depth / mesh /
normals / multiple views?
/ \
yes no
| |
structural accept that output
guidance is a plausible RGB
+ inpainting reconstruction,
not recovered truth
For a production system this can even be a cascade:
cheap model first
→ confidence / artifact check
→ expensive model only on failures
That should be substantially cheaper than running a large image-edit model for every mask.
License / deployment note
Since you specifically mentioned a free/open-source license, I would check this at the component level.
“Open source model” can hide several different licenses:
repository source code
model checkpoint
base model
dataset
third-party runtime/library
They are not necessarily the same.
For the main candidates above:
- LaMa: repository is Apache-2.0.
- RORem: repository and current HF model card are marked Apache-2.0; it is derived from SDXL-Inpainting, so also inspect the relevant base-model conditions for your intended deployment.
- OSOR: source code is MIT, but the released checkpoints intentionally retain different base-model licenses:
- SDXL version: CreativeML Open RAIL+±M
- FLUX version: FLUX.1-dev Non-Commercial
So if “free” specifically means commercially usable without a non-commercial restriction, I would not treat the FLUX and SDXL OSOR checkpoints as interchangeable.
So, for your exact description — indoor photo + mask + modest VRAM + “erase this, don’t replace it with another random object” — my first three tests would simply be:
1. LaMa
2. RORem
3. OSOR-SDXL only for cases where shadows/reflections/incomplete masks are the hard part
and I would compare the exact mask against one slightly expanded mask before adding more models.
If LaMa works, it is probably the nicest operational answer because it is so lightweight. If it is too conservative, RORem looks like the most directly targeted next step. If even that fails because a large piece of furniture has hidden the actual room geometry, I would stop treating that failure as “find a better eraser” and move the structural information — depth/layout/multiview/mesh if available — into a separate guidance stage.
Hi John,
Thank you for the detailed breakdown and the suggestions of LaMa, RORem, and OSOR. I’ve spent the last few days benchmarking these against my specific use case, and I wanted to share the results, as they highlight a significant gap in performance for high-end virtual staging.
The Current Workflow:
I am using a fine-tuned RF-DETR Seg Large model to generate the initial furniture masks. To ensure we capture shadows and contact edges, I apply a 20px dilation before feeding the mask to the inpainting models.
The Results:
- LaMa: As you noted, it is a great “cheap” eraser, but for “defurnishing” (removing large items like beds/sofas), it fails semantically. It essentially “smudges” the surrounding colors into the hole. It creates a blurry, greyish-red “cloud” where the bed was, rather than reconstructing the hidden floor grain or wall texture. It lacks the “imagination” required for architectural reconstruction.
- RORem: While RORem is built on SDXL and is much faster than Qwen, the results have been disappointing in terms of Texture Integrity and Blending. I am seeing significant “texture melting” (as seen in the attached output). The floor patterns become warped/liquid, and the transition between the original pixels and the AI-generated pixels is very jarring, even with feathered masks.
I used:
pipe = AutoPipelineForInpainting.from_pretrained(
“LetsThink/RORem”,
torch_dtype=torch.float16,
variant=None).to(“cuda”) - The Qwen Comparison: Honestly, Qwen-Image-Edit remains the gold standard in my tests. Even though it is VRAM-heavy and slow, it is the only model that actually “understands” the instruction to reveal the architectural shell. It reconstructs the hardwood floor lines and wall textures with a level of logic that the others haven’t matched.
Visual Evidence (Attached):
- Image 1: Original Photo (Furnished)
- Image 2: My RF-DETR + Dilated Mask (The input provided to models)
- Image 3: RORem Output (Showing the “melting” floor and poor blending)
My Concern:
We are aiming for a commercial-grade product where we need to meet strict metrics (SSIM ≥≥ 0.95 and EdgeMAD ≤≤ 2px). While Qwen hits the visual quality, the latency is the blocker. However, RORem and LaMa seem to fail the visual “realism” test entirely for large-scale removal.
Do you have any suggestions on how to stabilize the geometry in RORem (perhaps via ControlNet-MLSD or Depth)? Or is there a lighter “instruction-based” model that behaves more like Qwen but with the speed of SDXL?
Looking forward to your thoughts!