Is there a way to run with 12g VRAM? Is there a way to optimize it

[libai-lab]

Is there a way to run with 12g VRAM? Is there a way to optimize it

[tsjslgy]

Same

[erosDiffusion]

10 would be great ...

actually I have a virtualized docker ubuntu image of this and it worked on my 10gb card (sharing roughly 10gb additional ram memory)
the quality was great on the first run.

[0lento]

I made a fork that runs on my 8GB GPU:
https://github.com/0lento/TRELLIS/

It loads and unloads different stage models on demand so it doesn't require 16GB always. It can occasionally go OOM but it's pretty random. On windows you can enable Sysmem Fallback from Nvidia drivers and it'll not crash on those cases. This may even work on 6GB GPU's with these changes if you have that fallback enabled, but I can't test this myself.

Note that most of that model management code is written by AI, so there can still be issues with it but it does work for me and few others.

[tsjslgy]

I made a fork that runs on my 8GB GPU: https://github.com/0lento/TRELLIS/

It loads and unloads different stage models on demand so it doesn't require 16GB always. It can occasionally go OOM but it's pretty random. On windows you can enable Sysmem Fallback from Nvidia drivers and it'll not crash on those cases. This may even work on 6GB GPU's with these changes if you have that fallback enabled, but I can't test this myself.

Note that most of that model management code is written by AI, so there can still be issues with it but it does work for me and few others.

I pulled your fork and tried to run it, but it still resulted in an OOM error. I started investigating the issue. Since I was using xformers, I switched to flash-attn to see if it would help, but the OOM error persisted. At that time, I hadn't set NVIDIA to system fallback, so I made that adjustment, but it still didn't resolve the issue.

For reference, my system is Windows 10 (22H2), with a 3060 (12GB VRAM) and 64GB of RAM. Perhaps I made some mistakes in my setup, and I’m continuing to troubleshoot. However, so far, it hasn't worked for me.

Thank you for sharing your fork; I really appreciate it!

[0lento]

After reviewing the code, it's pretty far from ideal and has bunch of logic issues, but it let me run this at reasonable speed. I've implemented more structured way to keeping models on CPU and only loading to GPU since but it doesn't actually make any impact since this initial quick hack generation yields similar gains, in fact I think a lot simpler model flush just before the initial mesh generation would have the same impact here. But I won't be pushing new changes to this until I find something more sophisticated that actually makes a difference.

The issue seems to be that the first OOM happens on initial mesh generation and next when you actually simplify and bake the texture and they uses most GPU resources at once, those smaller models used on earlier stages aren't that big so this naive model unloading scheme isn't going to fix everything.

I've found out that details on your generation image seem to matter a lot on VRAM usage, my previous test image with car almost never went OOM but my new test image with cartoon character eating spaghetti does it always.

At that time, I hadn't set NVIDIA to system fallback, so I made that adjustment, but it still didn't resolve the issue.

Make sure the setting got applied, it really shouldn't OOM in normal use after you have it enabled. It should just make everything slower when you go over the limit. Fortunately TRELLIS isn't going over the VRAM amount by huge margin on initial mesh generation so this doesn't make it that much slower.

For reference, my system is Windows 10 (22H2), with a 3060 (12GB VRAM) and 64GB of RAM.

I'm on Win 11 Pro (24H2) with 2070 Super (8GB) and 64GB RAM. I'm using xformers.

There are a ton of desktop apps that just take their own small slice from the GPU VRAM when you have GPU acceleration enabled on them all. I've used Process Explorer for MS to identify these (on my system even mouse software has some hidden AI prompt system running on the background). After cleaning most of the unneeded usage on desktop, I'm at 0.8 GB while writing this. It used to be close to 2GB on desktop when I first started looking into this, every small change counts when you are this close to limit.

Besides the VRAM issue, you can also make the generation a lot faster by omitting extra video gens (if you use the demo gui, just keep the gaussian video if you want to preview results) and by using mesh baking set to fast option and by using 256 texture if you just want something to review quickly. These changes need only minor edits to app.py and postprocessing_utils.py.

[erosDiffusion]

@0lento

Besides the VRAM issue, you can also make the generation a lot faster by omitting extra video gens (if you use the demo gui, just keep the gaussian video if you want to preview results) and by using mesh baking set to fast option and by using 256 texture if you just want something to review quickly. These changes need only minor edits to app.py and postprocessing_utils.py.

Could you show how to disable the video generation ? I tried but failed.

[0lento]

Could you show how to disable the video generation ? I tried but failed.

I actually have version of that webui here that does directly give the mesh output with some extras parameters to tweak, but I've been stuck on trying to optimize the gen itself so haven't had time to clean it up.

If you just want to reduce video gens to gaussian alone, that's easy since you just omit these lines:

TRELLIS/app.py

Lines 115 to 116 in ab1b84a

	video_geo = render_utils.render_video(outputs['mesh'][0], num_frames=120)['normal']
	video = [np.concatenate([video[i], video_geo[i]], axis=1) for i in range(len(video))]

this alone makes it a lot faster step. To fully bypass the video you need some further gradio edits.

[erosDiffusion]

@0lento
the video itself does not take too long (roughly one minute on 3080 10gb, but it seems to depend on input size/order of the run)
but disabling the steps as you suggested made it overall a bit faster (thanks!)

I have run the example.py and noticed for me the most time is spent in texturing (not actual mesh generation which is reasonably fast).

it would be interesting to disable that. do you know if it's possible ?

in general there is performance degradation after the first run (on the main gradio app)
first run is quite fast. subsequent runs gets slower and slower. or maybe it's the model complexity. not sure.

edit:
i've modified this file to skip texture baking. it makes it much much faster and i don't need it for my goals) downloading the glb works but the preview is a bit washed out (whiteish to the point you can't see the model well)
postprocessing_utils.py.txt

[0lento]

edit: i've modified this file to skip texture baking. it makes it much much faster and i don't need it for my goals) downloading the glb works but the preview is a bit washed out (whiteish to the point you can't see the model well) postprocessing_utils.py.txt

Note that if you skip the videos and texturing, you also don't need to create gaussian, only mesh. this doesn't have huge impact on speed. this is on my list to implement with the ui overhaul (I already have texture disable there but it still generates gaussian) but I'm currently looking into making things faster first.

[psychobee]

edit: i've modified this file to skip texture baking. it makes it much much faster and i don't need it for my goals) downloading the glb works but the preview is a bit washed out (whiteish to the point you can't see the model well) postprocessing_utils.py.txt

Note that if you skip the videos and texturing, you also don't need to create gaussian, only mesh. this doesn't have huge impact on speed. this is on my list to implement with the ui overhaul (I already have texture disable there but it still generates gaussian) but I'm currently looking into making things faster first.
would disabling this work? video = render_utils.render_video(outputs['gaussian'][0], num_frames=120)['color'] which is above these video_geo = render_utils.render_video(outputs['mesh'][0], num_frames=120)['normal']
video = [np.concatenate([video[i], video_geo[i]], axis=1) for i in range(len(video))]

[iliagrigorevdev]

I made some small changes to make it work with 12gb vram:
iliagrigorevdev@f519063

[psychobee]

edit: i've modified this file to skip texture baking. it makes it much much faster and i don't need it for my goals) downloading the glb works but the preview is a bit washed out (whiteish to the point you can't see the model well) postprocessing_utils.py.txt

Note that if you skip the videos and texturing, you also don't need to create gaussian, only mesh. this doesn't have huge impact on speed. this is on my list to implement with the ui overhaul (I already have texture disable there but it still generates gaussian) but I'm currently looking into making things faster first.

app_with_noglb.txt
this works i think

[0lento]

I force-pushed cleaned up version on my repo on this branch now: https://github.com/0lento/TRELLIS/tree/low-vram
I've measured it taking 6-8GB cuda memory on use but it depends a lot on image and generated latent structure complexity. For example too low sample step amounts can consume way more memory (talking about something like 2-3 steps).

[psychobee]

I force-pushed cleaned up version on my repo on this branch now: https://github.com/0lento/TRELLIS/tree/low-vram I've measured it taking 6-8GB cuda memory on use but it depends a lot on image and generated latent structure complexity. For example too low sample step amounts can consume way more memory (talking about something like 2-3 steps).

it worked on my 4 gb but took 23 minutes . but i included @erosDiffusion not baking textures and not rendering gaussian and normal video ,like in text i uploaded 12 steps of bothe

both worked and were detailed

[0lento]

it worked on my 4 gb but took 23 minutes

4GB huh? well, you could make it slightly faster on my fork by changing this line:
https://github.com/0lento/TRELLIS/blob/a4cafa627564f770f86a179e3263acc553a243a8/trellis/pipelines/trellis_image_to_3d.py#L304

into:

        self.unload_models(['sparse_structure_flow_model','sparse_structure_decoder', 'slat_flow_model', 'slat_decoder_mesh', 'slat_decoder_gs', 'slat_decoder_rf'])

These additional models were left from initial unload because it's just extra shuffling with GPU's that have 8+ GB (and these latent structure related models don't even take the whole VRAM on 8GB GPU.

This won't make it fast on that VRAM amount but I guess every little can help.