# ComfyStream and DepthAnything

**Published by:** [Eric](https://paragraph.com/@ericxtang/)
**Published on:** 2024-11-09
**URL:** https://paragraph.com/@ericxtang/comfystream-and-depthanything

## Content

SummaryThe idea of leveraging AI for rich interactivity in live video is fascinating, and I’ve been itching to build a real time AI video pipeline myself. So this weekend, I got my hands dirty, using a few open source tools to create my first real time AI video pipeline (spoiler alert: it’s really fun and you can do it too!) I used ComfyStream - a tool for running live streams through ComfyUI, created by Yondon Fu. Here is a guide for setting it up on RunPod, based on my experience. This tutorial comes from the perspective of a AI pipeline developer, who uses a remote ComfyUI environment that gives them access to the terminal. The goal of this tutorial is to set up the depth map real time AI pipeline that Yondon shared in this demo video. I’m using SSH Tunneling to simplify the networking setup. Here is a high level technical architecture for our setupHigh Level Architecture Diagram for This TutorialStep 1: Set up ComfyUI (on Runpod)Runpod is an easy service to get started. I used:RunPod Stable Diffusion template4090 as my GPUNote - I had issues using community cloud on Runpod due to UDP port issues, so I would recommend sticking with the server cloud. If you already have ComfyUI set up, you can skip this step. You can monitor the container logs through Runpod, which is very helpful for debugging issues as you install nodes. In my experience, hacking on ComfyUI workflows require a decent amount of debugging around installing nodes. I also recommend setting up SSH public keys, so you can directly ssh into the GPU server. This will help us with installing custom nodes, and setting up ssh tunnels. Your /workspace will be the persistent disk drive. Make sure to install everything in there, so the data will persist between reboots. After bringing up the pod, it usually takes a little while for environment to completely sync. During this period, you may not be able to ssh into the server. But I found you can use web terminal to log in and start setting up the machine. After launching web terminal, install ComfyUI and node manager by:cd /workspacegit clone https://github.com/comfyanonymous/ComfyUI.gitcd ComfyUIpip install -r requirements.txt 1.cd ComfyUI/custom_nodesgit clone https://github.com/ltdrdata/ComfyUI-Manager.gitEnjoy generating your first purple bottle image.Step 2: Install ComfyStreamComfyStream is a companion package to ComfyUI. It will by default talk to localhost and leverage ComfyUI’s inference engine. First, we install Conda.cd /workspacewget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.shchmod +x Miniconda3-latest-Linux-x86_64.sh./Miniconda3-latest-Linux-x86_64.shI like to change the installation path to /workspace/miniconda3, so everything is persisted over time.eval "$(/workspace/miniconda3/bin/conda shell.bash hook)" - if you start a new shell session, make sure to run this command so you can use condaVerify Conda has been installed using which conda - you should see /workspace/miniconda3/bin/conda Next, we create and activate a Conda environmentconda create -n comfystream python=3.11conda activate comfystream - This should activate your comfystream environment in conda. If you start a new shell session, make sure to run this command again.Next, we install Pytorchconda install pytorch torchvision -c pytorchNext, we install ComfyStream. There is currently a dependency issue, and there is a branch with a workaround.cd /workspacegit clone https://github.com/yondonfu/comfystream.gitcd comfystreampip install .After that, we copy the nodes/tensor_utils nodes into the custom_nodes folder in the ComfyUI workspace (at /workspace/ComfyUI/custom_nodes). This gives you the LoadTensor and SaveTensor nodes, accelerating your live streaming workflow.cp -r nodes/tensor_utils /workspace/ComfyUI/custom_nodes/Step 3: Install DepthAnything Tensorrt NodeThe https://github.com/yuvraj108c/ComfyUI-Depth-Anything-Tensorrt custom node generates depth maps from images. Installing the node is pretty manual. Install the Node You can use the Comfy manager to install the node. Download and Build Tensorrt Enginecd /workspace/ComfyUI/custom_nodes/ComfyUI-Depth-Anything-Tensorrtwget https://huggingface.co/yuvraj108c/Depth-Anything-2-Onnx/resolve/main/depth_anything_v2_vitb.onnx?download=truemv depth_anything_v2_vitb.onnx?download=true depth_anything_v2_vitb.onnxOpen up export_trt.py, update 2 variables:trt_path to ./depth_anything_v2_vitb-fp16.engineonnx_path to "./depth_anything_v2_vitb.onnx"python export_trt.py - to export the onnx engine filemkdir -p /workspace/ComfyUI/models/tensorrt/depth-anything/mv depth_anything_v2_vitb-fp16.engine /workspace/ComfyUI/models/tensorrt/depth-anything/Test Out in Depth Map Generation in ComfyUIOpen up the custom node manager, and you should see ComfyUI Depth Anything TensorRT (search for “tensorrt”) as a custom node that’s installed. Sometimes it says the node is not installed correctly, you can click in “fix it”.You have to click on “Restart”, and reload the browser.If you double-click on an empty part of the canvas, you should be able to search and find Depth Anything Tensorrt. Add it to the canvas.If this doesn’t show up, try and see if it’s because the custom node is installed correctly. If not, you can try to “fix it”. The fixing process will take a while, since it’ll be installing some packages.Add a ”Load Image” node as the input, and the “Preview Image” node as the output. You should be able to pick an image and see the depth map.Running DepthAnything in ComfyUIStep 4: Bring Up ComfyStream ServerTunnel to open up UDP connectionOn the remote server (remote_host), run:apt-get install socatsocat TCP4-LISTEN:4321,fork UDP4:127.0.0.1:5678On the local machine: Open an SSH tunnel on port 4321, run ssh -L 4321:localhost:4321 {remote_user@remote_host and other params from Runpod ssh cmd over exposed TCP port} - make sure this SSH tunnel stays openOn the local machine, run socat UDP4-LISTEN:1234,fork TCP4:127.0.0.1:4321Now, traffic sent to localhost:1234 on the local machine will be forwarded over the SSH tunnel and will reach 127.0.0.1:5678 on the remote server. Bring up ComfyStream Server on the remote server, so it can interact with the Comfy inference enginecd /workspace/comfystreampip install -r requirements.txtpython install.py --workspace /workspace/ComfyUIOpen another terminal over port 8889:Copy that command, and add -L 8889:localhost:8889For example: ssh -L 8889:localhost:8889 {remote_user@remote_host and other params from Runpod ssh cmd over exposed TCP port}python server/app.py --workspace /workspace/ComfyUI --port=8889 --media-ports=5678We should now have the ComfyStream server running on port 8889, with media port 5678 listening on UDP. It’s also using the same workspace and sharing the available nodes / models as the ComfyUI engine you were using. Traffic sent to localhost:1234 on the local machine will be forwarded over the SSH tunnel and will reach 127.0.0.1:5678 on the remote server. If you are having issues with traffic tunneling, try following the TURN server instructions in the ComfyStream repo.Step 5: Run the Front EndYou can run the frontend on your local machine. Just git clone the same repo, and run the following:cd uinpm install --legacy-peer-depsnpm run devNow you should be able to visit localhost:3000 in your browser. For stream URL, use http://127.0.0.1:8888 , pick the “depth anything” workflow in the workflows directory from comfystream. After that, you should see the depth map live stream based on you webcam. SummaryThere are a few other projects in the Comfy ecosystem that support live video workflows. What makes ComfyStream stand out is the usage of WebRTC, which makes it possible to stream video across the internet with low latency. I’m excited to continue playing with this tool and build upon the DepthAnything workflow. This is my first public experiment in building a real time AI video pipeline. If you are interested in this technology, join the waitlist for the Livepeer Real-time AI Video Showcase.EpilogueI noticed that sometimes restarting a Runpod instance is a pain because you need to wait for a GPU to become available on your machine. They mentioned using network mounted drives can solve this issue, but I haven’t gotten around to it. Also, if you are restarting the instance, here is a number of steps to re-establish tunneling and restart the server:SSH into your instance: ssh -L 4321:localhost:4321 -L 8889:localhost:8889 {remote_user@remote_host and other params from Runpod ssh cmd over exposed TCP port}Run socat on remote server: socat TCP4-LISTEN:4321,fork UDP4:127.0.0.1:5678Run socat on local machine: socat UDP4-LISTEN:1234,fork TCP4:127.0.0.1:4321Load up Conda on remote server: eval "$(/workspace/miniconda3/bin/conda shell.bash hook)"Activate comfystream environment: conda activate comfystreamcd /workspace/comfystreamStart comfystream server: python server/app.py --workspace /workspace/ComfyUI --port=8889 --media-ports=5678

## Publication Information

- [Eric](https://paragraph.com/@ericxtang/): Publication homepage
- [All Posts](https://paragraph.com/@ericxtang/): More posts from this publication
- [RSS Feed](https://api.paragraph.com/blogs/rss/@ericxtang): Subscribe to updates
- [Twitter](https://twitter.com/ericxtang): Follow on Twitter