# ComfyStream and DepthAnything By [Eric](https://paragraph.com/@ericxtang) · 2024-11-09 --- Summary ======= The idea of leveraging AI for rich interactivity in live video is fascinating, and I’ve been itching to build a [real time AI video pipeline](https://mirror.xyz/livepeer.eth/bCruUtv0PJWWlFxfbCMATGM_h15hUncwvyFW1A3z7Ag) myself. So this weekend, I got my hands dirty, using a few open source tools to create my first real time AI video pipeline (spoiler alert: it’s really fun and you can do it too!) I used [ComfyStream](https://github.com/yondonfu/comfystream) - a tool for running live streams through ComfyUI, created by [Yondon Fu](https://x.com/YondonFu). Here is a guide for setting it up on RunPod, based on my experience. This tutorial comes from the perspective of a AI pipeline developer, who uses a remote ComfyUI environment that gives them access to the terminal. The goal of this tutorial is to set up the depth map real time AI pipeline that Yondon shared in this [demo video](https://www.reddit.com/r/comfyui/comments/1ggn4oy/comfystream_run_comfy_workflows_on_live_video/). I’m using SSH Tunneling to simplify the networking setup. Here is a high level technical architecture for our setup ![High Level Architecture Diagram for This Tutorial](https://storage.googleapis.com/papyrus_images/137f1a800047fb4fe9fde18cea45d37d42836cff536e138214571dfeec9a2b87.png) High Level Architecture Diagram for This Tutorial Step 1: Set up ComfyUI (on Runpod) ================================== Runpod is an easy service to get started. I used: * RunPod Stable Diffusion template * 4090 as my GPU Note - I had issues using community cloud on Runpod due to UDP port issues, so I would recommend sticking with the server cloud. If you already have ComfyUI set up, you can skip this step. You can monitor the container logs through Runpod, which is very helpful for debugging issues as you install nodes. In my experience, hacking on ComfyUI workflows require a decent amount of debugging around installing nodes. I also recommend setting up SSH public keys, so you can directly ssh into the GPU server. This will help us with installing custom nodes, and setting up ssh tunnels. Your `/workspace` will be the persistent disk drive. Make sure to install everything in there, so the data will persist between reboots. After bringing up the pod, it usually takes a little while for environment to completely sync. During this period, you may not be able to ssh into the server. But I found you can use web terminal to log in and start setting up the machine. After launching web terminal, install ComfyUI and node manager by: * `cd /workspace` * `git clone https://github.com/comfyanonymous/ComfyUI.git` * `cd ComfyUI` * `pip install -r requirements.txt` 1. * `cd ComfyUI/custom_nodes` * `git clone https://github.com/ltdrdata/ComfyUI-Manager.git` Enjoy generating your first purple bottle image. ![](https://storage.googleapis.com/papyrus_images/562f56d64035ed535946eded2e68ea83efd5b01ebe6fc0fb2c7121f251a66668.png) Step 2: Install ComfyStream =========================== ComfyStream is a companion package to ComfyUI. It will by default talk to localhost and leverage ComfyUI’s inference engine. First, we install Conda. * `cd /workspace` * `wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh` * `chmod +x Miniconda3-latest-Linux-x86_64.sh` * `./Miniconda3-latest-Linux-x86_64.sh` * I like to change the installation path to `/workspace/miniconda3`, so everything is persisted over time. * `eval "$(/workspace/miniconda3/bin/conda shell.bash hook)"` - if you start a new shell session, make sure to run this command so you can use `conda` Verify Conda has been installed using `which conda` - you should see `/workspace/miniconda3/bin/conda` Next, we create and activate a Conda environment * `conda create -n comfystream python=3.11` * `conda activate comfystream` - This should activate your comfystream environment in conda. If you start a new shell session, make sure to run this command again. Next, we install Pytorch * `conda install pytorch torchvision -c pytorch` Next, we install ComfyStream. There is currently a dependency issue, and there is a branch with a workaround. * `cd /workspace` * `git clone https://github.com/yondonfu/comfystream.git` * `cd comfystream` * `pip install .` After that, we copy the `nodes/tensor_utils` nodes into the `custom_nodes` folder in the ComfyUI workspace (at `/workspace/ComfyUI/custom_nodes`). This gives you the LoadTensor and SaveTensor nodes, accelerating your live streaming workflow. * `cp -r nodes/tensor_utils /workspace/ComfyUI/custom_nodes/` Step 3: Install DepthAnything Tensorrt Node =========================================== The `https://github.com/yuvraj108c/ComfyUI-Depth-Anything-Tensorrt` custom node generates depth maps from images. Installing the node is pretty manual. **Install the Node** You can use the Comfy manager to install the node. **Download and Build Tensorrt Engine** * `cd /workspace/ComfyUI/custom_nodes/ComfyUI-Depth-Anything-Tensorrt` * `wget https://huggingface.co/yuvraj108c/Depth-Anything-2-Onnx/resolve/main/depth_anything_v2_vitb.onnx?download=true` * `mv depth_anything_v2_vitb.onnx?download=true depth_anything_v2_vitb.onnx` * Open up `export_trt.py`, update 2 variables: * `trt_path` to `./depth_anything_v2_vitb-fp16.engine` * `onnx_path` to `"./depth_anything_v2_vitb.onnx"` * `python export_trt.py` - to export the onnx engine file * `mkdir -p /workspace/ComfyUI/models/tensorrt/depth-anything/` * `mv depth_anything_v2_vitb-fp16.engine /workspace/ComfyUI/models/tensorrt/depth-anything/` **Test Out in Depth Map Generation in ComfyUI** * Open up the custom node manager, and you should see `ComfyUI Depth Anything TensorRT` (search for “tensorrt”) as a custom node that’s installed. Sometimes it says the node is not installed correctly, you can click in “fix it”. * You have to click on “Restart”, and reload the browser. * If you double-click on an empty part of the canvas, you should be able to search and find `Depth Anything Tensorrt`. Add it to the canvas. * If this doesn’t show up, try and see if it’s because the custom node is installed correctly. If not, you can try to “fix it”. The fixing process will take a while, since it’ll be installing some packages. * Add a ”Load Image” node as the input, and the “Preview Image” node as the output. You should be able to pick an image and see the depth map. ![Running DepthAnything in ComfyUI](https://storage.googleapis.com/papyrus_images/54bc6f9755462fa2ce78428333976b83741eb020e3e6db65cfbf066b076bc807.png) Running DepthAnything in ComfyUI Step 4: Bring Up ComfyStream Server =================================== Tunnel to open up UDP connection * On the remote server (remote\_host), run: * `apt-get install socat` * `socat TCP4-LISTEN:4321,fork UDP4:127.0.0.1:5678` * On the local machine: Open an SSH tunnel on port 4321, run `ssh -L 4321:localhost:4321 {remote_user@remote_host and other params from Runpod ssh cmd over exposed TCP port}` - make sure this SSH tunnel stays open * On the local machine, run `socat UDP4-LISTEN:1234,fork TCP4:127.0.0.1:4321` Now, traffic sent to `localhost:1234` on the local machine will be forwarded over the SSH tunnel and will reach `127.0.0.1:5678` on the remote server. Bring up ComfyStream Server on the remote server, so it can interact with the Comfy inference engine * `cd /workspace/comfystream` * `pip install -r requirements.txt` * `python install.py --workspace /workspace/ComfyUI` * Open another terminal over port 8889: * Copy that command, and add `-L 8889:localhost:8889` * For example: `ssh -L 8889:localhost:8889 {remote_user@remote_host and other params from Runpod ssh cmd over exposed TCP port}` * `python server/app.py --workspace /workspace/ComfyUI --port=8889 --media-ports=5678` We should now have the ComfyStream server running on port `8889`, with media port `5678` listening on UDP. It’s also using the same workspace and sharing the available nodes / models as the ComfyUI engine you were using. Traffic sent to `localhost:1234` on the local machine will be forwarded over the SSH tunnel and will reach `127.0.0.1:5678` on the remote server. If you are having issues with traffic tunneling, try following the TURN server instructions in the [ComfyStream repo](https://github.com/yondonfu/comfystream?tab=readme-ov-file#run-server). Step 5: Run the Front End ========================= You can run the frontend on your local machine. Just git clone the same repo, and run the following: * `cd ui` * `npm install --legacy-peer-deps` * `npm run dev` Now you should be able to visit `localhost:3000` in your browser. For stream URL, use `http://127.0.0.1:8888` , pick the “depth anything” workflow in the workflows directory from `comfystream`. After that, you should see the depth map live stream based on you webcam. [![]({{DOMAIN}}/editor/youtube/play.png)](https://www.youtube.com/watch?v=rSUOeU6zxrA) Summary ------- There are a few other projects in the Comfy ecosystem that support live video workflows. What makes ComfyStream stand out is the usage of WebRTC, which makes it possible to stream video across the internet with low latency. I’m excited to continue playing with this tool and build upon the DepthAnything workflow. This is my first public experiment in building a real time AI video pipeline. If you are interested in this technology, join the [waitlist for the Livepeer Real-time AI Video Showcase.](https://airtable.com/app6RKujYxlnB1GJN/shrqUfMoHDYgl82VY) Epilogue -------- I noticed that sometimes restarting a Runpod instance is a pain because you need to wait for a GPU to become available on your machine. They mentioned using network mounted drives can solve this issue, but I haven’t gotten around to it. Also, if you are restarting the instance, here is a number of steps to re-establish tunneling and restart the server: * SSH into your instance: `ssh -L 4321:localhost:4321 -L 8889:localhost:8889 {remote_user@remote_host and other params from Runpod ssh cmd over exposed TCP port}` * Run socat on remote server: `socat TCP4-LISTEN:4321,fork UDP4:127.0.0.1:5678` * Run socat on local machine: `socat UDP4-LISTEN:1234,fork TCP4:127.0.0.1:4321` * Load up Conda on remote server: `eval "$(/workspace/miniconda3/bin/conda shell.bash hook)"` * Activate comfystream environment: `conda activate comfystream` * `cd /workspace/comfystream` * Start comfystream server: `python server/app.py --workspace /workspace/ComfyUI --port=8889 --media-ports=5678` --- *Originally published on [Eric](https://paragraph.com/@ericxtang/comfystream-and-depthanything)*