Drone Simulation with Cesium & Isaac Sim

Why did I start this project?

After graduating with an Aerospace Engineering degree in May 2025, I joined Dr. Bradley Denby’s research group at STARBELT from June to August 2025 to apply my programming skills alongside my aerospace engineering background. My job was to develop synthetic data generation pipelines for training Vision-Language Models (VLMs). While platforms like MATLAB/Simulink and Gazebo are popular in aerospace research, they lack the photorealistic rendering capabilities required for synthetic data generation. This project focused on whether integrating Cesium’s geospatial 3D photorealistic rendering with NVIDIA Isaac Sim’s physics-based simulation could provide a better approach for VLM training workflows.

To explore this, I developed three projects in parallel: a CesiumJS web application, an Isaac Sim environment with Cesium’s photorealistic 3D tiles, and a Python-based obstacle course simulation.

What Is Cesium?

Cesium, originally developed for aerospace applications at Analytical Graphics, Inc. (now Ansys), is an open-source platform for 3D geospatial visualization. Its 3D Tiles technology enables real-time streaming and rendering of massive datasets. For this research, I used Google Photorealistic 3D Tiles via Cesium in both the CesiumJS web application and Isaac Sim.

1. Waypoints Synthetic Data Generation of Drone in Virtual Campus

The great thing about software and programming is anyone can use it for free to make cool projects. Previously I have made various web applications and I wanted to use these skills on this research as well. Next.js is a framework built on top of React which is good at web page rendering and dynamic user interface and used to develop web application quickly. TypeScript is a superset of JavaScript and it was used for type-safe development to reduce any errors in production level. TailwindCSS is a modern, lightweight CSS framework and used it to reduce development time and make cool UI/UX quickly. CesiumJS was used to load Google 3D photorealistic scene and WebGL was used to render 3D graphics on Web interface. FFmpeg was used to record the scene from drone’s front camera perspective and make users download video-format data as synthetic data of virtual campus flight.

The workflow of this project originally started with the research team collecting GPX data from real-life drone flights. I wrote a script in TypeScript to convert the GPX format into structured JSON data for the rendering pipeline, so that a 3D drone model imported from Cesium Ion would follow those waypoints accurately. As the drone model follows the waypoints, the video capture feature I built using Next.js and React records the entire path from the drone’s perspective.

The key advantage of this approach was automation. By programmatically varying the coordinates in the GPX/JSON data, I could generate multiple synthetic flight path variations without manual effort. This dramatically reduced the time and cost compared to flying real drones repeatedly, while producing diverse training data that could feed VLM models much faster and more effectively.

---

2. Isaac Sim + Cesium Integration

The purpose of this pipeline was to explore Isaac Sim’s capabilities and evaluate its integration potential with Cesium for photorealistic synthetic data generation. The hypothesis was straightforward: more realistic simulation environments would produce higher-quality visual training data for VLM models.

Integration Workflow via OpenUSD:

Isaac Sim integrates with Cesium through the Cesium for Omniverse extension, which uses the OpenUSD (Universal Scene Description) framework. The workflow involved:

1. Installing Cesium for Omniverse extension in Isaac Sim 4.5
2. Enabling Fabric Scene Delegate for efficient 3D tile streaming
3. Loading Google Photorealistic 3D Tiles from Cesium Ion into the simulation environment

This integration allowed me to stream real-world geospatial content (photogrammetry, terrain, buildings) directly into Isaac Sim’s physics-based simulation environment.

Critical Limitation - Fabric Scene Delegate vs. PhysX Conflict:

Isaac Sim 4.5 has a significant architectural limitation: the Fabric Scene Delegate (FSD) and PhysX physics engine cannot run simultaneously. FSD is required to render Cesium’s 3D tiles efficiently, but enabling it disables PhysX physics simulation. This meant I could have photorealistic Cesium environments OR physics-accurate drone dynamics, but not both at the same time in a single pipeline.

This limitation forced me to separate the photorealistic rendering (Pipeline 2) from the physics-based obstacle navigation (Pipeline 3), as combining them was technically impossible with the current software versions.

3. Drone Obstacle course using Python and Isaac Sim

Tech Stack

• NVIDIA Isaac Sim 4.5 — The main simulation application built on Omniverse, providing GPU-based PhysX physics and RTX rendering for physically-based virtual environments
• OpenUSD (Universal Scene Description) — The core format at the heart of Isaac Sim that allows me to programmatically create and manipulate 3D scenes
• PhysX — High-fidelity GPU-accelerated physics engine simulating gravity, rotor thrust, and collisions
• Python — Used to write standalone scripts for scene creation and ROS2 integration
• ROS2 (Humble) — Robot Operating System for drone control and teleoperation
• Linux (Ubuntu 22.04) — Required operating system for ROS2 Humble compatibility
• Docker — Containerization to package Ubuntu 22 + ROS2 + Isaac Sim for reproducible setup

The main reason I built this pipeline with Python was to make the process reproducible. Anyone who wants to repeat this setup for ROS2 integration with Isaac Sim can just run the Python scripts instead of manually clicking through the GUI.

Creating the Scene with Python:

I used Isaac Sim’s Python API to build the simulation scene programmatically. This included setting up the physics scene with gravity (9.81 m/s² pointing down), loading the quadcopter model, and creating the obstacle course with color tiles. The Python scripts also configured the PhysX physics engine to simulate realistic drone dynamics.

Controlling the Rotors:

The Python code controlled each of the four rotors independently by setting their angular velocities, which generated thrust to make the drone fly. This physics-based approach simulated realistic quadcopter flight behavior.

---

Challenges and What I Learned

1. Open-Source Platform Compatibility Issues

When I first joined the team in June 2025, I spent most of my time trying to integrate Cesium with O3DE (Open 3D Engine) to create a unified simulation environment. The idea was to combine Cesium’s photorealistic 3D tiles with O3DE’s physics simulation in one open-source platform. Unfortunately, the O3DE community had stopped actively maintaining the project, which caused version incompatibilities between Cesium’s SDK and O3DE’s rendering pipeline. After weeks of debugging, I realized this approach wasn’t going to work. In early July, I switched to NVIDIA Isaac Sim instead. This turned out to be the right decision since Isaac Sim had better integration support through Omniverse and active development.

2. ROS2 and Isaac Sim Version Dependencies

Switching to Isaac Sim solved the O3DE compatibility problem, but created a new dependency issue. Running Isaac Sim 4.5 required specific versions of ROS2 (Humble) and Ubuntu 22.04, which conflicted with my local development environment. If I installed these directly, I would have had to reformat my system or set up dual-boot, which wasn’t practical for a 3-month project.

Solution: Docker Containerization

I learned how to use Docker to run Ubuntu 22.04 with ROS2 Humble in isolated containers. This let me test teleoperation and drone control without messing up my local system. Here’s the workflow I created:

1. Launch Isaac Sim with the ROS2 Humble bridge enabled and export the necessary environment variables
2. Start my custom Docker container (includes Ubuntu 22.04, ROS2 Humble, and all required packages)
3. Inside the Docker container, launch RViz2 to monitor the Carter robot model’s built-in camera feed from the simulation
4. In another terminal within the container, run teleop_twist_keyboard to control the robot in real-time using keyboard inputs

This Docker approach solved the dependency problem and made the entire setup reproducible. Other researchers could just run my Docker container instead of manually configuring their environment, which saved a lot of setup time.

3. Timeline Limitations and Future Work

Given the 3-month research timeline (June-August 2025), I successfully completed three synthetic data pipelines but didn’t have time to fully integrate ROS2 with my custom drone obstacle course simulation (Pipeline 3).

My original plan was to extend Pipeline 3 by integrating ROS2 Humble for real-time drone teleoperation and autonomous navigation. I had learned the ROS2 + Isaac Sim workflow through tutorials (using the Apriltag robot example shown above), and developed the Python scripts for scene creation and physics simulation. The next step would have been to combine these components: connecting my custom quadcopter model to ROS2 topics, implementing built-in camera to create synthetic data in simulation, and testing collision avoidance in the obstacle course environment.

---

References

1. Vision Language Models — NVIDIA Glossary
2. Cesium 3D Tileset Essentials
3. NVIDIA Isaac Sim Documentation
4. ROS2 Humble Documentation
5. Starbelt Research Group
6. Fabric Scene Delegate in Omniverse