This is my Bachelor's thesis. The aim was to develop a tool for Unity which allowed the estimation of various simulation parameters within Unity. It includes a differentiable physics engine built from scratch in C++, a Python module that does the optimizing using backpropagation to calculate the gradient of the loss function and a Unity project to display the results.
The framework is able to generate a simulation with some parameters for the mass and stiffness of a cloth and use numerical optimization combined with differentiable simulation and the backpropagation algorithm to guess those parameters based on the position of that simulation over many frames.
sudo dnf install glslc python3-develWayland development headers (wayland-devel) are typically preinstalled on desktop
systems. If you prefer X11 add the X11 devel packages described in the Ubuntu
section below and set GLFW_BUILD_WAYLAND=OFF GLFW_BUILD_X11=ON in the viewer
CMakeLists.
sudo apt install cmake g++ \
libvulkan-dev glslang-tools glslc \
libglfw3-dev libxinerama-dev libxrandr-dev libxcursor-dev libxi-dev \
python3-devbrew install cmake glfw molten-vk glslangInstall Vulkan SDK, CMake, and
Visual Studio 2022 with C++ workload. The Vulkan SDK bundles glslc, headers,
and the loader. FetchContent handles Eigen, nlohmann_json, pybind11, GLFW,
and ImGui automatically.
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -jOptions (all default to ON):
| Flag | Effect |
|---|---|
-DDPE_BUILD_PYTHON=OFF |
Skip Python module (no Python headers required) |
-DDPE_BUILD_UNITY_BRIDGE=OFF |
Skip Unity C API shared library |
-DDPE_BUILD_TESTS=OFF |
Skip unit tests |
-DDPE_BUILD_VIEWER=OFF |
Skip standalone Vulkan viewer |
build/python/DifferentiablePhysicsEngine.*.so— Python module. Addbuild/pythontoPYTHONPATHandimport DifferentiablePhysicsEngine.build/unity/libDifferentiablePhysicsEngine.so— Unity native plugin (copied automatically intoUnityProject/Assets/).build/dpe_tests— unit test executable. Run withctest --test-dir build.build/dpe_viewer— standalone Vulkan real-time viewer.
build/dpe_viewer UnityProject/Assets/Files/side.txtControls: left-click drag to orbit, W/S to zoom, R to reset camera, Esc to quit.
python -m pip install -r requirements.txtThe framework has four components that work together:
┌──────────────┐
│ Scene JSON │ ← hand-authored or exported from Unity
└──────┬───────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────┐ ┌───────────────┐
│ Python Optimizer │ │ dpe_core │ │ Vulkan Viewer │
│ (scipy L-BFGS-B)│◄┤ engine ├►│ (real-time) │
└────────┬─────────┘ │ (C++17) │ └───────────────┘
│ └────┬─────┘
▼ │
┌─────────────────┐ │
│ Unity (optional) │◄─────┘ (via C API bridge)
└─────────────────┘
dpe_core — the physics engine. A mass-spring cloth simulator with implicit
Euler integration. Crucially it is differentiable: every force has an
analytical Jacobian (dF/dx, dF/dv), and the whole integration step can be
backpropagated through. This is what lets the optimizer compute gradients
without finite differences.
Python optimizer — wraps the engine via pybind11. Given a scene JSON with known parameters and a target animation, scipy's L‑BFGS‑B minimizes the mean-squared position error by adjusting mass/stiffness values. The gradient is computed by backpropagating through every simulation step (adjoint method).
Vulkan viewer — standalone C++ application. Loads a scene JSON, runs the simulation on a background thread, and renders the cloth mesh in real time with an orbit camera. No Unity or Python required.
Unity (optional) — the original visualization front-end. The engine is loaded as a native plugin via P/Invoke and the cloth is rendered with Unity's mesh system.
Simulations are defined by JSON files (see UnityProject/Assets/Files/ for
examples). Key fields:
optimizationSettings controls which parameters the optimizer tunes.
A two-character string per object:
| Mode | Meaning |
|---|---|
n / N |
Ignore this parameter (keep it fixed) |
G / g |
Treat as one global value for the object |
L / l |
Treat as a local value per vertex / spring |
Example: "nL" = keep mass fixed, optimize every spring stiffness individually.
Build the Python module (requires python3-devel):
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -jAdd the module to your Python path and install dependencies:
export PYTHONPATH="$PWD/build/python:$PYTHONPATH" # Linux / macOS
pip install -r requirements.txtRun one of the experiment scripts from the Python/ directory:
cd Python
python Experiment1.pyExperiment1.py recovers the cloth parameters for each scene
(side, corners, flag) with four parameterizations (homogeneous /
heterogeneous mass, homogeneous / heterogeneous stiffness) and prints a
results table. Experiment2.py repeats this with different numbers of
simulation steps to study how the error metric behaves (thesis Chapter 5).
The scripts load scene files from UnityProject/Assets/Files/, run the
L‑BFGS‑B optimization, print the results, and write optimized scene files into
UnityProject/Assets/Files/Experiment1/ and Experiment2/. You can then open
these in the viewer:
build/dpe_viewer UnityProject/Assets/Files/Experiment1/side_nL_optimized.txtThe green wireframe shows the simulation with the recovered parameters; visually it should closely follow the target animation you generated.
The error metric is the root mean square position difference between the optimized simulation and the target, per vertex per frame. Reported values below were produced on a Ryzen-class desktop (Release build); runtimes scale roughly with the number of simulation steps and parameters.
Each scene is optimized with four parameterizations — Gn (one global mass),
Ln (per-vertex masses), nG (one global stiffness), nL (per-spring
stiffnesses).
| Scene | Settings | Error | Time (s) |
|---|---|---|---|
| side (one edge fixed) | Gn |
0.18 | 10.5 |
Ln |
1.08 | 7.7 | |
nG |
0.03 | 17.7 | |
nL |
0.06 | 10.0 | |
| corners (four corners fixed) | Gn |
0.05 | 8.2 |
Ln |
0.04 | 27.9 | |
nG |
0.06 | 12.1 | |
nL |
0.17 | 30.4 | |
| flag (wind) | Gn |
1.93 (fails to converge) | 0.1 |
Ln |
0.51 | 3.9 | |
nG |
0.01 | 0.7 | |
nL |
0.00 | 27.0 |
Observations:
- Local parameterizations (
nL) recover the parameters almost perfectly (error ≤ 0.17 everywhere), confirming the analytic gradients through the implicit solver are correct. - Global stiffness (
nG) matches the target well even though the target was generated with per-spring stiffnesses — the optimizer finds the single value that best reproduces the overall behavior. - The flag scene with a single global mass (
Gn) cannot be fitted: the optimizer diverges immediately (error 1.93). A cloth under wind needs spatial parameter variation.
Experiment2.py optimizes heterogeneous stiffness (nL). Each row is a
separate optimization run, fitted over N frames (20, 40, ..., 200). Two errors
are reported per run:
- Error (N steps) — how well the recovered parameters reproduce the target over the same N frames they were fitted to (the training error).
- Error (500 steps) — the same recovered parameters, re-simulated over 500 frames and compared to the 500-frame target. This is not a separate optimization: it validates whether the N-step solution generalizes beyond the horizon it was fitted to (thesis §5.4.1 — a solution can fit the first N frames perfectly and diverge later).
| Scene | Steps | Error (N steps) | Error (500 steps) | Time (s) |
|---|---|---|---|---|
| side | 20 | 0.00 | 0.05 | 8.3 |
| 40 | 0.00 | 0.05 | 18.5 | |
| 60 | 0.03 | 0.06 | 8.3 | |
| 80 | 0.04 | 0.06 | 10.1 | |
| 100 | 0.06 | 0.07 | 9.9 | |
| 120 | 0.06 | 0.10 | 11.9 | |
| 140 | 0.06 | 0.07 | 20.2 | |
| 160 | 0.07 | 0.08 | 20.0 | |
| 180 | 0.06 | 0.08 | 16.1 | |
| 200 | 0.06 | 0.07 | 36.6 | |
| flag | 20 | 0.00 | 0.85 | 29.8 |
| 40 | 0.07 | 330.63 | 4.6 | |
| 60 | 0.13 | 0.99 | 3.8 | |
| 80 | 0.76 | 390.22 | 6.3 | |
| 100 | 0.97 | 1.37 | 10.2 | |
| 120 | 1.81 | 41.94 | 13.3 | |
| 140 | 1.62 | 1.56 | 8.7 | |
| 160 | 1.07 | 176.26 | 101.7 | |
| 180 | 91.91 | 463.32 | 43.8 | |
| 200 | 112.72 | 446.19 | 60.4 |
Observations:
- The side scene is robust: the recovered parameters are essentially perfect at every step count (error ≤ 0.07, and ≤ 0.10 on the 500-frame validation).
- The flag scene confirms the thesis §5.4.1 finding: solutions can fit the optimization horizon well and diverge later. E.g. with 40 steps the training error is 0.07 but the 500-frame validation error is 330.63. Long horizons (180–200 steps) make the optimizer fall into bad local minima altogether (training error jumps to ~100).
- The 500-step validation is what makes the difference visible — the training error alone would hide the divergence.
-
Build the Unity bridge:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -jThe build automatically copies the native library into
UnityProject/Assets/. -
Open
UnityProject/in Unity 2020.3 LTS or newer. -
Open the
Simulationscene inAssets/Scenes/. -
Press Play. The cloth simulates in real time using the C++ engine running on a separate thread, with vertex positions streamed back to Unity via the C API bridge.
-
The
OptimizerGameObject in the scene can invoke the Python optimizer at runtime (requires Python 3 installed and the engine module importable).
{ "title": "my scene", "delta": 0.02, // time step (seconds) "integrationMethod": 2, // 0=explicit 1=symplectic 2=implicit "tolerance": 0.001, // CG solver tolerance "optimizationIterations": 100, // number of simulation frames "forwardSubSteps": 1, // substeps per frame "printTimes": false, "windVel": {"x":0, "y":0, "z":0}, "objects": [{ "vertPos": [{"x":0,"y":1,"z":0}, …], "vertIsFixed": [true, false, …], "vertMass": [1.0, 1.0, …], "springs": [0, 1, 1, 2, …], // pairs of vertex indices "springStiffness": [200.0, 200.0, …], "triangles": [0, 1, 2, …], // triples of vertex indices "dragCoefficient": 1.0, "damping": 0.5, "optimizationSettings": "nG" // mass-mode + stiffness-mode: }], // n=ignore G=global L=local "colliders": [] }