mesh2audio

Import or generate 3D models and transform them into real-time playable physical audio models!

The generated audio model can be played in real-time by "striking" (clicking) on mesh vertices in the 3D mesh viewer or by exciting them with an audio input device (such as a microphone).

• Project video: https://youtu.be/RwxgOHVBDvc
• Example audio output is in the sound_samples directory.

This project started as the final project for Karl Hiner and Ben Wilfong for GA Tech CSE-6730 Modeling & Simulation, Spring 2023. The code as it was at the end of the school project is here. This original repo also supports 2D axisymmetric audio model generation with lower fidelity due to only having two degrees of freedom.

Work on this project continues here, including a full Vulkan rewrite and many new features.

Build app

Install dependencies

Mac

$ brew install glew llvm eigen

Linux

(Only tested on Ubuntu.)

$ sudo apt install llvm libc++-dev libc++abi-dev libeigen3-dev
$ ln -s llvm-config-17 llvm-config
$ export PATH="$(llvm-config --bindir):$PATH"

Install GTK (for native file dialogs) and OpenGL dependencies:

$ sudo apt install build-essential libgtk-3-dev libglu1-mesa-dev freeglut3-dev mesa-common-dev libglew-dev

Clone, clean, and build app

$ git clone --recurse-submodules [email protected]:khiner/mesh2audio.git
$ cd mesh2audio

• Clean:
  • Clean up everything: ./script/Clean
  • Clean debug build only: ./script/Clean -d [--debug]
  • Clean release build only: ./script/Clean -r [--release]
• Build:
  • Debug build (default): ./script/Build
  • Release build: ./script/Build -r [--release]

Debug build is generated in the ./build directory relative to the project (repo) root. Release build is generated in ./build-release.

To run the freshly built application:

# The application assumes it's being run from the build directory when locating its resource files.
$ cd build # or build-release
$ ./mesh2audio

Stack

• ImGui + SDL3: Immediate-mode UI/UX.
• Faust: Render the mesh to an audio graph, with real-time interactive vertex excitation.
• miniaudio: Continuously render the modal physical model of the input 3D volumetric mesh to audio.
• glm: Graphics math.
• OpenMesh: Main polyhedral mesh representation data structure.
• tetgen: Convert triangular 3D surface meshes into tetrahedral meshes.
• 3D FEM: VegaFEM for generating mass/stiffness matrices + Spectra for finding eigenvalues/vectors.
• ReactPhysics3D: Collision detection and physics.
• nativefiledialog-extended: Native file dialogs.
• nanosvg: Read path vertices from SVG files.
• ImPlot: Plotting.
• ImGuizmo: Mesh transform and camera rotation gizmos.
• ImSpinner: Wicked cool loading spinners for ImGui.
• libnpy: Read .npy files (used in WIP RealImpact explorer).