Technology used:
- Python
- OpenCV
In this project, I attempted to implement multi-sweep space carving outlined in “A Theory of Shape by Space-Carving.” The github repo can be found here.
SCENE
I created a model of a simple figure in Blender, to which I added a color and light source. I added several cameras at different points around the figure, and rendered the images of the cameras.
I placed each camera’s associated information—namely, its position and Euler rotation angles—into a single text file, cam_info.txt. To parse this information, I created a method that converts the Euler rotation angles to u,v,w vectors and used these vectors to create associated matrices that convert coordinates to each camera’s camera space.
ALGORITHM
The algorithm sweeps through a grid of voxels in each of the six principal directions, where it splits the grid into a series of planes that it iterates through. In a plane sweep, each voxel is projected onto the image of each camera that faces in the direction of the sweep. If the standard deviation of the colors of the pixels that the voxel is projected onto are within a set threshold, the voxel is colored, and otherwise, the voxel is removed.
In a plane sweep, if a voxel is determined to be photo-consistent, then the voxels below it are no longer checked for photo-consistency. This ensures that the backside of the object is not carved out during the plane sweep.
After the set of six plane sweeps, all of the voxels set to be carved are removed from the voxel grid, and the voxels set to be colored are checked for photo-consistency again for each direction in which that voxel was colored. This process repeats until no voxels are removed after a multi-sweep.
I had some trouble deciding how to implement the consist() function for RGB colors. I tried other ways like averaging the standard deviations of the red, green, and blue components and calculating the Euclidean distance between colors and the mean color. However, summing the standard deviations of the red, green, and blue components yielded the best results, though this may be because the scene has a red object against a gray background.
I used Open3D for displaying the final volume of voxels.
RESULTS
The resultant carved voxel grid, for the most part, resembles the Blender model, though it had a hard time carving in areas with heavy shadows, the top of the head, and near the legs.
