Research
Ph. D. Thesis: 3D Reconstruction of Bone Surfaces from Ultrasound
In 3D bone reconstruction from ultrasound, there are two problems to solve: First, the delineation of the bone surface in the 2D ultrasound images. As ultrasound imaging has a low signal-to-noise ratio and suffers from various artifacts, this has been a difficult task for decades. Researchers proposed a heuristic based on signal processing and human reasoning to automatically segment the bone, but the accuracy and especially the robustness was quite low. This changed fundamentally with the rise of machine learning. By training convolutional neural networks, we were able to solve this task.
Second, ultrasound barely penetrates bones and as such, the inner part of the knee joint is occluded. Thus, the point cloud segmented in the first step needs to be completed. Statistical shape models can be used to describe the bone’s shape. By fitting these to the partial point cloud, a full model can be obtained. In the near future, I expect machine learning-based point cloud processing to become the new state of the art in this regard.
The experimental setup: The patient takes a seat and his leg is immobilized. Both, the leg and the wireless ultrasound probe are tracked with an optical tracking system. The computer in the back shows the GUI: On the left, the ultrasound image is shown and on the right, the reconstructed bone, in this case, the tibia, is displayed.
The processing pipeline: The ultrasound probe is tracked by an optical tracking system. After segmenting the bone surface, it is combined with the pose data to obtain a partial 3D model. Finally, it is extrapolated to a full femur model.
Further Projects
Full-stack implementation of ultrasound probe calibration and freehand 3D records
• Research topic of Ph. D. thesis
• Calibration method based on spheres
• Determination of sphere center in real world and in image data
• Fully automatic vision pipeline based on Hough voting technique
• API to Atracsys FTK500, Northern Digital Inc. Polaris (optical) and Aurora (electromagnetic) tracking
• API to Ultrasound system Clarius
• Frame-grabber based analog connection to Ultrasound system Ultrasonix
• Real-time segmentation of ultrasound image using a CNN
• Real-time 3D visualization of point cloud
• Implemented in C++ with Qt
Automatic annotation of ultrasound images by optical tracking of a printed phantom
• Research topic of Ph. D. thesis
• Database of knee bones from CT. CAD design including marker geometry
• 3D printing of phantom bodies
• Implementation of transformation chain, transferring the database model into the image space
• Image-based refinement step
• Implemented in Python
Synthesizing large-scale annotated training data for instrument detection in the operating room
• Supervision of industrial cooperation, team size: 6
• Task: Surgical count – checking the instruments on the instrument table
• Instrument 3D geometry available from CAD data
• Custom definition of surface material properties
• Physical simulation for realistic, overlapping, instrument placement
• Foto-realistic rendering of millions of annotated training images
• Implemented in Python using Blender
Knee implant design study based on a large-scale patient data sets
• Industrial cooperation, team size: 3
• Task: Definition of off-the-shelf (OTS) implant sizes (AP & ML)
• Large data base of implant sizes available
• Optimization of OTS sizes by partitioning of data
• Implemented in Matlab
Implementation of image processing machine learning architectures
• Research topic of Ph. D. thesis
• Task: Segmentation of the bone surface
• Data set of several thousand annotated ultrasound images
• Various architectures: U-Net, PSP-Net, Res-Net, Deeplabv3+, HR-Net, nnU-Net, Swin-Transformer, Mask-RCNN, CNN-LSTM, …
• Implementation, training, hyperparameter optimization and testing
• Implemented in Python with Tensorflow and Pytorch
The list does not end here! Get in contact to learn about other projects I implemented, also as a hobby.