Techn. Fakultät Willkommen am Institut für Informatik FAU-Logo

Time-of-Flight Endoscopy

Research at the LME

The idea behind Time-of-Flight endoscopy is to augment conventional endoscopic 2-D data with additional 3-D surface information. In collaboration with Richard Wolf GmbH, a first 2-D/3-D endoscope was developed recently. This prototype is capable of acquiring both 3-D distanceand 2-D photometric color measurements that are aligned using camera calibration. Combining these complementary data holds potentials for applications in minimally invasive procedures. Among others, potential applications include instrument detection, tracking or navigation assistance using augmented reality.

This project is in collaboration with the German Cancer Research Center(DKFZ) in Heidelberg and the Minimally invasive InterdisciplinaryTherapeutical Intervention (MITI) group of the Technical University (TU) Munich.


Patch Based Specular Reflection Removal for Range Images in Hybrid 3-D Endoscopy
Sven Haase, Jakob Wasza, Mustafa Safak, Thomas Kilgus, Lena Maier-Hein, Hubertus Feußner, Joachim Hornegger
  • In minimally invasive surgery, hybrid 3-D endoscopy is an evolving field of research that aims to augment conventional video based systems by metric 3-D measurements. One crucial issue with these systems arises from specular reflections that result in overexposed RGB values and invalid range measurements. In this paper, we address this problem by registering video and range information acquired from different view points using a patch based approach. This allows to replace invalid measurements caused by specular reflections in one view with valid data from non-specular regions in the other view. In contrast to previous approaches that employ interpolation techniques our method utilizes actual scene information being advantageous in a medical environment. In our experiments, we show that our method decreases the mean absolute error in common situations on average by more than 30% compared to conventional interpolation. Moreover, for challenging scenarios we outperform interpolation by more than 1 mm and reconstruct important structures that inherently could not be restored by conventional interpolation.

Temporal Non-Local-Means Filtering in Hybrid 3-D Endoscopy
Tobias Lindenberger, Sven Haase, Jakob Wasza, Thomas Kilgus, Lena Maier-Hein, Hubertus Feuner, Joachim Hornegger
  • Time-of-Flight (ToF) cameras are a novel and fast developing technology for acquiring 3-D surfaces. In recent years they have gathered interest from many fields including 3-D endoscopy. However, preprossessing of the obtained images is absolutely mandatory due to the low signal-to-noise ratio of current sensors. One possibility to increase image quality is the non-local-means (NLM) filter that utilizes local neighborhoods for denoising. In this paper we present an enhanced NLM filter for hybrid 3-D endoscopy. The introduced filter gathers the structural information from an RGB image that shows the same scene as the range image. To cope with camera movements, we incorporate a temporal component by considering a sequence of frames. Evaluated on simulated data, the algorithm showed an improvement in range accuracy of 70% when compared to the unfiltered image.

Laparoscopic Instrument Localization using a 3-D Time-of-Flight/RGB Endoscope
Sven Haase, Jakob Wasza, Thomas Kilgus, Joachim Hornegger
  • Minimally invasive procedures are of importance in modern surgery due to reduced operative trauma and recovery time. To enable robot assisted interventions, automatic tracking of endoscopic tools is an essential task. State-of-the-art techniques rely on 2-D color information only which is error prone for varying illumination and unpredictable color distribution within the human body. In this paper, we use a novel 3-D Time-of-Flight/RGB endoscope that allows to use both color and range information to locate laparoscopic instruments in 3-D. Regarding color and range information the proposed technique calculates a score to indicate which information is more reliable and adopts the next steps of the localization procedure based on this reliability. In experiments on real data the tool tip is located with an average 3-D distance error of less than 4 mm compared to manually labeled ground truth data with a frame-rate of 10 fps.

    Articles in Conference Proceedings
    Haase, Sven; Köhler, Thomas; Kilgus, Thomas; Maier-Hein, Lena; Hornegger, Joachim; Feußner, Hubertus
    Computer- und Roboter Assistierte Chirurgie (CURAC 2013), Innsbruck, 28.11, pp. 194-197, 2013 (BiBTeX, Who cited this?)
    Articles in Conference Proceedings
    IEEE Workshop on Applications of Computer Vision (WACV), Clearwater, 18.01, pp. 449-454, 2013 (BiBTeX, Who cited this?)
GPU Accelerated Time-of-Flight Super-Resolution for Image-Guided Surgery
Jens Wetzl, Oliver Taubmann, Sven Haase, Thomas Köhler, Martin Kraus, Joachim Hornegger
ToF/RGB Sensor Fusion for Augmented 3D Endoscopy using a Fully Automatic Calibration Scheme
S. Haase, C. Forman, T. Kilgus, R. Bammer, L. Maier-Hein, J. Hornegger
  • 3-D Endoscopy is an evolving field of research and offers great benefits for minimally invasive procedures. Besides the pure topology, color texture is an inevitable feature to provide an optimal visualization. Therefore, in this paper, we propose a sensor fusion of a Time-of-Flight (ToF) and an RGB sensor. This requires an intrinsic and extrinsic calibration of both cameras. In particular, the low resolution of the ToF camera (64x50 px) and inhomogeneous illumination precludes the use of standard calibration techniques. By enhancing the image data the use of self-encoded markers for automatic checkerboard detection, a re-projection error of less than 0.23 px for the ToF camera was achieved. The relative transformation of both sensors for data fusion was calculated in an automatic manner.

    Articles in Conference Proceedings
    Haase, Sven; Forman, Christoph; Kilgus, Thomas; Bammer, Roland; Maier-Hein, Lena; Hornegger, Joachim
    Bildverarbeitung für die Medizin (Bildverarbeitung für die Medizin 2012), Berlin, 19.03, pp. 111-116, 2012, ISBN 978-3-642-28501-1 (BiBTeX, Who cited this?)
    Journal Articles
    Haase, Sven; Forman, Christoph; Kilgus, Thomas; Bammer, Roland; Maier-Hein, Lena; Hornegger, Joachim
    Current Medical Imaging Reviews, vol. 9, no. 2, pp. 113-119, 2013 (BiBTeX, Who cited this?)
Optimierte endoskopische Time-of-Flight Oberflächenrekonstruktion durch Integration eines Struktur-durch-Bewegung-Ansatzes
A. Groch, S.Haase, M. Wagner, T.Kilgus, H. Kenngott, H.-P. Schlemmer, J. Hornegger, H.-P. Meinzer, L. Maier-Hein
  • Eine der größten Herausforderungen computergestützter Assistenzsysteme für laparoskopische Eingriffe ist die intraoperative akkurate und schnelle Rekonstruktion der Organoberfläche. Während Rekonstruktionstechniken basierend auf Multiple View Methoden, beispielsweise Stereo-Rekonstruktion, schon länger Gegenstand der Forschung sind, wurde erst kürzlich das weltweit erste Time-of-Flight (ToF) Endoskop vorgestellt. Die Vorteile gegenüber Stereo liegen in der hohen Aktualisierungsrate und dem dichten Tiefenbild unabhängig von der betrachteten Szene. Demgegenüber stehen allerdings Nachteile wie schlechte Genauigkeit bedingt durch hohes Rauschen und systematische Fehler. Um die Vorteile beider Verfahren zu vereinen, wird ein Konzept entwickelt, die ToF-Endoskopie-Technik mit einem stereoähnlichen Multiple-View-Ansatz (Struktur durch Bewegung) zu fusionieren. Der Ansatz benötigt keine zusätzliche Bildgebungsmodalität wie z.B. ein Stereoskop, sondern nutzt die ohnehin akquirierten (Mono-) Farbdaten des ToF-Endoskops. Erste Ergebnisse zeigen, dass die Genauigkeit der Oberflächenrekonstruktion mit diesem Ansatz verbessert werden kann.


    Articles in Conference Proceedings
    Groch, Anja; Haase, Sven; Wagner, Martin
    Bildverarbeitung für die Medizin (Bildverarbeitung für die Medizin 2012), Berlin, 19.03, pp. 39-44, 2012, ISBN 978-3-642-28501-1 (BiBTeX, Who cited this?)