Print
Univis
Search
 
FAU-Logo
Techn. Fakultät Website deprecated and outdated. Click here for the new site. FAU-Logo

Thomas Köhler M. Sc.

Alumnus of the Pattern Recognition Lab of the Friedrich-Alexander-Universität Erlangen-Nürnberg

Robust Computational Methods for Multi-Frame Super-Resolution

Confidence-Aware Levenberg-Marquardt Optimization for Joint Motion Estimation and Super-Resolution
Cosmin Bercea, Andreas Maier, Thomas Köhler
  • Extension of our IEEE TCI paper "Robust Multiframe Super-Resolution Employing Iteratively Re-Weighted Minimization"

     

    Motion estimation across low-resolution frames and the reconstruction of high-resolution images are two coupled subproblems of multi-frame super-resolution. This paper introduces a new joint optimization approach for motion estimation and image reconstruction to address this interdependence. Our method is formulated via non-linear least squares optimization and combines two principles of robust super-resolution. First, to enhance the robustness of the joint estimation, we propose a confidence-aware energy minimization framework augmented with sparse regularization. Second, we develop a tailor-made Levenberg-Marquardt iteration scheme to jointly estimate motion parameters and the high-resolution image along with the corresponding model confidence parameters. Our experiments on simulated and real images confirm that the proposed approach outperforms decoupled motion estimation and image reconstruction as well as related state-of-the-art joint estimation algorithms.

     

    Supplementary material:

    Articles in Conference Proceedings
    Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP) (International Conference on Image Processing), Phoenix, Arizona, 2016, pp. 1136-1140, 2016 (BiBTeX, Who cited this?)
Robust Multiframe Super-Resolution Employing Iteratively Re-Weighted Minimization
Thomas Köhler, Xiaolin Huang, Frank Schebesch, André Aichert, Andreas Maier, Joachim Hornegger
  • Multiframe super-resolution algorithms reconstruct high-resolution images by exploiting complementary information in multiple low-resolution frames. However, despite their success under ideal conditions, most existing methods rely on simplistic approximations to the physics of image acquisition and show limited robustness in real-world applications. This paper proposes spatially adaptive Bayesian modeling and an iterative algorithm for robust super-resolution imaging. In particular, we introduce a weighted Gaussian observation model to consider space variant noise and weighted bilateral total variation to exploit sparsity of natural images. Based on this model, we develop a majorization-minimization algorithm implemented as iteratively re-weighted minimization. The proposed method simultaneously estimates model parameters and the super-resolved image in an iterative coarse-to-fine scheme. Compared to prior work, our approach combines the benefits of achieving robust and edge preserving image reconstruction with small amount of parameter tuning, while being flexible in terms of motion models, computationally efficient and easy to implement. Our experimental evaluation confirms that our approach outperforms state-of-the-art algorithms under various practical conditions, e.g. inaccurate geometric and photometric registration or invalid measurements.

     

    Supplementary material:

Multi-Sensor Super-Resolution for Hybrid Imaging and Multi-Channel Data (with Applications in RGB-D Imaging)

A Unified Bayesian Approach to Multi-Frame Super-Resolution and Single-Image Upsampling in Multi-Sensor Imaging
Thomas Köhler, Johannes Jordan, Andreas Maier, Joachim Hornegger
  • Extension of our GCPR'14 paper "Guided Image Super-Resolution: A New Technique for Photogeometric Super-Resolution in Hybrid 3-D Range Imaging" to more than two image channels/modalities

     

    For a variety of multi-sensor imaging systems, there is a strong need for resolution enhancement. In this paper, we propose a unified method for single-image upsampling and multi-frame super-resolution of multi-channel images. We derive our algorithm from a Bayesian model that is formulated by a novel image prior to exploit sparsity of individual channels as well as a locally linear regression between the complementary channels. The reconstruction of high-resolution multi-channel images from low-resolution ones and the estimation of associated hyperparameters to define our prior model is formulated as a joint energy minimization. We introduce an alternating minimization scheme to solve this non-convex optimization problem efficiently. Our framework is applicable to various types of multi-sensor setups that are addressed in our experimental evaluation, including color, multispectral and 3-D range imaging. Comprehensive qualitative and quantitative comparisons demonstrate that our method outperforms state-of-the-art algorithms.

     

    Supplementary material:

    • Our 1-page BMVC abstract that summarizes the core ideas of our work is available Initiates file downloadhere.
    • The poster for this work is available Opens external link in new windowhere.
    • Source code: The MATLAB source code of our method is available in our Opens internal link in current windowmulti-frame super-resolution toolbox.

    Articles in Conference Proceedings
    Proceedings of the British Machine Vision Conference (BMVC) (26th British Machine Vision Conference (BMVC 2015)), Swansea, Wales, 07. - 10.09.2015, pp. 143.1-143.12, 2015, ISBN 1-901725-53-7 (BiBTeX, Who cited this?)
Guided Image Super-Resolution: A New Technique for Photogeometric Super-Resolution in Hybrid 3-D Range Imaging
Florin C. Ghesu, Thomas Köhler, Sven Haase, Joachim Hornegger
  • In this paper, we augment multi-frame super-resolution with the concept of guided filtering for simultaneous upsampling of 3-D range data and complementary photometric information in hybrid range imaging. Our guided super-resolution algorithm is formulated as joint maximum a-posteriori estimation to reconstruct high-resolution range and photometric data. In order to exploit local correlations between both modalities, guided filtering is employed for regularization of the proposed joint energy function. For fast and robust image reconstruction, we employ iteratively re-weighted least square minimization embedded into a cyclic coordinate descent scheme. The proposed method was evaluated on synthetic datasets and real range data acquired with Microsoft's Kinect. Our experimental evaluation demonstrates that our approach outperforms state-of-the-art range super-resolution algorithms while it also provides super-resolved photometric data.

     

    Supplementary material:

    Articles in Conference Proceedings

Super-Resolution for Image-Guided Surgery

Outlier Detection for Multi-Sensor Super-Resolution in Hybrid 3-D Endoscopy
Thomas Köhler, Sven Haase, Sebastian Bauer, Jakob Wasza, Thomas Kilgus, Lena Maier-Hein, Hubertus Feußner, Joachim Hornegger
  • In hybrid 3-D endoscopy, range data is used to augment photometric information for minimally invasive surgery. As range sensors suffer from a rough spatial resolution and a low signal-to-noise ratio, subpixel motion between multiple range images is used as a cue for super-resolution to obtain reliable range data. Unfortunately, this method is sensitive to outliers in range images and the estimated subpixel displacements. In this paper, we propose an outlier detection scheme for robust super-resolution. First, we derive confidence maps to identify outliers in the displacement fields by correlation analysis of photometric data. Second, we apply an iteratively re-weighted least squares algorithm to obtain the associated range confidence maps. The joint confidence map is used to obtain super-resolved range data. We evaluated our approach on synthetic images and phantom data acquired by a Time-of-Flight/RGB endoscope. The proposed method improves the median peak-signal-to-noise ratio by 1.1 dB compared to super-resolution without outlier detection.

     

    Supplementary material:

    Articles in Conference Proceedings
    Köhler, Thomas; Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Feußner, Hubertus; Hornegger, Joachim
    Bildverarbeitung für die Medizin 2014, Aachen, 2014, pp. 84-89, 2014 (BiBTeX, Who cited this?)
Multi-Sensor Super-Resolution for Hybrid Range Imaging with Application to 3-D Endoscopy and Open Surgery
Thomas Köhler, Sven Haase, Sebastian Bauer, Jakob Wasza, Thomas Kilgus, Lena Maier-Hein, Joachim Hornegger, Hubertus Feußner
  • Extension of our MICCAI'13 paper "ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy"

     

    In this paper, we propose a multi-sensor super-resolution framework for hybrid imaging to super-resolve data from one modality by taking advantage of additional guidance images of a complementary modality. This concept is applied to hybrid 3-D range imaging in image-guided surgery, where high-quality photometric data is exploited to enhance range images of low spatial resolution. We formulate super-resolution based on the maximum a-posteriori (MAP) principle and reconstruct high-resolution range data from multiple low-resolution frames and complementary photometric information. Robust motion estimation as required for super-resolution is performed on photometric data to derive displacement fields of subpixel accuracy for the associated range images. For improved reconstruction of depth discontinuities, a novel adaptive regularizer exploiting correlations between both modalities is embedded to MAP estimation. We evaluated our method on synthetic data as well as ex-vivo images in open surgery and endoscopy. The proposed multi-sensor framework improves the peak signal-to-noise ratio by 2 dB and structural similarity by 0.03 on average compared to conventional single-sensor approaches. In ex-vivo experiments on porcine organs, our method achieves substantial improvements in terms of depth discontinuity reconstruction.

     

    Supplementary material:

    Journal Articles
    Köhler, Thomas; Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Stock, Christian; Hornegger, Joachim; Feußner, Hubertus
    Medical Image Analysis, vol. 24, no. 1, pp. 220–234, 2015 (BiBTeX, Who cited this?)
ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy
Thomas Köhler, Sven Haase, Sebastian Bauer, Jakob Wasza, Thomas Kilgus, Lena Maier-Hein, Hubertus Feußner, Joachim Hornegger
  • 3-D endoscopy is an evolving field of research with the intention to improve safety and efficiency of minimally invasive surgeries. Time-of-Flight (ToF) imaging allows to acquire range data in real-time and has been engineered into a 3-D endoscope in combination with an RGB sensor (640 x 480 px) as a hybrid imaging system, recently. However, the ToF sensor suffers from a low spatial resolution (64 x 48 px) and a poor signal-to-noise ratio. In this paper, we propose a novel multi-frame super-resolution framework to improve range images in a ToF/RGB multi-sensor setup. Our approach exploits high-resolution RGB data to estimate subpixel motion used as a cue for range super-resolution. The underlying non-parametric motion model based on optical flow makes the method applicable to endoscopic scenes with arbitrary endoscope movements. The proposed method was evaluated on synthetic and real images. Our approach improves the peak-signal-to-noise ratio by 1.6 dB and structural similarity by 0.02 compared to single-sensor super-resolution.

     

    Supplementary material:

    Articles in Conference Proceedings
    Köhler, Thomas; Haase, Sven; Bauer, Sebastian; Wasza, Jakob; Kilgus, Thomas; Maier-Hein, Lena; Feußner, Hubertus; Hornegger, Joachim
    Medical Image Computing and Computer-Assisted Intervention - MICCAI 2013, Lecture Notes in Computer Science (International Conference on Medical Image Computing and Computer-Assisted Intervention), Nagoya, Japan, 2013, vol. 8149, pp. 139-146, 2013, ISBN 978-3-642-40810-6 (BiBTeX, Who cited this?)
    Haase, Sven; Köhler, Thomas; Kilgus, Thomas; Maier-Hein, Lena; Hornegger, Joachim; Feußner, Hubertus
    Computer- und Roboter Assistierte Chirurgie (CURAC 2013), Innsbruck, 28.11, pp. 194-197, 2013 (BiBTeX, Who cited this?)
GPU Accelerated Time-of-Flight Super-Resolution for Image-Guided Surgery
Jens Wetzl, Oliver Taubmann, Sven Haase, Thomas Köhler, Martin Kraus, Joachim Hornegger
  • In the field of image-guided surgery, Time-of-Flight (ToF) sensors are of interest due to their fast acquisition of 3-D surfaces. However, the poor signal-to-noise ratio and low spatial resolution of today's ToF sensors require preprocessing of the acquired range data. Super-resolution is a technique for image restoration and resolution enhancement by utilizing information from successive raw frames of an image sequence. We propose a super-resolution framework using the graphics processing unit. Our framework enables interactive frame rates, computing an upsampled image from 10 noisy frames of 200x200 px with an upsampling factor of 2 in 109 ms. The root-mean-square error of the super-resolved surface with respect to ground truth data is improved by more than 20% relative to a single raw frame.

     

    Supplementary material:

    • Source code: The source code of our super-resolution framework is available Opens internal link in current windowhere. The source code of L-BFGS used for non-linear optimization is available Opens external link in new windowhere.
    • Data: The datasets used in our evaluation are available Opens internal link in current windowhere.

    Articles in Conference Proceedings