Xinxin Zhang, Christopher Gilliam, and Thierry Blu
We show several comparing experimental results. The registered image is put on top of the target image. When your mouse pointer hovers over the image, you can see the target image. Perfect alignment is achieved when you cannot see the movement between the registered image and the target image. The salience correlation (SalC), parsimony (Pars) and computation time are shown below each image.
Click images to show the displacement fields of different algorithms. Each color of the color coding for the deformation field represents a different direction of the deformation.
All algorithms are run on an Intel Core i7-5930K CPU @ 3.50 GHz with 64 GB RAM, using MATLAB R2016b.
Download
[PDF] [Supplemental material] [dataset] [MATLAB Code]Quick navigation
| Example 1. The size of the images is 480 × 640. | |||||
| target & source | |||||
|
|
|||||
| SalC: 15% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
|
||
| SalC: 62%, Pars: 1.45 | SalC: 89%, Pars: 1.00 | SalC: 43%, Pars: 0.97 | SalC: 60%, Pars: 1.02 | ||
| Time: 1.5s | Time: 6.6s | Time: 19.2s | Time: 1763.8s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
|
||
| SalC: 89%, Pars: 1.38 | SalC: 33%, Pars: 1.50 | SalC: 36%, Pars: 4.13 | SalC: 43%, Pars: 1.00 | ||
| Time: 9.4s | Time: 152.8s | Time: 46.3s | Time: 10.4s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
|
||
| SalC: 25%, Pars: 1.35 | SalC: 79%, Pars: 1.74 | SalC: 42%, Pars: 1.87 | SalC: 89%, Pars: 1.00 | ||
| Time: 419.9s | Time: 37.2s | Time: 229.4s | Time: 4.4s | ||
| Example 2. The size of the images is 1600 × 1600. | |||||
| target & source | |||||
|
|||||
| SalC: 15% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC: 15% | SalC: 21%, Pars: 1.05 | SalC: 24%, Pars: 1.32 | SalC: 45%, Pars: 0.94 | ||
| Time: 14.8s | Time: 61.3s | Time: 46.6s | Time: 1338.7s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 61%, Pars: 1.67 | SalC: 23%, Pars: 1.32 | SalC: 24%, Pars: 9.44 | SalC: 25%, Pars: 0.94 | ||
| Time: 77.3s | Time: 23.3s | Time: 312.3s | Time: 41.4s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 77%, Pars: 1.07 | SalC: 59%, Pars: 2.37 | SalC: 14%, Pars: 7.52 | SalC: 66%, Pars: 0.94 | ||
| Time: 3929.2s | Time: 325.3s | Time: 227.4s | Time: 60.1s | ||
| Example 3. The size of the images is 960 × 1280. | |||||
| target & source | |||||
|
|||||
| SalC: 11% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC: 16%, Pars: 3.44 | SalC: 39%, Pars: 1.04 | SalC: 20%, Pars: 0.76 | SalC: 55%, Pars: 0.91 | ||
| Time: 9.3 s | Time: 40.3 s | Time: 21.1 s | Time: 1464.4 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 64%, Pars: 1.02 | SalC: 12%, Pars: 1.50 | SalC: 17%, Pars: 7.85 | SalC: 25%, Pars: 0.97 | ||
| Time: 48.8 s | Time: 11.2 s | Time: 85.6 s | Time: 25.3 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 37%, Pars: 1.31 | SalC: 35%, Pars: 3.46 | SalC: 19%, Pars: 1.46 | SalC: 79%, Pars: 0.97 | ||
| Time: 1766.9 s | Time: 152.8 s | Time: 139.0 s | Time: 38.6 s | ||
| Example 4. The size of the images is 780 × 1040. | |||||
| target & source | |||||
|
|||||
| SalC: 15% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:91%, Pars: 1.29 | SalC: 25%, Pars: 1.12 | SalC: 35%, Pars: 0.99 | SalC: 57%, Pars: 1.02 | ||
| Time: 4.6 s | Time: 19.2 s | Time: 17.3 s | Time: 1744.1 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 71%, Pars: 1.99 | SalC: 32%, Pars: 1.44 | SalC: 29%, Pars: 6.16 | SalC: 36%, Pars: 0.98 | ||
| Time: 12.6 s | Time: 53.9 s | Time: 73.3 s | Time: 16.7 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 23%, Pars: 1.35 | SalC: 82%, Pars: 1.72 | SalC: 24%, Pars: 3.79 | SalC: 92%, Pars: 0.99 | ||
| Time: 1766.5 s | Time: 100.2 s | Time: 226.5 s | Time: 11.9 s | ||
| Example 5. The size of the images are 583 × 583. | |||||
| target & source | |||||
|
|||||
| SalC: 21% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:44%, Pars: 1.66 | SalC: 21%, Pars: 0.96 | SalC: 20%, Pars: 1.00 | SalC: 13%, Pars: 1.10 | ||
| Time: 1.9 s | Time: 8.4 s | Time: 24.7 s | Time: 566.1 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 21%, Pars: 2.21 | SalC: 17%, Pars: 1.81 | SalC: 27%, Pars: 4.89 | SalC: 19%, Pars: 2.22 | ||
| Time: 13.0 s | Time: 4.1 s | Time: 116.3 s | Time: 26.7 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 25%, Pars: 4.03 | SalC: 23%, Pars: 8.26 | SalC: 19%, Pars: 5.23 | SalC: 52%, Pars: 0.08 | ||
| Time: 956.6 s | Time: 41.7 s | Time: 97.1 s | Time: 7.6 s | ||
| Example 6. The size of the images is 720 × 540. | |||||
| target & source | |||||
|
|||||
| SalC: 9% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:18%, Pars: 3.34 | SalC: 13%, Pars: 1.17 | SalC: 15%, Pars: 0.97 | SalC: 22%, Pars: 0.46 | ||
| Time: 2.0 s | Time: 9.2 s | Time: 24.5 s | Time: 448.2 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 32%, Pars: 1.56 | SalC: 26%, Pars: 2.04 | SalC: 58%, Pars: 4.83 | SalC: 20%, Pars: 1.44 | ||
| Time: 15.5 s | Time: 11.1 s | Time: 157.9 s | Time: 27.2 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 48%, Pars: 2.41 | SalC: 65%, Pars: 0.95 | SalC: 22%, Pars: 2.90 | SalC: 53%, Pars: 0.39 | ||
| Time: 912.9 s | Time: 48.8 s | Time: 313.2 s | Time: 16.4 s | ||
Notice that, here, the MIRT algorithm and the SIFT flow algorithm yield better alignments than our algorithm (larger SalC); yet, this alignment is incorrect (the overlap region is too large: only 8 rows of tiles are common to the original images, not 9 (for MIRT) or 11 (for SIFT flow)) and this is betrayed by a higher complexity of the displacement field—the parsimony. |
|||||
| Example 7. The size of the images are 1080 × 1440. | |||||
| target & source | |||||
|
|||||
| SalC: 8% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:10%, Pars: 3.12 | SalC: 8%, Pars: 1.05 | SalC: 13%, Pars: 0.65 | SalC: 43%, Pars: 0.14 | ||
| Time: 8.5 s | Time: 36.4 s | Time: 27.7 s | Time: 1464.6 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 37%, Pars: 0.67 | SalC: 9%, Pars: 1.12 | SalC: 12%, Pars: 8.32 | SalC: 16%, Pars: 0.13 | ||
| Time: 98.3 s | Time: 16.2 s | Time: 166.2 s | Time: 25.3 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 16%, Pars: 1.30 | SalC: 16%, Pars: 1.80 | SalC: 8%, Pars: 5.97 | SalC: 55%, Pars: 0.16 | ||
| Time: 3772.6 s | Time: 201.0 s | Time: 247.4 s | Time: 30.7 s | ||
| Example 8. The size of the images is 1440 × 1080. | |||||
| target & source | |||||
|
|||||
| SalC: 12% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:26%, Pars: 2.78 | SalC: 32%, Pars: 0.71 | SalC: 32%, Pars: 0.71 | SalC: 37%, Pars: 0.72 | ||
| Time: 14.0 s | Time: 57.9 s | Time: 29.4 s | Time: 886.7 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 24%, Pars: 1.53 | SalC: 27%, Pars: 1.35 | SalC: 41%, Pars: 7.52 | SalC: 31%, Pars: 0.96 | ||
| Time: 38.6 s | Time: 13.3 s | Time: 581.4 s | Time: 32.9 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 48%, Pars: 2.07 | SalC: 35%, Pars: 4.18 | SalC: 29%, Pars: 1.41 | SalC: 46%, Pars: 0.73 | ||
| Time: 3698.0 s | Time: 198.7 s | Time: 148.9 s | Time: 27.1 s | ||
| Example 9. The size of the images is 900 × 1200. | |||||
| target & source | |||||
|
|||||
| SalC: 15% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:48%. Pars: 1.26 | SalC: 59%, Pars: 0.50 | SalC: 36%, Pars: 0.51 | SalC: 30%, Pars: 0.68 | ||
| Time: 10.1 s | Time: 44.5 s | Time: 30.6 s | Time: 1837.1 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 51%, Pars: 2.04 | SalC: 22%, Pars: 1.10 | SalC: 43%, Pars: 4.31 | SalC: 30%, Pars: 1.87 | ||
| Time: 64.3 s | Time: 50.5 s | Time: 269.4 s | Time: 39.3 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 23%, Pars: 1.52 | SalC: 54%, Pars: 1.98 | SalC: 29%, Pars: 1.44 | SalC: 59%, Pars: 0.50 | ||
| Time: 1748.5 s | Time: 136.3 s | Time: 155.2 s | Time: 22.0 s | ||
| Example 10. The size of the images is 640 × 853. | |||||
| target & source | |||||
|
|||||
| SalC: 20% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:52%, Pars: 0.91 | SalC: 23%, Pars: 1.35 | SalC: 32%, Pars: 0.59 | SalC: 44%, Pars: 0.55 | ||
| Time: 3.2 s | Time: 13.5 s | Time: 30.7 s | Time: 2190.8 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 22%, Pars: 0.41 | SalC: 20%, Pars: 1.42 | SalC: 26%, Pars: 5.79 | SalC: 29%, Pars: 0.78 | ||
| Time: 31.4 s | Time: 7.2 s | Time: 44.2 s | Time: 16.8 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 61%, Pars: 1.12 | SalC: 54%, Pars: 1.31 | SalC: 20%, Pars: 4.04 | SalC: 58%, Pars: 0.57 | ||
| Time: 943.4 s | Time: 68.8 s | Time: 315.2 s | Time: 4.5 s | ||
| Example 11. The size of the images is 960 × 1280. | |||||
| target & source | |||||
|
|||||
| SalC: 12% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:33%, Pars: 1.54 | SalC: 20%, Pars: 0.84 | SalC: 11%, Pars: 0.54 | SalC: 14%, Pars: 0.86 | ||
| Time: 7.1 s | Time: 30.1 s | Time: 25.6 s | Time: 1842.3 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 20%, Pars: 6.38 | SalC: 14%, Pars: 2.05 | SalC: 21%, Pars: 5.34 | SalC: 18%, Pars: 1.85 | ||
| Time: 56.4 s | Time: 9.3 s | Time: 317.4 s | Time: 48.7 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 14%, Pars: 2.05 | SalC: 33%, Pars: 3.17 | SalC: 27%, Pars: 4.88 | SalC: 34%, Pars: 0.13 | ||
| Time: 956.6 s | Time: 154.5 s | Time: 100.8 s | Time: 22.1 s | ||
| Example 12. The size of the images is 960 × 1280. | |||||
| target & source | |||||
|
|||||
| SalC: 13% | |||||
| (a) SelFlow [1] | (b) ECC [2] | (c) NTG [3] | (d) LAFP [4] | ||
|
|
|
|
||
| SalC:35%, Pars: 0.51 | SalC: 29%, Pars: 0.11 | SalC: 35%, Pars: 0.21 | SalC: 35%, Pars: 0.64 | ||
| Time: 6.8 s | Time: 28.9 s | Time: 22.6 s | Time: 2382.1 s | ||
| (e) LAP [5] | (f) Demons [6] | (g) MIRT [7] | (h) bUnwarpJ [8] | ||
|
|
|
|
||
| SalC: 16%, Pars: 1.40 | SalC: 20%, Pars: 2.42 | SalC: 22%, Pars: 4.44 | SalC: 13%, Pars: 0.10 | ||
| Time: 9.5 s | Time: 80.2 s | Time: 62.5 s | Time: 13.6 s | ||
| (i) GIT [9] | (j) SIFT flow [10] | (k) Elastix [11] | (l) Ours | ||
|
|
|
|
||
| SalC: 23%, Pars: 3.60 | SalC: 35%, Pars: 1.80 | SalC: 29%, Pars: 4.38 | SalC: 36%, Pars: 0.20 | ||
| Time: 1910.3 s | Time: 150.8 s | Time: 310.1 s | Time: 29.3 s | ||
| [1] | Liu P, Lyu M, King I, et al. SelFlow: Self-Supervised Learning of Optical Flow. |
| [2] | G. D. Evangelidis and E. Z. Psarakis, “Parametric image alignment using enhanced correlation coefficient maximization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, pp. 1858–1865, 2008. |
| [3] | S.-J. Chen, H.-L. Shen, C. Li, and J. H. Xin, “Normalized total gradient: a new measure for multispectral image registration,” IEEE Transactions on Image Processing, vol. 27, no. 3, pp. 1297–1310, 2018. |
| [4] | Y. Matsushita, “Aligning images in the wild,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 1–8. |
| [5] | C. Gilliam and T. Blu, “Local all-pass geometric deformations,” IEEE Transactions on Image Processing, vol. 27, no. 2, pp. 1010–1025, 2018. |
| [6] | H. Lombaert, L. Grady, and X. P. et al., “Diffeomorphic demons: Efficient non-parametric image registration,” NeuroImage, vol. 45, pp. S61–S72, 2009. |
| [7] | A. Myronenko and X. Song, “Intensity-based image registration by minimizing residual complexity,” IEEE Transactions on Medical Imaging, vol. 29, pp. 1882–1891, 2010. |
| [8] | I. Arganda-Carreras, C. O. S. Sorzano, and R. M. et al., “Consistent and elastic registration of histological sections using vector-spline regularization,” in International Workshop on Computer Vision Approaches to Medical Image Analysis (CVAMIA), 2006, pp. 85–95. |
| [9] | S. Periaswamy and H. Farid, “Elastic registration in the presence of intensity variations,” IEEE Transactions on Medical Imaging, vol. 22, pp. 865–874, 2003. |
| [10] | C. Liu, J. Yuen, and A. Torralba, “SIFT flow: Dense correspondence across scenes and its applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 978–994, 2011. |
| [11] | S. Klein, M. Staring, K. Murphy, M. A. Viergever, and J. P. Pluim, “Elastix: a toolbox for intensity-based medical image registration,” IEEE Transactions on Medical Imaging, vol. 29, no. 1, pp. 196–205, 2010. |