Abstract: Accurate alignment of multi-session medical imaging is essential to the analysis of disease progression. By comparing the magnetic resonance imaging (MRI) data captured before and after a course of neoadjuvant chemoradiation (nCRT) treatment, physicians are able to evaluate the tumor response for further treatment of the disease. However, rectal MRI data captured in multi-session are often misaligned and not guaranteed to have one-to-one correspondence, making it challenging for physicians to observe the treatment response of tumor. To address this issue, we propose an unsupervised learning based volume registration framework, which enables accurate alignment even under a high degree of deformation between multi-session rectal data. Moreover, it works without the assumption of one-to-one correspondence between multi-session data, and hence is a general solution to rectal MRI volume registration. The experimental results show that the proposed registration framework accurately aligns rectal cancer images and outperforms other state-of-the-art methods in medical image registration. By providing accurate registration, it can potentially increase the efficiency and reduce the workload for physicians to evaluate the rectal tumor response to nCRT.