Renal Cancer Cell Nuclei Detection from Cytological Images Using Convolutional Neural Network for Estimating Proliferation Rate

Md Shamim Hossain, Hamid A. Jalab, Fariha Zulfiqar, Mahfuza Pervin


The Cytological images play an essential role in monitoring the progress of cancer cell mutation. The proliferation rate of the cancer cell is the prerequisite for cancer treatment. It is hard to accurately identify the nucleus of the abnormal cell in a faster way as well as find the correct proliferation rate since it requires an in-depth manual examination, observation and cell counting, which are very tedious and time-consuming. The proposed method starts with segmentation to separate the background and object regions with K-means clustering. The small candidate regions, which contain cell region is detected based on the value of support vector machine automatically. The sets of cell regions are marked with selective search according to the local distance between the nucleus and cell boundary, whether they are overlapping or non-overlapping cell regions. After that, the selective segmented cell features are taken to learn the normal and abnormal cell nuclei separately from the regional convolutional neural network. Finally, the proliferation rate in the invasive cancer area is calculated based on the number of abnormal cells. A set of renal cancer cell cytological images is taken from the National Cancer Institute, USA and this data set is available for the research work. Quantitative evaluation of this method is performed by comparing its accuracy with the accuracy of the other state of the art cancer cell nuclei detection methods. Qualitative assessment is done based on human observation. The proposed method is able to detect renal cancer cell nuclei accurately and provide automatic proliferation rate.


Cell nucleus; Convolution neural network; Cytological images; Renal cancer;

Full Text:



Y. M. Alomari, S. N. H. S. Abdullah, R. R. M. Zin, and K. Omar, "Iterative randomized irregular circular algorithm for proliferation rate estimation in brain tumor Ki-67 histology images," Expert Systems with Applications, vol. 48, pp. 111-129, 2016.

S. I. Ferlay J, Ervik M, Dikshit R. (2015). Kidney cancer statistics. Available:

P. Weissleder, Nam JM, Thaxton CS, Mirkin CA. (2017). Earlier Detection and Diagnosis. Available:

L. J. Clague J, Cassidy A,. (2017). Early Detection, Diagnosis, and Staging. Available:

W. William, A. Ware, A. H. Basaza-Ejiri, and J. Obungoloch, "A review of Image Analysis and Machine Learning Techniques for Automated Cervical Cancer Screening from pap-smear images," Computer Methods and Programs in Biomedicine, 2018.

K. Zhang et al., "Multi-scale Colorectal Tumour Segmentation Using a Novel Coarse to Fine Strategy," in BMVC, 2016.

S. K. Mandal, "Performance Analysis Of Data Mining Algorithms For Breast Cancer Cell Detection Using Naïve Bayes, Logistic Regression and Decision Tree," International Journal Of Engineering And Computer Science, vol. 6, no. 2, 2017.

M. Veta, P. J. Van Diest, R. Kornegoor, A. Huisman, M. A. Viergever, and J. P. Pluim, "Automatic nuclei segmentation in H&E stained breast cancer histopathology images," PloS one, vol. 8, no. 7, p. e70221, 2013.

J. Vink, M. Van Leeuwen, C. Van Deurzen, and G. De Haan, "Efficient nucleus detector in histopathology images," Journal of microscopy, vol. 249, no. 2, pp. 124-135, 2013.

H. Sharma et al., "A Multi-resolution Approach for Combining Visual Information using Nuclei Segmentation and Classification in Histopathological Images," in VISAPP (3), 2015, pp. 37-46.

H. Chung, G. Lu, Z. Tian, D. Wang, Z. G. Chen, and B. Fei, "Superpixel-based spectral classification for the detection of head and neck cancer with hyperspectral imaging," in Medical Imaging 2016: Biomedical Applications in Molecular, Structural, and Functional Imaging, 2016, vol. 9788, p. 978813: International Society for Optics and Photonics.

X. Zheng, Y. Wang, G. Wang, and J. Liu, "Fast and robust segmentation of white blood cell images by self-supervised learning," Micron, vol. 107, pp. 55-71, 2018.

Y. M. Alomari, S. Abdullah, S. N. Huda, R. R. MdZin, and K. Omar, "Adaptive localization of focus point regions via random patch probabilistic density from whole-slide, Ki-67-stained brain tumor tissue," Computational and mathematical methods in medicine, vol. 2015, 2015.

Y. M. George, H. H. Zayed, M. I. Roushdy, and B. M. Elbagoury, "Remote computer-aided breast cancer detection and diagnosis system based on cytological images," IEEE Systems Journal, vol. 8, no. 3, pp. 949-964, 2014.

Y. Song et al., "A deep learning based framework for accurate segmentation of cervical cytoplasm and nuclei," in Engineering in Medicine and Biology Society (EMBC), 2014 36th annual international conference of the IEEE, 2014, pp. 2903-2906: IEEE.

Y. Song, L. Zhang, S. Chen, D. Ni, B. Lei, and T. Wang, "Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning," IEEE Transactions on Biomedical Engineering, vol. 62, no. 10, pp. 2421-2433, 2015.

R. Zhang, B. L. Osinski, T. J. Taxter, J. Perera, D. J. Lau, and A. A. Khan, "Adversarial deep learning for microsatellite instability prediction from histopathology slides."

Y. Song et al., "Accurate cervical cell segmentation from overlapping clumps in pap smear images," IEEE transactions on medical imaging, vol. 36, no. 1, pp. 288-300, 2017.

C. D. Malon and E. Cosatto, "Classification of mitotic figures with convolutional neural networks and seeded blob features," Journal of pathology informatics, vol. 4, 2013.

H. Wang et al., "Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection," in Medical Imaging 2014: Digital Pathology, 2014, vol. 9041, p. 90410B: International Society for Optics and Photonics.

H. Hu, Q. Guan, S. Chen, Z. Ji, and L. Yao, "Detection and Recognition for Life State of Cell Cancer Using Two-Stage Cascade CNNs," IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2017.

A. A. Cruz-Roa, J. E. A. Ovalle, A. Madabhushi, and F. A. G. Osorio, "A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection," in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2013, pp. 403-410: Springer.

J. Xu et al., "Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images," IEEE transactions on medical imaging, vol. 35, no. 1, pp. 119-130, 2016.

D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, "Mitosis detection in breast cancer histology images with deep neural networks," in International Conference on Medical Image Computing and Computer-assisted Intervention, 2013, pp. 411-418: Springer.

Y. Xie, X. Kong, F. Xing, F. Liu, H. Su, and L. Yang, "Deep voting: A robust approach toward nucleus localization in microscopy images," in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015, pp. 374-382: Springer.

Y. Xie, F. Xing, X. Shi, X. Kong, H. Su, and L. Yang, "Efficient and robust cell detection: A structured regression approach," Medical image analysis, vol. 44, pp. 245-254, 2018.

L. Ma et al., "Deep learning based classification for head and neck cancer detection with hyperspectral imaging in an animal model," in Medical Imaging 2017: Biomedical Applications in Molecular, Structural, and Functional Imaging, 2017, vol. 10137, p. 101372G: International Society for Optics and Photonics.

N. Coudray, A. L. Moreira, T. Sakellaropoulos, D. Fenyo, N. Razavian, and A. Tsirigos, "Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning," bioRxiv, p. 197574, 2017.

L. Zhang, L. Lu, I. Nogues, R. M. Summers, S. Liu, and J. Yao, "DeepPap: Deep convolutional networks for cervical cell classification," arXiv preprint arXiv:1801.08616, 2018.

V. Chandran, D. Kumar, P. Geetha, and R. Nidhya, "Deep Learning Neural Network with Semi supervised Segmentation for Predicting Retinal and Cancer Cell Diseased Images."

K. Sirinukunwattana, S. E. A. Raza, Y.-W. Tsang, D. R. Snead, I. A. Cree, and N. M. Rajpoot, "Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images," IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1196-1206, 2016.

F. Xing, Y. Xie, and L. Yang, "An automatic learning-based framework for robust nucleus segmentation," IEEE transactions on medical imaging, vol. 35, no. 2, pp. 550-566, 2016.

Y. Li, Y. Li, H. Kim, and S. Serikawa, "Active contour model-based segmentation algorithm for medical robots recognition," Multimedia Tools and Applications, vol. 77, no. 9, pp. 10485-10500, 2018.

P. L. Narayanan, S. E. A. Raza, A. Dodson, B. Gusterson, M. Dowsett, and Y. Yuan, "DeepSDCS: Dissecting cancer proliferation heterogeneity in Ki67 digital whole slide images," arXiv preprint arXiv:1806.10850, 2018.

M. Saha, C. Chakraborty, and D. Racoceanu, "Efficient deep learning model for mitosis detection using breast histopathology images," Computerized Medical Imaging and Graphics, vol. 64, pp. 29-40, 2018.

M. Khoshdeli and B. Parvin, "Feature-Based Representation Improves Color Decomposition and Nuclear Detection Using a Convolutional Neural Network," IEEE Transactions on Biomedical Engineering, vol. 65, no. 3, pp. 625-634, 2018.

O. Akin, Elnajjar, P., Heller. (Apr 27, 2017). The Cancer Imaging Archive Available:

R. K. Yip, P. K. Tam, and D. N. Leung, "Modification of Hough transform for circles and ellipses detection using a 2-dimensional array," Pattern recognition, vol. 25, no. 9, pp. 1007-1022, 1992.

A. Paul and D. P. Mukherjee, "Mitosis detection for invasive breast cancer grading in histopathological images," IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 4041-4054, 2015.

B. Korbar et al., "Deep learning for classification of colorectal polyps on whole-slide images," Journal of pathology informatics, vol. 8, 2017.


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

ISSN: 2180-1843

eISSN: 2289-8131