A Test Case Selection Framework and Technique: Weighted Average Scoring Method

Rafaqat Kazmi, Dayang N. A. Jawawi, Radziah Mohamad, Imran Ghani, Muhammad Younas

Abstract


Test case selection techniques identify and eliminate the modification revealing test cases and try to reduce the test suite size for optimization of regression testing. The objective of this experiment is to assess the effectiveness of weighted average scoring method of test case selection against single objective test case selection techniques. The multiobjective test case selection with the weighted average scoring framework and technique are proposed in this study to select the test cases. This method is trying to solve conflicting test case selection objectives with six selection scenarios. The method used test data of cost, coverage, fault detection ability and code change information, convert them into the weighted average score as scalar function and presented this score to 100-index slabs related to low to high scores, then select the test cases. The results for these selection scenarios are computed and evaluated using size reduction of the test suite, inclusiveness, and precision. The results showed that all scenarios performed acceptable level within conditions applied from 17% minimum to 86% maximum in size reduction metrics. The inclusiveness showed 17% to 88% and 33% to 85% for precision metric.

Keywords


Regression Testing; Test Case Selection; Test Suite Effectiveness;

Full Text:

PDF

References


J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection,” IEEE Transactions on Software Engineering, vol. SE-1, no. 2, pp. 156-173, 1975.

K. F. Fischer, “A test case selection method for the validation of software maintenance modifications,” in Proceedings of COMPSAC, 1977, pp. 421-426.

B. Beizer, Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, Inc., 1995.

M. Zavvar and F. Ramezani, “Measuring of software maintainability using adaptive fuzzy neural network,” International Journal of Modern Education & Computer Science, vol. 10, pp. 27-32, 2015.

M. A. Askarunisa, M. L. Shanmugapriya, and D. N. Ramaraj, “Cost and coverage metrics for measuring the effectiveness of test case prioritization techniques,” INFOCOMP Journal of Computer Science, vol. 9, no. 1, pp. 43-52, 2010.

S. Elbaum, P. Kallakuri, A. Malishevsky, G. Rothermel, and S. Kanduri, “Understanding the effects of changes on the cost‐ effectiveness of regression testing techniques,” Software Testing, Verification and Reliability, vol. 13, no. 2, pp. 65-83, 2003.

A. Orso, N. Shi, and M. J. Harrold, “Scaling regression testing to large software systems,” in ACM SIGSOFT Software Engineering Notes, 2004, pp. 241-251.

W. E. Lewis, Software Testing and Continuous Quality Improvement. CRC press, 2016.

W. E. Wong, J. R. Horgan, S. London, and H. A. Bellcore, “A study of effective regression testing in practice,” in ISSRE '97 Proceedings of the Eighth International Symposium on Software Reliability Engineering, 1997, pp. 264-274.

J. D. Musa, “Operational profiles in software-reliability engineering,” IEEE Software, vol. 10, no. 2, pp. 14-32, 1993.

C. Larman and V. R. Basili, “Iterative and incremental developments. a brief history,” Computer, vol. 36, no. 6, pp. 47-56, 2003.

A. Bertolino, “Software testing research: Achievements, challenges, dreams,” in Future of Software Engineering, 2007 (FOSE '07), 2007, pp. 85-103.

R. Kazmi, D. N. Jawawi, R. Mohamad, and I. Ghani, “Effective regression test case selection: a systematic literature review,” ACM Computing Surveys (CSUR), vol. 50, no.2, pp. 1-29, 2017.

R. Kazmi, I. Ghani, R. Mohamad, M. Tariq, I. S. Bajwa, and S. R. Jeong, “Trade-off between automated and manual testing: a production possibility curve cost model,” Int. J. Advance Soft Compu. Appl, vol. 8, no.1, pp. 12-27, 2016.

H. K. Leung and L. White, “A cost model to compare regression test strategies,” in Proceedings Conference on Software Maintenance, 1991, pp. 201-208.

T. Koju, S. Takada, and N. Doi, “Regression test selection based on intermediate code for virtual machines,” in Proceedings of International Conference on Software Maintenance (ICSM 2003), 2003, pp. 420-429.

G. Rothermel, M. J. Harrold, and J. Dedhia, “Regression test selection for C++ software,” Software Testing Verification and Reliability, vol. 10, pp. 77-109, 2000.

D. D. Nardo, N. Alshahwan, L. Briand, and Y. Labiche, “Coverage‐ based regression test case selection, minimization and prioritization: A case study on an industrial system,” Software Testing, Verification and Reliability, vol. 25, no. 4, pp. 371-396, 2015.

M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi, “Regression test selection for Java software,” in OOPSLA '01 Proceedings of the 16th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications 2001, pp. 312-326.

P. K. Chittimalli and M. J. Harrold, “Recomputing coverage information to assist regression testing,” IEEE Transactions on Software Engineering , vol. 35, no.4, pp. 452-469, 2009.

L. S. de Souza, R. B. Prudêncio, and F. d. A. Barros, “A Hybrid binary multi-objective particle swarm optimization with local search for test case selection,” in Intelligent Systems (BRACIS), 2014 Brazilian Conference, 2014, pp. 414-419.

A. A. L. de Oliveira, C. G. Camilo-Junior, and A. M. Vincenzi, “A coevolutionary algorithm to automatic test case selection and mutant in mutation testing,” in 2013 IEEE Congress on Evolutionary Computation (CEC), 2013, pp. 829-836.

E. D. Ekelund and E. Engström, “Efficient regression testing based on test history: An industrial evaluation,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2015, pp. 449-457.

A. Konak, D. W. Coit, and A. E. Smith, “Multi-objective optimization using genetic algorithms: A tutorial,” Reliability Engineering & System Safety, vol. 91, no. 9, pp. 992-1007, 2006.

S. Wang, S. Ali, and A. Gotlieb, “Minimizing test suites in software product lines using weight-based genetic algorithms,” in Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, 2013, pp. 1493-1500.

T. Hesterberg, “Weighted average importance sampling and defensive mixture distributions,” Technometrics, vol. 37, no. 2, pp. 185-194, 1995.

L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 435-445.

P. K. Sen, “Estimates of the regression coefficient based on Kendall's Tau,” Journal of the American Statistical Association, vol. 63, no. 324, pp. 1379-1389, 1968.

Nayuki, Project Nayuki. University of Toronto, 2017. Available at https://www.nayuki.io/

JodaTime. Joda.org., 2017. Available at http://www.joda.org/jodatime/

Java Editor. The Eclipse Foundation. Available at https://eclipse.org/

M. R. Hoffmann, “Java Code Coverage,” 2017.

PIT. (2016). Available at http://pitest.org/

M. B. Doar, JDiff-An HTML Report of API Differences. LGPL. Available at http://javadiff.sourceforge.net/


Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

ISSN: 2180-1843

eISSN: 2289-8131