Internal quality evolution of a large test system – an industrial study

Open access


This paper presents our empirical observations related to the evolution of a large automated test system. The system observed is used in the industry as a test tool for complex telecommunication systems, itself consisting of more than one million lines of source code. This study evaluates how different changes during the development have changed the number of observed Code Smells in the test system. We have monitored the development of the test scripts and measured the code quality characteristics over a five years period.

The observations show that the introduction of continuous integration, the existence of tool support for quality improvements in itself, changing the development methodologies (from waterfall to agile), changing technical and line management structure and personnel caused no measurable change in the trends of the observed Code Smells. Internal quality improvements were achieved mainly by individuals intrinsic motivation. Our measurements show similarities with earlier results on software systems evolutions presented by Lehman.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] A. Chatzigeorgiou A. Manakos Investigating the evolution of bad smells in object-oriented code. Proc. 2010 Seventh International Conference on the Qual­ity of Information and Communications Technology QUATICTO pp. 106 115 Washington DC USA 2010. IEEE Computer Society. 219. 232 233 236

  • [2] M. E. Conway. How do committees invent? Datamation 145 (1968) 28-31. [accessed 26-Aug-2015]. ⇒234

  • [3] A. v. Deursen L. Moonen A. v. d. Bergh G. Kok Refactoring test code Proc. 2nd International Conference on Extreme Programming and Flexible Processes (XP2001) pp. 92-95. University of Cagliari 2001. ⇒220

  • [4] M. Fowler Refactoring: Improving the Design of Existing Code Addison-Wesley Longman Publishing Co. Inc. Boston MA USA 1999. ⇒218

  • [5] A. Israeli D. G. Feitelson The Unux kernel as a case study in software evolution J. Syst. Softw. 83 3 (2010) 485-501. ⇒220 235

  • [6] C. Izurieta J. Bieman The evolution of freebsd and linux Proc. 2006 ACM/IEEE International Symposium on Empirical Software Engineering ISESEm pp. 204-211 New York NY USA 2006. ACM. ⇒220 235

  • [7] K. Johari A. Kaur Effect of software evolution on software metrics: An open source case study SIGSOFT Softw. Eng. Notes 36 5 (2011) 1-8. ⇒220 235

  • [8] C. F. Kemerer S. Slaughter An empirical approach to studying software evolu­tion IEEE Trans. Softw. Eng. 25 4 (1999) 493-509. ⇒220 235

  • [9] F. Khomh M. Di Penta Y.-G. Gueheneuc An exploratory study of the impact of code smells on software change-proneness Proc. 16th Working Conference on Reverse Engineering WCRE’09 pp. 75-84 Washington DC USA 2009. IEEE Computer Society. ⇒218

  • [10] M. J. Lawrence An examination of evolution dynamics Proc. 6th International Conference on Software Engineering ICSE’82 pp. 188-196 Los Alamitos CA USA 1982. IEEE Computer Society Press. ⇒219 235

  • [11] M. M. Lehman The programming process 1969. IBM Research Report RC 2722. ⇒235

  • [12] M. M. Lehman Laws of software evolution revisited Proc. 5th European Work­shop on Software Process Technology EWSPT ’96 pp. 108-124. London UK UK 1996 Springer-Verlag. ⇒235

  • [13] M. M. Lehman Feast/2 final report grant number gr/m44101 2001. ⇒ 220 235

  • [14] M. M. Lehman J. F. Ramil Towards a theory of software evolution - and its practical impact (working paper) Proc. IntL Symposium on Principles of Softw. Evolution (invited talk). ISPSE 2000 1-2 Nov pp. 2 11. Press 2000. ⇒219

  • [15] M. M. Lehman J. F. Ramil Evolution in software and related areas Proc. 4th International Workshop on Principles of Software Evolution IWPSE’01 pages 1-16 New York NY USA 2001 ACM. ⇒219 235

  • [16] M. M. Lehman J. F. Ramil Rules and tools for software evolution planning and management Ann. Softw. Eng. 11 I (2001) 15-44.⇒219 233 234 235

  • [17] M. M. Lehman J. F. Ramil D. E. Perry On evidence supporting the feast hypothesis and the laws of software evolution Proc. 5th International Symposium on Software Metrics METRICS ’98 pp. 84 - Washington DC USA 1998. IEEE Computer Society. ⇒ 220 235

  • [18] I. Macia J. Garcia D. Popescu A. Garcia N. Medvidovic A. von Staa Are automatically-detected code anomalies relevant to architectural modularity?: An exploratory analysis of evolving systems Proc. Ilth Annual International Con­ference on Aspect-oriented Software Development AOSD T2. pp. 167 178. New York NY USA. 2012 ACM. ⇒218

  • [19] N. Moha Y.-G. Gueheneuc L. Duchien A.-F. Le Meur Decor: A method for the specification and detection of code and design smells IEEE Trans. Softw. Eng.36 I (2010) 20-36. ⇒218

  • [20] R. Moser P. Abrahamsson W. Pedrycz A. Sillitti G. Succi A case study on the impact of refactoring on quality and productivity in an agile team in Balancing Agility and Formalism in Software Engineering pp. 252 266 Springer-Verlag Berlin Heidelberg 2008. ⇒218

  • [21] H. Neukirchen M. Bisanz Utilising code smells to detect quality problems in ttcn-3 test suites Proc. 19th IFIP TC6/WG6.1 International Conference and 7th International Conference on Testing of Software and Communicating Sys­tems TestCom07/FATES07 pp. 228-243 Berlin. Heidelberg. 2007 Springer- Verlag. ⇒218

  • [22] S. Olbrich D. S. Cruzes V. Basili N. Zazworka The evolution and impact of code smells: A case study of two open source systems Proc. 2009 3rd International Symposium on Empirical Software Engineering and Measurement ESEM ’09 pp. 390 400 Washington DC USA 2009. IEEE Computer Society. ⇒218

  • [23] D. L. Parnas Software aging Proc. 16th International Conference on Software Engineering ICSE ’94 pp. 279-287 Los Alamitos CA USA 1994. IEEE Com­puter Society Press. ⇒ 219

  • [24] R. Peters A. Zaidman Evaluating the lifespan of code smells using software repository mining. Proc. 2012 16th European Conference on Software Mainte­nance and Reengineering CSMR’12 pp. 411-416 Washington DC USA 2012. IEEE Computer Society. 219 232 233 236

  • [25] D. I. K. Sjoberg A. Yamashita B. Anda A. Mockus T. Dyba Quantifying the effect of code smells on maintenance effort IEEE Trans. Softw. Eng. 39 8 (2013) 1144-1156. ⇒218

  • [26] K. Szabados Structural analysis of large ttcn-3 projects. Proc. 21st IFIP WG 6.1 International Conference on Testing of Software and Communication Systems and 9th International FATES Workshop TESTCOM ’09/FATES ’09 pp. 241 246 Berlin. Heidelberg 2009 Springer-Verlag. ⇒217 220 236

  • [27] K. Szabados Titanium 2015. [On­line; accessed 26-Aug-2015]. ⇒221

  • [28] K. Szabados A. Kovács Advanced ttcn-3 test suite validation with titan Proc. 9th Conference on Applied Informatics pp. 273-281 2014. ⇒217 236

  • [29] K. Szabados A. Kovács Test software quality issues and connections to interna­tional standards Acta Universitatis Sapientiae Informatica 5 I (2014) 77 102. ⇒217 220 236

  • [30] K. Szabados and A. Kovács Up-to-date list of code smells . 2015. [Online; accessed 26-Aug- 2015]. ⇒220224225

  • [31] C. Taube-Schock R. J. Walker I. H. Witten Can we avoid high coupling? Proc. 25th European Conference on Objectoriented Programming ECOOPTI pp. 204 228 Berlin Heidelberg 2011 Springer-Verlag. ⇒220

  • [32] W. M. Turski The reference model for smooth growth of software systems re­visited IEEE Trans. Softw. Eng. 28 8 (2002) 814-815. ⇒220 235

  • [33] E. Van Emden L. Moonen Java quality assurance by detecting code smells Proc. Ninth Working Conference on Reverse Engineering (WCRE’02) pp. 97 106 Washington DC USA 2002. IEEE Computer Society. ⇒218

  • [34] A. Yamashita L. Moonen Do code smells reflect important maintainability as­pects? Proc. 2012 IEEE International Conference on Software Maintenance ICSM T2 pp. 306-315.Washington DC USA 2012. IEEE Computer Society. ⇒219 236

  • [35] A. Yamashita L. Moonen Do developers care about code smells? an exploratory survey Proc. 20th Working Conference on Conference: Reverse Engineering pp. 242 251. IEEE Computer Society 2013. ⇒219 232 233 236

  • [36] A. Yamashita L. Moonen Exploring the impact of inter-smell relations on soft­ware maintainability: An empirical study Proc. 2013 International Conference on Software Engineering IC1SE *13 pp. 682 691 Piscataway NJ USA 2013. IEEE Computer Society Press. ⇒219 236

  • [37] A. Yamashita L. Moonen To what extent can maintenance problems be predicted by code smell detection? - an empirical study Inf. Softw. Technoi. 55 12 (2013) 2223-2242. ⇒218

  • [38] A. Zaidman B. Rompaey A. Deursen5 S. Demeyer Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining Empirical Softw. Engg. 16 3 (2011) 325-364. ⇒ 220

  • [39] B. ZeiC5 D. Vega5 I. Scheferdecker5 H. Neukirchen5 J. Grabowski5 Applying the ISO 9126 Quality Model to Test Specifications Exemplified for TTCN-3 Test Specifications Software Engineering 2007. Lecture Notes in Informatics. Copy­right Gesellschaft für Informatik5 Mar. 2007. ⇒ 220

Journal information
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 273 101 4
PDF Downloads 115 40 1