What Makes Testing Work: Nine Case Studies of Software Development Teams Christopher D Thomson* Business School University of Hull Hull, UK c.thomson@hull.ac.uk Mike Holcombe, Anthony J H Simons Department of Computer Science University of Sheffield Sheffield, UK {m.holcombe,a.simons}@dcs.shef.ac.uk Abstract— Recently there has been a focus on test first and test driven development; several empirical studies have tried to assess the advantage that these methods give over testing after development. The results have been mixed. In this paper we investigate nine teams who tested during coding to examine the effect it had on the external quality of their code. Of the top three performing teams two used a documented testing strategy and the other an ad-hoc approach to testing. We conclude that their success appears to be related to a testing culture where the teams proactively test rather than carry out only what is required in a mechanical fashion. Testing; test first; test driven development; extreme programming;emprical; qualitative; testing culture. I. INTRODUCTION Extreme programming (XP) [1] presents what is, on the surface, a simple but effective testing practice known as test first, or test driven development. The idea is reassuringly simple, that unit tests should be defined and run before any implementation is present. Several studies have attempted to measure the effect of the test first practice on the quality of the software produced and time taken, but the results presented are inconclusive. The testing practice of XP encompassed more than simply test first. System testing is automated, incremental, regular, and early. User acceptance testing is similar although often less or not at all automated [1]. But these features can be used independently of test first and perhaps with some success. This raises our research question: How does the practice of testing effect a team following XP – or if test first is not followed then do the other practices of XP still influence the way testing is performed and the external quality? We collected data from nine teams, which we present as case studies. In the case studies the teams were novice users of XP, who we provided with training. They were given the option of using test first and whilst three teams expressed enthusiasm they ultimately did not follow the practice accurately. We found that the teams had a high degree of variation in external quality that cannot be easily explained by the teams’ testing practice alone or the practices of XP alone. Instead we find that testing must become part of the culture of the team, and can do so in at least two different ways. II. LITERATURE REVIEW Test first (TF) is an established development technique which is essentially the same as test driven development (TDD). In both cases the aim is to write tests before writing the functional code. This should in theory aid development as the tests form the basis of the specification, design and functional tests, whereas testing after the code is written is regarded as only a testing technique [1; 2]. Whilst TF and TDD are well defined, the traditional or test last technique is interpreted differently by the studies in the literature investigating this phenomenon. The studies’ definitions of test last can be divided into roughly four categories, which we will refer to as TL 1 to 4: TL-1: Unspecified traditional method. Most experiments provided at least a few hints as to the method that the non-TF teams should follow, however some did not [3-5]. The following statement was typical of the broad definitions that these papers used: "the control group which followed the traditional process" ([5], p132). As no further discussion was made about the process followed we can't make generalizations about what affected their performance. TL-2: Specified traditional method. This method is typified by manual or ad-hoc testing and was used in three studies [6-8]. The method was defined thus: "no automated tests were written and the project was developed in traditional mode with a large up-front design and manual testing after the software was implemented." ([7], p71) and "a conventional design-develop-test (similar to waterfall)” ([6], p339). Lastly this study also noted that whilst the teams had been asked to use unit tests, only one team wrote any [6]. TL-3: Unit tests written at the end of the development cycle. Two studies identified a method that used unit testing at the end of the development cycle [7; 9]. These studies were able to provide some metrics on coverage to define the amount of testing undertaken, but assumed that all tests were only written and used at the end of the project. One study *Author contributed whilst at the University of Sheffield. also noted that in this method there was late integration of the code [9]. TL-4: Unit tests written just after the code in an iterative development cycle. In this final group the test method was analogous to test first in all regards except that: "automated tests were written shortly after code in an iterative test-last fashion" ([7], p87). Of the five studies that used this definition, three had iteration periods the same as test first [7; 10; 11], and two had longer iteration periods [12; 13]. With these definitions in mind we can explore the results of the previous experiments, which at first glance appear contradictory. Two studies have targeted the design- enhancing claims of TF. One group of studies showed that the quality of the design produced as result of TF is better, in that the units are less complex and smaller than those developed using TL-2 and TL-3 methods [7]. However other recent empirical evidence suggests that TF in some cases degrades the quality of the design when compared to TL-4 [11], this was also apparent in one of the earlier individual studies which used TL-4 [7]. The early studies that compared TF to TL-1 in terms of performance were conducted by Müller in an academic situation [5]. He found that TF teams were no quicker but that there were marginal benefits to reliability and that functions were reused more frequently. The other studies in an academic setting had conflicting results. In the first TL-4 was compared with TF [10], where it was found that TDD was marginally less efficient, with marginally higher quality (not significant) [10]. The second study, again in an academic setting, altered the work cycle so that the TL-4 group constructed larger chunks of code before testing, than the TF group. The authors concluded that quality between the teams was the same, but that TF wrote more tests, and their productivity was higher but not significantly so [12]. Three TL-2 studies in academia found similar effects: productivity was increased (but this was not significant, perhaps due to a high degree of variability in the data [3]) and that quality was about the same [3; 7; 8]. In one study it was noted that the increase in productivity was due to an increase in testing time, with a decrease in coding time [3]. Another study identified that TF developers generated more tests [7]. In contrast a TL-1 study found that the TF team were 50% faster in development but this result was compromised by the fact that the TF group had more experience in software development [4]. To date, the results most favorable toward TF have come from industrial studies [6; 7; 9; 13; 14]. In the case studies the first found that the introduction of TDD gave a 40% reduction of defects with the same overall productivity against TL-3 [9]. The second case study found a reduction in defects of 62% in TDD compared to TL-1 [14]. A final study looked at several development projects and concluded that when using TF mature developers delivered less complex code, in smaller units compared to TL-2 and TL-3 [7]. In the first comparative study the TDD group passed 18% more tests than the TL-2 group but took 16% more time to do so [6]. In the second, no significant differences were found between TF and TL-4 and the researcher concluded that the time permitted (2 hours) was not enough for differences to emerge [7]. The final study found that the TF teams wrote more tests and ran them more frequently than TL-4, although this might have been influenced by the experimental procedure which required TL-4 teams only to generate tests towards the end of a story implementation [13]. The previous literature gives contrasting results for similar experiments, in summary the results for TL-1 show that TF is either as good as or better than TL-1, against TL-2 and TL-3 TF is better, and against TL-4 the results are more mixed. The studies using TL-4 suggest that the issue of iteration may be important. Three of the five studies that included a concept of TL-4 concluded that TL-4 maybe better that TF [7; 10; 11] the exception was based on a version of TL-4 where the TL practice was iterated at the end of a story, whereas TF was at the level of a sub-story [12; 13]. It could be that the effect of the testing frequency is more important than when the tests are written. Thus deep exploratory studies on testing methods are required in order to identify confounding factors [15]. The literature therefore supports the argument that the practice of testing takes many years for practitioners to develop; nonetheless, novice developers often deliver good quality software – although quality can be varied. This basic aptitude or naïve method can confound the results of studies that attempt to evaluate empirically the effectiveness of well founded methods. In this paper we will use a qualitative analysis to uncover the naïve methods used by three successful teams. To do this we observed the testing practices and their effect in nine case studies of student teams following the XP method. III. STUDY CONTEXT In order to gain some further insight into the effects of the test techniques TL1-4 as used by novices, data was collected about the testing process followed by nine teams. The teams selected their own members, each consisting of between three to five students [16]. The teams were composed of second or third year undergraduate students. The students had no previous instruction on XP but had experience developing software using a plan-driven approach, which was presented in nine hours of lectures and tutorials [19]. The development projects ran over twelve weeks, for fifteen hours a week. There were three industrial clients who each provided a project, two required database driven websites (B and C) and the other an e-learning environment (A). Each client represented a small business and came with a project brief, none had experience in commissioning custom software. Client A had the most expertise in computing, but this was derived from his degree several decades previously having since moved into a different area. The distribution of teams to projects follows the randomized complete block experimental design [17]; however in this study we treat the teams as a multiple case study [18]. The relationship between projects and teams is shown in Table I, along with the programming language used by the teams. TABLE I. PROJECTS AND TEAMS 1 2 3 4 5 6 7 8 9 Project C B B C C B A A A Language P P CP P P P J J J P – PHP, J – Java, CP – existing PHP program customized. We collected data weekly in order to examine the testing process over time. The students were instructed to submit their test logs, test code/document, time spent testing and program code having run the tests on a lab computer. For this project, the authors selected three popular unit test tools to be used by the teams: JUnit (Java); PHPUnit (PHP classes) and Selenium (PHP pages). IV. MEASUREMENT AND EVALUATION METHOD In order to establish how successful the tests were, we recorded the number of test files present for each team and their outcome if run. Each week the number of tests was calculated by counting the tests found in the teams’ working directories. To investigate the value of the testing process we calculated the ratio successful test to unsuccessful test outcomes to against the eventual external quality. To determine the outcome of the test we referred to the logs. We recorded each batch of tests submitted as a single test and if one or more tests failed then a “fail” was recorded. In many cases the tests submitted did not run because they were incorrectly configured, in this case we recorded that they “did not run”. It was not always clear if manual tests documented had been run, therefore we recorded if they were successful, if they were changed in the week, or if there was evidence that they had been run, otherwise they were recorded as “not run”. These tests were counted per test document which contained one or more tests. The test coverage was calculated by examining the tests present in the final week of the project. This measurement would reveal if coverage were related to successful projects. This was calculated in terms of class coverage by the tests provided. We define class coverage in a weakly as the number of classes (files in the case of poorly structured PHP) which had one or more test cases. We also identified two additional types of test (which we counted within the manual tests before) which were partially automated by the teams: Abbot tests to test Java user interfaces [23]; and a custom test harness which was produced by team seven. We used the client’s mark to judge the external quality [24] of the product produced. Unlike other methods of assessing quality via humans, such as inspection, only one person assesses the quality. As the population of clients for each software system is exactly one, we sampled the whole population. Thus whilst an agreement score cannot be computed in the traditional sense, we can be confident that the score is correct. In addition there is no evidence that other product measurements are correlated to client satisfaction. Another approach would be to collect post deployment metrics (bug reports and so on); however this would not be appropriate here, as three products were developed for each problem and only one of these was selected to be used in production. Lastly, whist users of the software could also be surveyed; this is prone to problems in relation to the requirements specification, where the users and client are not in agreement. So, to measure the success of a project, the only acceptable method is to interview the client. The measurement of external quality (50 points in total) was split equally between the categories: demo, manual, install guide, maintenance guide, ease of use, understandability, completeness, innovation, robustness and happiness. They are based on the products required as part of the course (first four items) and secondly on product quality metrics as discussed by Fenton [25]. A detailed description is available in the lab pack [20]. This process selected the three winning teams (3, 5 and 8). Each developer completed a self assessment at the end of the project. In this paper we consider the responses to the open questions shown in Table II. The complete instrument is in a lab pack [20]. This captured the developers’ individual performance, which the developers described in their own words. Lastly, to evaluate if other project management factors were more important for predicted quality, each member of the team also answered a survey based on the Shodan questionnaire which is used to evaluate XP adherence [21]. We used a shortened version, as questions on the metaphor and changes to release plans were not relevant for our teams; the full set of modified questions is available in a technical report [22]. To calculate a value of adherence for each team we took the mean value of the responses from the team members. V. DESCRIPTION OF THE CASES STUDIED The teams were rated by the clients both in summary (as summarized later in Table VI) and an overall grade (Table III). The winning teams 3 and 5 scored the highest, but despite winning for their client, team 8 scored slightly lower than these teams and one of the others (team 6). Our first observation on test technique was that no team wrote tests before they wrote their code and all used a variant of TL-4. Typically a team member would develop the tests on his own local copy of the system before bringing them into the lab to run and record the tests, but configuration issues at this point meant that many of the tests failed to run correctly – leading to the tests not being run (Fig. 1). For example team 1 mostly found that their tests did not run (although they attempted to run them most weeks) with only two cases where they failed. In contrast team 4 was more successful with the tests running and failing after week 6 and TABLE II. SELF ASSESSMENT SURVEY QUESTIONS (EXTRACT) In your own words please comment on your achievements. In your own words please comment on your failures. In your own words please comment on your role in the team. TABLE III. TABLE III: EXTERNAL QUALITY FOR EACH TEAM Team 1 2 3 4 5 6 7 8 9 Mark 29 33 39 29 38 35 28 34 34 being successful in week 10. Therefore when we examined the cases, we used these results to indicate when tests were being run by the team, rather than considering the outcome. Fig. 2 summarizes the data collected on when manual tests were run. Only four of the teams recorded manual tests (3, 4, 5 and 7). The overall testing strategy in terms of code class coverage for each of the teams is summarized in Fig. 3. This shows that only teams 4 and 7 mixed manual and automated tests with team 7 using their own custom test harness and Abbot. Teams 3 and 5 only used manual tests (although they had automated tests, these had insignificant low coverage). Teams 1, 2, 6 and 8 only ran automated tests and team 9 ran both automated tests and Abbot tests. Fig. 4 shows the time that the teams reported for testing. This mostly correlates with the amount of testing shown in Figs. 1 and 2, with the exception of the manual testing for team 6, where the team did not record any documented manual tests. We pre-tested all the developers to establish their basic level of ability before the start of the project. Table IV shows the mean pre-test scores for each team, each out of 100. A previous result on a larger data set [16] showed that these tests could account for around 27% of the variance in the mark for external quality as awarded by the client based. For this set of data there does not appear to be any direct relationship between the pre-test score and external quality. Thus ability does not appear to be strongly related to performance for these teams. Table V shows the responses from the Shodan questionnaire. The data clearly shows that the teams all interpreted and applied the XP practices differently. For example for pair programming team 1 scored 8/10 and teams 3, 5 and 7 scored 3/10. We consider the impact of this effect in the discussion. Figure 1. Frequency of automated testing. Figure 2. Frequency of formal manual testing. 0 1 2 3 4 5 6 7 8 4 5 6 7 9 10 11 6 4 5 6 6 7 8 9 10 5 7 5 6 7 9 11 5 6 5 6 5 1 2 3 4 5 6 7 8 9 Te st r u n s passed failed didn't run Week: Team: 0 2 4 6 8 10 12 6 7 11 4 5 6 7 11 4 5 6 9 12 4 5 6 7 8 9 10 11 12 3 4 5 7 Te st r u n s passed failed not run Week: Team: Figure 3. Class coverage at the end of the project. Figure 4. Time in hours spent testing. Lastly, the data collected from the developer surveys was summarized along with comments received from the client (Table VI). For easy reference each column is labeled by a letter and each row by the team number. Empty cells indicate that either the team or client respectively did not comment on that subject. In some cases the client or team was vague in their descriptions of the problems encountered and this is reflected in evidence presented in Table VI. A brief analysis of the table indicates that the teams were highly variable in their approaches. In the discussion that follows we use the descriptions in Table VI to explore the interpretation of the other data. VI. DISCUSSION A. Overview Initial investigations into the quantitative data presented Figs. 1-4 and Tables 3-5 found no significant correlations between external quality and the various measurements. Furthermore the qualitative data in Table VI also gives no clear indication as to why certain teams produced the best systems. Therefore a more detailed analysis was required to compare the teams. In terms of success, the assessment of the teams’ products by the client was conducted using the external quality measurement (Table III) and comments (Table V). The clients’ comments highlighted two things that varied between the teams: the completeness of the functionality included and the robustness of the system developed (Table VI: C and D). B. Factors that Led to Success and Failure Completeness seems to have been a particular issue for three of the teams. In two cases there were instances of members not being fully involved (Teams 2 and 9, Table VI: C) and in team 2 the willingness of one member take on the majority of the coding and thus being overloaded (Table VI: D). Team 4 was reported both by the team (Table VI: D) and the client as being disorganized (Table VI: E). Robustness was a factor for two of the other teams (Table VI: F): In team 1 this may have been due to the time wasted on automated tests that never functioned as intended (Table VI: A-C), and for team 6 the problems with the tests not keeping pace with the code evolution (Table VI: A). Lastly team 7 was the only team where the client had serious difficulties understanding and installing the software (Table VI: F). In the end he was 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 6 7 8 9 Team Manual tests Automated tests Custom tests Abbot 0 10 20 30 40 50 60 70 80 90 1 2 3 4 5 6 7 8 9 Team Unit Testing Testing TABLE IV. PRETEST SCORES FOR EACH TEAM [16] Team 1 2 3 4 5 6 7 8 9 Pre Test 2 66 64 72 74 65 71 69 58 76 Pre Test 3 67 58 62 77 72 69 53 65 69 TABLE V. SHODAN SCORES FOR EACH TEAM Team 1 2 3 4 5 6 7 8 9 A Automated unit tests 4 4 0 2 1 4 2 3 2 B Customer acceptance tests 2 2 0 1 1 3 4 0 3 C Test-first design 5 4 0 2 1 3 0 0 3 D Pair programming 8 6 3 5 3 5 3 7 4 E Refactoring 3 5 2 4 3 5 4 4 2 F Release planning/planning game 4 4 5 3 4 5 2 4 2 G Short releases 5 7 4 1 5 3 3 3 5 H Stand-up meeting 1 6 0 8 3 6 0 1 2 I Continuous integration 7 5 2 6 4 6 5 9 4 J Coding standards 9 6 7 8 4 6 8 9 4 K Collective code ownership 7 6 5 8 6 9 6 9 5 L Sustainable pace 7 6 6 5 8 7 7 5 7 M Simple design 6 6 5 6 6 7 8 6 7 forced to mark their system based on a demonstration that the team gave him, with little time for him to explore it by himself. In summary most of the non-winning teams failed in a single area, most likely as an oversight. This leaves us with the winning teams. Teams 3 and 5 produced extensive manual tests in the second half of the project (Fig. 2), although team 5 left the writing of most until the last week of development (Fig. 3). The comments from the members of team 5 suggest that they did ad-hoc testing prior to this as the whole team highlighted their lack of documented tests (Table VI: B). Team 8 took a different approach and, based on their comments (Table VI: A), were more inclined to deliver functionality. By inspecting team 8’s code we found that they developed it in a highly iterative way. When we inspected team 8's directory and we discovered that there were working areas for each of the TABLE VI. TABLE VI: QUALITATIVE OVERVIEW OF THE TEAMS A B C D E F Team Automated testing Manual testing Team involvement in testing Team Communication Client impression Client assessment 1 Errors in the test script were difficult to resolve: “Our major failure was [the lack of] a runable test script”. After the first two weeks they abandoned the automated methods as they felt they were falling behind. Everyone, around 60 hours total. Frequent contact using a variety of online tools. Frequent meetings and pair programming. Slow to start but then worked well and creative. Functionally comprehensive product but some parts of the system failed. 2 The tester “found testing first with PHP time consuming and challenging”. Mostly ad-hoc as the code was written. One member was did most of the coding and testing. One member often did not turn up for meetings. They relied on MSN for communication. Not very innovative. Brief documentation, some missing functionality, robust. 3 Hard to write as they modified an open source product. This interfered with the tests. Detailed manual tests in the second half of the project. Several members. Used multiple lines of communication and met frequently. Documentation was clear, but was complex to use. Mostly functionally complete and robust. 4 “The selenium testing took a long time to complete successfully”. One member who was also responsible for the website. Ad-hoc meetings where "jobs were never really assigned so we didn't know who was doing what" Laid back and lacked urgency. Had some good ideas. Not all functionality was delivered and the documentation was not detailed. 5 Did not use PHP classes so could not use PHPUnit. "We never really did enough documented testing". Members worked on tests for their own code. Worked together in the lab but did not pair program or integrate their code frequently. Started slowly. Some features did not work as expected, but more robust than team 1. Provided features were comprehensive and creative. 6 Greatest number of tests. But these were reported to be difficult to maintain as the project evolved. Manual tests were used but undocumented. One member who spent most of his time testing. Shared their code regularly, but did not integrate it. Documentation was comprehensive but not detailed enough. Was not functionally complete and some functions reported errors. 7 Did not understand how to run or write unit tests. Used a custom test harness. Used both manual and Abbot tests. Communicated mainly by email with few meetings. Did not install and run properly and documentation was too complex. 8 11 unit tests: spending time on testing "could jeopardize the completion of the project." Very little testing recorded. Two members. The team frequently integrated their code (up to 3 times a week). Communicated frequently to set tasks and check progress. Clear documentation, with some unexpected but not erroneous results. 9 JUnit was used. Focused on ad-hoc testing, Abbot found to be unreliable. Two members who did most of the work. Described their team as having poor team-working skills. Documentation was brief but clear. Was not easy to use, and some options were hidden or disabled. Some features difficult to understand. members as well as a combined solution. The combined solution was copied up to three times a week as the individual elements were integrated; this showed the regular integration of the system. The ad-hoc testing associated with the iterative development appears to have ensured that they kept the project working. The client’s comment about unexpected but not erroneous results adds further weight to this argument (Table VI: F), as ad-hoc testing may be less likely to identify such problems if everything seems to be working fine. Given that both iteration and testing could be important factors we can reassess the remaining teams concentrating on these factors. C. The Effect of the XP Process The Shodan questionnaire (Table V) measured compliance to XP practices. Team 8 ranked highest on several questions (coding standards, shared code ownership, continuous integration and pair programming). This fits well with the other observations of team 8's method; however other teams (notably team 1) have similar patterns of process on the same questions but did not do so well overall. Furthermore it is surprising, given the iterative nature of the programming style that team 8 adopted, that they should rank so low on the short releases measurement. This indicates that their success was not dependent on client feedback (as they did not give releases to the client) but on internal feedback mechanisms within the team. Lastly the other two winning teams (3 and 5) did not rank highly on these questions with the exception of team 3 which was ranked fourth on the short releases question only. It is important to note that in the responses to each of the Shodan questions only one of the winning teams (3, 5 or 8) was present in the top four, indeed in several cases a winning team was also at the bottom of the scale. Thus the practices that were related to the questions were not universally important to success. We also investigated if it was the combination of factors that was important. As previously noted, despite ranking highly on similar questions, teams 8 and 1 had different outcomes. The main failure of team 1 as assessed by the client was that several parts of the system did not work (Table VI: F). The Shodan survey showed two differences between these teams: team 1 was ranked fourth for collective code ownership whereas 8 was ranked top (Table V) and team 1 attempted more unit testing but with similar coverage to team 8 (Table V, Fig. 3). Therefore we can speculate that a team using a highly iterative process, sharing its code and integrating this in regular builds is more likely to achieve greater external quality than one which does not share its code as frequently and relies on automated testing with low coverage. D. Success Without Documented Testing The conditions for success, as achieved by team 8, can be further refined by examining the problems the other teams encountered. For example, in team 2 only one member focused on writing the code for the system (Table VI: C) making activities like pair programming less advantageous (as there is no need to disseminate information and the pair observer may have had very little interest in the code being written) (Table V). In team 6 one person focused on testing (Table VI: C) and as a result found it hard to keep pace with rapidly evolving code (Table VI: A) as it was refactored (Table V – ranked top for refactoring). Thus we can broaden the definition of the advantageous method used by team 8 to include the observation that regular integration (by inspecting their code) that was led by two members (Table VI: C) led to ad-hoc code review and testing (Table VI: D). This was more effective than the other methods because of its regularity. We might then speculate that this led to team 8 being acutely aware of the tests required without the documentation process which teams 3 and 5 used. Furthermore as teams 3 and 5 did not benefit from the iterative approach we might further speculate that these teams required the documented testing method to ensure quality. E. Success Using Documented Testing In terms of manual tests, teams 3 and 5 achieved class coverage of 40% spending 20-30 hours testing mostly in the second half of the development period (Figs. 1, 2, 3 and 4). Two other teams recorded notable amounts of documented manual testing (4 and 7) and team 6 spent around 30 hours manually testing without documented records (Figs. 2 and 4). Team 4 regularly ran their tests from early on in their project (Figs. 1 and 2), but the tests only had low class coverage (Fig. 3), but despite this the client did not address the issue of robustness of the code, commenting instead on missing features and lack of urgency towards the end (Table VI: E and F). Team 7 achieved 30% class coverage for their tests (Fig. 3) but their product was too complex for the client to understand (Table VI: F). Team 6 did not leave evidence of manual tests but recorded time doing them so these must have been ad-hoc (Fig. 4). As with the unit testing for team 6 it seems these tests did not keep up with the final evolution of the code (Table VI: A), allowing the faults into the final version (Table VI: F). Thus to be successful, the tests must be accurate and up to date. For two of these teams, this seemed to be an issue but for the other team, although their code was robust, it was not complete. Thus we speculate that for testing to be effective the tests should be reviewed to ensure they are current and that this cannot be carried out by a single member effectively. VII. CONCLUSIONS A. Summary The most striking thing about the cases presented is the differences between the teams. All the teams were following much the same development technique with some variability and the evidence supports this. The variability in the process was noticeable in the testing method used (Figs. 1 to 4) and the way the team approached the development task with regards to the practices of XP (Figs. 7 and 8). Analysis of the cases suggested that none of the practices of XP, or testing alone, or XP and testing in conjunction could by themselves guarantee a high level of external quality. The winning teams all carried out testing that could identify errors, some documented and some ad-hoc. To achieve this it was apparent that a review process was required to ensure tests developed kept pace with evolution in the code. This could be achieved by documented tests, or ad-hoc tests that the team was aware of, due to their regular repetition. To achieve an effective review process more than one team member was needed to be involved in testing, so that the changes to the code could be identified. B. Testing Culture Teams 3 and 5 knew what to test because they had a document test plan; team 8 knew what to test because they had a culture of repeating a familiar set of ad-hoc tests. In both cases a 'testing culture' was present and the quality that client required was delivered. The 'testing culture' for the winning teams has the characteristics listed in Fig. 5. However for teams to win, well tested code was not enough. They also needed to completely satisfy the requirements of the client. The teams which did not wholly satisfy the requirements were disorganized in some way often due to members not fully taking part in the development process. C. Recommendations In summary, for teams to achieve the highest levels of external quality, as judged by a client, the team needs to have a testing culture and be organized enough to deliver the client’s requirements. Both test-first and a highly iterative test last method met the requirements for a testing culture which explains the previous positive results for these methods in the literature. A less documented or less iterative test last method is perhaps less likely to meet the requirements for a strong testing culture, leading to the negative results found in comparison. Based on the analysis of data collected, it is recommended that a testing culture is fostered in development teams, particularly if their members are novices. What counts is not so much the chosen testing method, but proactive involvement in testing, either through frequent ad-hoc testing or documented testing that increases towards the end of the project. In both cases the successfully collaborating team should contain at least two testers who are also involved with at least two testers who are also involved in the programming task. D. Validity We should consider the weight of the evidence in the light of the theoretical and literal replications of cases presented [18]. Theoretical replications represent those where the context or method is significantly different, thus we expect theoretical replications to have different outcomes. Literal replications are where the context and method are similar, and similar results are expected. In terms of theoretical replication no test-first cases were available although a variety of TF-cases in the TL-4 group were available. Unlike case studies found in the literature unit, manual and ad-hoc testing techniques were also investigated. Given this partitioning we had teams in each of the three replications thus we had some literal replication, although some teams fell into multiple partitions. However, the instruction given to the teams to help them use unit tests was clearly not sufficient, given the number of tests that failed on technical grounds. Thus the relationship between the cases studied and unit testing is not theoretically replicated. Given the results of the analysis, the theoretical replications as partitioned by testing method was not enough to explain the differences and some consideration to testing culture must be given instead. This suggests that further theoretical replications should be planned to investigate the effects of strong testing culture and unit testing, since there were no cases studied thus far with both. E. Future Work The emergence of a testing culture with novice developers during highly iterative development and regular integration is an alluring concept. In general, software developers are not keen on testing. Extreme programming and test-first have gone some way to address this, but writing unit tests, even after the code exists, is still challenging, as can be seen from the case studies. If an iterative method with the right features can lead to the same outcome then this is desirable if the effect can be replicated and controlled. Therefore we recommend that future research looks more closely at the effect of iteration and how developers cope with testing during integration. Ideally a future research project should observe, in detail, a development team that plans to use an iterative approach with frequent integration and ad-hoc testing. Given the difficulty of analyzing the quantitative data in a meaningful way we recommend the use of ethnography to discover the significant features of this process which are likely to be in the detail of the observations. ACKNOWLEDGMENT We thank the reviewers for their invaluable help in improving the presentation of this paper. EPSRC Grant awarded: £500K over three years (March 2006-2009) to carry out research in the Observatory. EP/D031516 - the Sheffield Software Engineering Observatory. A balance between testing and programming. o Where testers are part of the programming team. More than one person responsible for testing. Tests kept current. Either: o Documented testing, increasing towards the end of the project, or o Ad-hoc throughout the project combined with regular integration of code. Figure 5. The characteristics of a testing culture. REFERENCES [1] K. Beck, and C. Andres, Extreme Programming Explained: Embrace Change, 2004. [2] B. Vodde, and L. Koskela, “Learning Test-Driven Development by Counting Lines,” Software, IEEE, vol. 24, no. 3, 2007, pp. 74-79. [3] L. Huang, “Analysis and Quantification of Test-first programming,” master's thesis, Dept. Computer Science, University of Sheffield, 2007. [4] R. Kaufmann, and D. Janzen, “Implications of test-driven development: a pilot study,” Proc. 18th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2003), New York, ACM, 2003, pp. 298- 299. [5] M. Müller, and O. Hagner, “Experiment about test-first programming,” Software, IEE Proceedings, vol. 149, no. 5, 2002, pp. 131-136. [6] B. George, and L. Williams, “A structured experiment of test-driven development,” Information and Software Technology, vol. 46, no. 5, 2004, pp. 337-342. [7] D. Janzen, “An Empirical Evaluation of the Impact of Test-Driven Development on Software Quality,” doctoral thesis, University of Kansas, 2006. [8] A. Gupta, and P. Jalote, “An Experimental Evaluation of the Effectiveness and Efficiency of the Test Driven Development,” Proc. First International Symposium on Emprical Software Engineering and Measurement, IEEE, 2007, pp. 285-294. [9] E. M. Maximilien, and L. Williams, “Assessing test-driven development at IBM,” Proc. 25th International Conference on Software Engineering, 2003, pp. 564-569. [10] M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar, “Towards empirical evaluation of test-driven development in a university environment,” Proc. EUROCON 2003. Computer as a Tool. The IEEE Region 8, IEEE, 2003, pp. 83-86, vol. 2. [11] M. Siniaalto, and P. Abrahamsson, “Does test-driven development improve program code? Alarming results from a comparative case study,” Proc. Second IFIP TC 2 Central and East European Conference on Software Engineering Techniques, (CEE-SET 2007), Poznan, Poland, Springer-Verlag, 2007, pp. 143-156. [12] H. Erdogmus, M. Morisio, and M. Torchiano, “On the effectiveness of the test-first approach to programming,” Software Engineering, IEEE Transactions on, vol. 31, no. 3, 2005, pp. 226-237. [13] A. Geras, M. Smith, and J. Miller, “A prototype empirical evaluation of test driven development,” Proc. 10th International Symposium on Software Metrics, 2004, pp. 405-416. [14] L. Crispin, “Driving Software Quality: How Test-Driven Development Impacts Software Quality,” Software, IEEE, vol. 23, no. 6, 2006, pp. 70-71. [15] D. Janzen, and H. Saiedian, “Test-driven development concepts, taxonomy, and future direction,” Computer, vol. 38, no. 9, 2005, pp. 43-50. [16] C. D. Thomson, and M. Holcombe, The Sheffield Software Engineering Observatory Archive: Six Years of Empirical Data Collected from 74 Complete Projects, tech. report CS-09-01, Dept. Computer Science, Univ. of Sheffield, 2009. [17] B. Ostle, and L. C. Malone, Statistics in Research: Basic Concepts and Techniques for Research Workers, Iowa State Press, 1987. [18] R. Yin, Case study research: Design and methods, Sage, 2008. [19] M. Holcombe, Running an Agile Software Development Project, Wiley, 2008. [20] C. Thomson, and M. Holcombe, Software Hut Lab Pack, tech. report CS-09-03, Dept. Computer Science, University of Sheffield, 2009. [21] L. Williams, L. Layman, and W. Krebs, Extreme Programming Evaluation Framework for Object-Oriented Languages--Version 1.4, tech. report TR-2004-18, Department of Computer Science, North Carolina State University, 2004. [22] J. Karn, C. D. Thomson, S. J. Wood and G. Michaelides, The Shodan Adherence Survey for Extreme Programming: Sheffield Revision 1, tech. report CS-09-05, Dept. Computer Science, University of Sheffield, 2009. [23] T. Wall, “Abbot framework for automated testing of Java GUI components and programs,” 2002-2008; http://abbot.sourceforge.net/doc/overview.shtml. [24] F. J. Macias, “Empirical assessment of extreme programming,” doctoral thesis, Dept. Computer Science, University of Sheffield, 2005. [25] N. Fenton, and S. Pfleeger, Software Metrics: A Rigorous and Practical Approach, PWS Publishing Co, Boston, 1997.