I recently submitted the below text as a research paper for a homework assignment in my Software Testing and Test Driven Development class. The assignment called on me to react to a case study and submit my response in the form of a proposal to a manager.
Here is the situation I was tasked with responding to:
Case Study on Testing Processes and Software Development Lifecycle
You are a new manager of a team consisting of development and testing engineers at a company you just joined. Your team has been assigned to work on a high-profile IT project that is due in 2.5 months. When you meet with the business analysts for the project, they inform you that the requirements they received for the project are high-level and incomplete and they are spending more time than originally planned to fully document all the detailed requirements for the project and create functional specifications. Business analysts are in constant meetings with business stakeholders to clarify and detail the project requirements.
When you meet with the development and testing engineers for the project, they inform you they cannot really start any work without the completed functional specifications from the business analysts and that the project timeline of 2.5 months may be achievable if all functional specifications would be ready right now, but without functional specifications they cannot tell you when the project realistically can be delivered. You try to get a high-level timeline estimate from development and testing teams and end up with a 4 month delivery timeframe, instead of the original 2.5 months.
After further review of the software development lifecycle, you realize that the company is using the traditional waterfall software development lifecycle approach. Additionally, the software you are working on is considered to be “mission-critical”, which means it needs to meet high availability and reliability standards and go through a very rigorous testing process so that the rate of defects is very low once the software is in production. Your boss reminds you that a failure of this project will put the company on the front pages of the news and you certainly do not want that as your first experience leading a project at this new company.
When you discuss the current testing process with your test lead, you find out that the current testing approach consists of test engineers using functional specifications as the starting point for test strategy and test case creation. Test engineers work on test strategy and test case creation while the development team is working on design, implementation, and unit testing. When code is delivered to a testing environment, test engineers perform manual test execution, document results, and create defects for any issues found. Defects are tracked in a defect tracking system, reviewed by business analysts, assigned to developers for fixing and then retested by testers manually. Test automation is currently limited to selected test cases used for regression testing. The testing lead seems very concerned about having sufficient time to do thorough testing on the project, especially considering the high-profile of the project and business consequences of missed defects.
And my response is below:
As a new manager of software engineering and test engineering teams at ABC Software, LLC, is a current project that needs immediate attention. There are many challenges facing this project:
- Our customer has yet to provide a fully functional scope of work for the software project we are currently developing yet expect it to be delivered in 2.5 months
- Our software engineering team cannot complete the work in the 2.5 month time frame but state that they can likely finish it in 4 months if the functional specification was delivered to them immediately
- Our development model is the traditional waterfall method for a client that is facing changing needs
- The software we deliver needs to meet high-availability and high-reliability quality standards and any failures of the product will gain national attention
- Our test engineers are currently limited to selected test cases for regression testing and they are concerned about having enough time to test the software adequately to meet the projects stated availability and reliability standards
With these issues in mind, phasing in an agile development model for our business analysts, project managers, software engineers, and test engineers will be a key part of our strategy to deliver software that is built with quality in mind from the start. Migrating to an agile strategy is part of a three-prong strategy to effectively manage this project and make our technical and business staff able to deliver key objectives to customers in the future:
- Ensure our upper-management is engaged with the client so they understand we cannot deliver software without a signed scope of work and engage with the client’s upper-management so they are aware we will be delivering subsets of functionality in 2-4 week iterations in the future.
- Engage in activities to help our business analysts, project managers, software engineers, and test engineers why we are switching to an agile model for software development. This means making sure they are involved in the changes and understand what is expected of them.
- Build quality into the software by having defined processes for software testing that we do not deviate from. All team members need to understand that it is their responsibility to deliver quality software, not just to do their sole task and hand it off to another individual/team.
Software Lifecycle Model
Transitioning from our traditional waterfall model to an agile model will help in delivering discreet sections of functionality to our customer while meeting our internal software quality and delivery goals. The waterfall method works for large projects where change is difficult and expensive.
The delivery and continuous integration needed this project and our future projects can better be served by utilizing an approach that can be more reactive to our customer’s changing needs. Our business analysts and project managers can work with the client to develop break down the high level functional specification that we have from then into deliverables for each 2-4 week iteration. This will give our software engineers and test engineers the information needed to perform verification and validation testing which is needed to provide the high-availability and reliability standards that our company is known for.
Feasibility of a New Lifecycle Model
Changing the current waterfall development model to agile may not be an easy transition. Changing to an agile model will be a challenge but ensuring the global development team, including business analysts and project managers, understands the change and feels empowered throughout the process will help deliver quality software while meeting the business’ goals.
Phasing in agile methodology with our current waterfall model will make the transition easier for all team members involved. This means transitioning to delivering functionality in 2-4 week iterations but keeping certain workflows that are familiar to the team members such as using the V-Model to plan the iterations and test phases.
Lessons learned meetings after each iteration that involve team members from each of the functional areas (business analysts, project managers, software engineers, test engineers, and managers) will facilitate knowledge transfer and help improve our processes for the next iteration. Constant improvement is the goal of transitioning to the agile model.
Challenges of Introducing a New Lifecycle Model
Introducing a new development lifecycle model will be a challenge, but one our team is ready for and one that will prepare us for the future as we grow as an organization. Getting the team’s mindset on-board will be one of the biggest ways that we can ensure a smooth transition because it will help the team understand how each and every member’s role is about delivering quality software. Some of the long-term challenges of agile adoption include the need for discipline, constant reinforcement of principles and goals, and the fact the people change at different rates (Abdelnour-Nocera & Sharp, 2012).
Any distractions that arise can be eased by listening to the team’s concerns and addressing them during the change or letting the specific team member know how their specific concern will be addressed so they feel empowered instead of silenced.
Training will also be provided in the development methodology as well as the specific tools that will be used during the project. Our workforce should be constantly learning and able to understand what is being asked of them. This will help us retain our current talent as well as ensure our newer hires can be valuable members of our team as they grow professionally.
As a method to facilitate the long-term strategy shift to an agile model we will train key members of the business analyst, project management, software engineering, and test engineering teams to be agile evangelists. These team members will employ a strategy of ‘embedded coaching’ (Abdelnour-Nocera & Sharp, 2012) where they teach the other members of their team the agile strategies we are putting in place in order to implement a cultural change toward agile methodologies (Abdelnour-Nocera & Sharp, 2012).
Another challenge is making sure that the software engineers and test engineers understand that out software quality standard is fixed, but the scope of each iteration is not (Samanta & Mani, 2015, pp. 135-139). This means that quality cannot be compromised. The scope of the functionality that needs to be completed within each iteration can be scaled back to provide more time for testing but code that does not meet our quality standards will not be integrated until it passes all required tests in the test execution phase of testing.
Ensuring that the middle managers are on board with the change to an agile model is key as well (Abdelnour-Nocera & Sharp, 2012). There is always the possibility that they do not share the vision (or additional work load creating stories for defects/changes) of the development team and such will blame any failures or delays on agile itself. One way to mitigate this is to bring the managers of the business analysts and project managers early in the decision-making process. Ensuring that the switch to agile is a top priority and is represented in C-level management’s project portfolio will also assist in applying the needed pressure to be able to shift towards building and releasing higher quality functionality to customers faster.
Software Engineering and Testing Best Practices
Changing to an agile development methodology will assist in delivering discreet sections of functionality to the customer. Ensuring that the test team members are empowered and treated as equals to their feature developer counterparts will help build quality into the code before the verification step.
Having a valid specification for the upcoming iterations will let the development team know what they are building and will help keep deadlines in mind when scoping out the next iteration. Having cross functional teams attend iteration planning meetings will help the team members understand the challenges that may be faced in the coming iterations. This means having experienced team members as well as newer members in the meetings so they can learn the entire process and gain the experience so the new members can become experienced members faster.
Our testing team needs to ensure unit tests cover each class and they are run automatically as code is checked in. The test engineers also need to configure our source code repository so that when code is checked into the repository there are meaningful comments so other developers do not need to read the code to understand what the checked in changes do.
We can ensure that all database scripts, configurations, external libraries, automated tests, and all related code is in the repository so replicating issues is not a large task. The test engineers and management team need to make sure that all automated tests pass before releasing code to production. Our test engineers also need to work closely with the SWE and BA for the verification testing.
Our test engineers and software engineers will use the appropriate Unit Testing Framework, such as PHPUnit, to run unit tests of the software to ensure as complete code coverage via tests as possible. The software engineers will write their own unit tests to cover discreet functions and methods from their code. The test engineers will test higher level frameworks and classes to validate their functions, ensure that their interfaces work as expected, and perform regression testing.
Depending on the operating system in place, we will use tools like Puppet for operating system configuration management. Using Puppet will allow us to have a variety of servers for testing, development, staging, and production and ensure they are all configured appropriately. Other tools such as Vagrant will help individual software engineers spin up a new virtual machine with the base operating system and automatically run the Puppet configuration script so the developers do not need to worry about making changes or writing bad code on the virtual machine. With one command, the developer should be able to get back to a baseline system and deploy their code to it.
Our test engineers will use front-end automated testing tools like Selenium for automated user interface testing. This will be useful for testing how large amounts of the codebase interact with each other from a user’s perspective.
During the iteration planning process, we will need to focus on 6 main testing phases (Bartlett 2016):
- Requirements Analysis
- Test Planning
- Test Case Development
- System Environment
- Test Execution
- Test Reporting
The requirements analysis phase is where the test engineers, software engineers, project managers, and business analysts meet to discuss what is testable in the iteration after reviewing the scope of work and functional specifications.
The test planning phase will involve the test engineers, software engineers, and the manager of development operations. This is when the test plan is developed and each team member understands what they are responsible for testing. The manager of development operations is involved due to their experience with testing software systems so they can provide any needed insights to the team to ensure that all of the needed functionality is in the test plan.
The test case development phase details how to test the application and involves team members from the test engineers and the software engineers.
The test cases should guide each tester through the testing process including any data or external interfaces (stubs or mocks) needed to adequately test a particular section of code or functionality. Employing a higher number of tests and test cases has been correlated with a higher baseline quality and productivity for developers and test engineers (Fucci & Turham, 2014), so ensuring an adequate number of tests covers the codebase is key. The knowledge and experience of management, the software engineers, and test engineers will assist in achieving this goal without the need for arbitrary “x number of tests per line of code” goals that can tend to demoralize team members.
The system environment phase details the environment the software will be running in or tested in. Team members from the business analyst, test engineer, and software engineer teams should be represented while working on this phase. Testing should occur in environments as similar to production as possible. All configuration changes should be logged in a central repository and any configuration scripts should be stored in the code repository.
The test execution phase is when the test engineers and software engineers execute the test plan and log the results of each test. It is important notate whether each test passes, fails, or was skipped. If a test fails the results of what the software did needs to be documented along with the expected output. If a test is skipped it is important to document why it was skipped so it can be reviewed to see if it is necessary to be included in the next iteration’s test plan. The results of the test execution phase are gathered and compiled into the output of the test reporting phase.
The test reporting phase involves any person that ran a test to provide a summary of the pass/fail/skip status including details of the tests to the manager of development operations. The manager will review to ensure all necessary testing has been completed and will conduct a meeting with representatives from the business analyst, project management, software engineering, and test engineering teams to develop a ‘lessons learned’ document in the development and testing process of the current iteration. In the meeting insight from the customer base and project management teams for any insights to how we can improve processes during future iterations.
Discovery of Business Requirements
The agile model will help assist us with the client’s inability to provide lower level functional requirements. Our managing directors or C-level management will need to have talks with the client’s business in order to get the high level functional specification signed off as a scope of work. We then can work on getting functional specifications for each iteration completed. Our upper management also may need to stay engaged with their counterparts in the customer’s business in order to ensure that we are not only maintaining the level of service that the customer expects, but to keep them engaged and able to provide us the information needed to start each iteration on time.
Breaking up the project in discreet sections of functionality in each iteration will help by focusing the customer on what they need from smaller sub sections of the entire application. This will prevent them from potentially causing delays by attempting to over-think their needs based on the entire large software project. Our upper management will help the process by working to the customer’s business and letting them know that there are delays on the customer’s end that they may be able to break through to deliver results.
The project likely cannot be finished in 2.5 months because we do not have a signed scope of work document or project charter that is well defined from the client. Acknowledging that we attempted to work with the client and started development before the scope of work was completed which was outside of our normal business practices but we cannot honor our given deadline as the scope of the project continues to change.
Alerting the client that we can work on their project and deliver functionality to their users sooner with our changes to the way we manage their project will inspire goodwill. The change will also allow us to alleviate the pressure from their staff from needing to gather requirements from the entire system internally to deliver the functional specification to us at once and allow them to focus their efforts while being more reactive to change.
Activating the highest levels of our management team to reach out to the client’s business to explain the situation and get a scope of work signed off on will allow us to complete the project in the 4-month time frame. The client will be happier to know that they will start getting software features delivered sooner and that our development teams will be responsive to any changes that need to occur and changes can be implemented within 2-4 weeks.
Starting a media campaign involving blog posts, social media posts, videos on media platforms, as well as having our external client facing reps alerting their contacts touting our business’ development strategies. We can also focus on how we are innovating in the software development and testing fields will help bring a sense of understanding when media reports come out regarding the software being delayed. A preemptive media campaign can also allow us to shape the message that we are delivering features of the software package before the initial 2.5-month timeline even though the initial timeline was for the entire software package to be delivered then.
Our focus is on delivering quality software with high reliability and high availability. While continuing on with our current strategies would allow us to complete the tasks needed, changing our software development method will allow us to be more responsive to our customer’s needs, build quality into the software development process, and have a more engaged team that shares values and information.
Bartlett, J. (2016, August 02). Software Testing Life Cycle Phases (STLC). Retrieved September 17, 2017, from https://blog.testlodge.com/software-testing-life-cycle/
Samanta, U., & Mani, V. S. (2015). Successfully transforming to Lean by changing the mindset in a global product development team. 2015 IEEE 10th International Conference on Global Software Engineering, 135-139. doi:10.1109/ICGSE.2015.17
Fucci, D., & Turham, B. (2014). On the role of tests in test-driven development: a differentiated and partial replication. Empirical Software Engineering, 277-302. doi: 10.1007/s10664-013-9259-7
Abdelnour-Nocera, J., & Sharp, H. (2012). Understanding Con icts in Agile Adoption through Technological Frames. International Journal of Sociotechnology and Knowledge Development, 29-45. doi:10.4018/jskd.2012040104