DevSecOps Modernization

at NASA’s Agency Application Office

Agency Application Office

NASA's Agency Application Office (AAO) is located at Marshall Space Flight Center and manages and modernizes critical enterprise applications and IT systems across the agency. It oversees the adoption of innovative technologies, cloud infrastructure, and DevSecOps practices to enhance efficiency, reduce costs, and improve security in NASA’s IT operations. The NASA Application Office has since been renamed to Application and Platform Services (APS) since the time this case study occurred.

The Problem

In 2015, AAO supported over 100 applications and services. The approaches AAO used for software development, delivery, security, and maintenance had grown outdated and this was causing considerable obstacles that hindered their effectiveness and their ability to bring on more workloads.

The following challenges existed:

Manual Code Deployment: Code was being deployed manually by human operators across various environments, making the process error-prone, slow and labor intensive.

Mutable Infrastructure: Software was run on Virtual Machines that were updated in place, manually over time. Drift naturally occurred and not all environments were the same leading to runtime inconsistencies.

Siloed Teams: Development, security, and operations teams operated in silos, with limited and often contentious communication. The majority of communication was not direct but proxied through Jira.

Limited Traceability: Auditing code deployments was challenging. In many cases it was impossible to know which Subversion commit was actually deployed to the various environments.

Inconsistent Testing Practices: Automated testing was limited or absent. Manual testing varied between teams or environments, leading to inconsistent quality and undetected regressions.

Infrequent Releases: Due to the friction in the deployment process, releases were infrequent and large, increasing the risk of integration failures and making troubleshooting more complex.

Acknowledging the need for change, NASA collaborated with TekFive partners to devise and execute a strategy to tackle these issues. Employing NASA’s agile Scrum framework, the TekFive team began efforts to address these challenges.

The Solution

We began the effort by creating a comprehensive modernization strategy focused on adopting DevSecOps principles and transitioning AAO to cloud-based infrastructure. As part of this effort we performed an Opportunity Assessment that detailed the potential benefits, risks, and a high- level roadmap for migration, emphasizing business value (key challenges addressed), workforce training, and enhanced security posture.

One of the key details of the approach was that AAO would first be transitioned to a private cloud solution. At the time, NASA had stricter requirements about which workloads could run off premises and there was more general concern about non-Government infrastructure. This simplifying solution allowed NASA to gain more immediate benefit from the technology, people and process transformation while paving the road for an eventual Agency hybrid cloud solution.

TekFive partnered with the Marshall and Agency Computing Services (MACS) team to identify RedHat’s OpenShift as the ideal hosting solution, blending MACS’s infrastructure expertise with TekFive’s platform integration skills, and then crafted a detailed, multi-year Project Execution Plan—approved by NASA leadership—that outlined technical milestones and workforce training, targeting three critical areas: source code control modernization, CI/CD pipeline implementation, and legacy application transformation.

Migrating AAO projects from Subversion to Git was important due to the general community adoption of Git and its native integration with many Kubernetes tools. TekFive conducted a product evaluation of available Git providers and chose GitLab due to its community support, open source code base, and additional capabilities it provides such as pipelines. TekFive team members implemented GitLab and then developed a process for safely transitioning AAO Subversion projects to GitLab while maintaining the full commit history. The Git conceptual model is significantly different from Subversion and some of these new concepts were difficult for team members to grasp. TekFive served as the subject matter experts on all things Git and provided numerous team and one-on-one training sessions as well as producing AAO specific Git documentation to bring developers up to speed.

Implementing a CI/CD pipeline was key because it would help eliminate many of AAO issues including manual code deployments, siloed groups and traceability. Originally we envisioned this pipeline would enable teams to perform Continuous Deployment but found that this was challenging to get approval for in NASA. NASA leadership still wanted a full chain of manual control that explicitly approved each release. We also found that even with the automation and approval in place, many NASA owners still wanted their applications to be deployed during the weekly outage window.

Finding a CI/CD platform that supported these requirements proved difficult. Instead TekFive implemented a custom pipeline platform that worked in conjunction with GitLab’s CI capabilities to provide the following features.

The custom pipeline platform supported NASA’s specific needs:
  • Automatic and continual container image scanning for known vulnerabilities.
  • DAST and Accessibility scanning of web applications.
  • Publishing pipeline scanning results to security team’s system of records.
  • Human in the loop approval state that allowed NASA officials to manually sign off on all staging and production code releases.
  • Scheduling capabilities to automate the code deployment at certain day and time.
  • Full code deployment traceability back to Git commit history.
  • Automatic code rollbacks.

At the time of this transformation AAO supported over 100 applications. TekFive led the effort to transition these legacy applications so they could be deployed through the pipeline, run on OpenShift and take advantage of each. Early on in this process TekFive piloted the transformation of multiple applications. This provided us early and continual feedback on the operation and usability of the DevSecOps platform and served as the blueprint for other application transformations.

Going forward, TekFive worked directly with each team to provide guidance and expertise on how OpenShift and the CI/CD platform works and how their applications should be modified. TekFive was also responsible for transitioning “orphaned” applications that were not associated with an active team. We took a Minimal Viable Transformation (MVT) approach during the transformation process by only modifying the necessary pieces to be deployed and run on the new platforms. This resulted in a much shorter transition period and a much more stable transition while still gaining much of the benefit from this new approach.

Additionally TekFive team members established a DevSecOps platform team that was responsible for building, maintaining shared and base images, architecting and supporting configuration approach, supporting the CI/CD platform, implementing new features to the platform as needed, and providing ongoing support and training for AAO developers.

The Benefits

Overall the transition to DevSecOps and private cloud infrastructure has been a huge success for AAO. Their overall time to release went from, on average, 3 months to 2 weeks with Annual savings of over $2M in labor and infrastructure costs. Other key metrics include the following.

The custom pipeline platform supported NASA’s specific needs:
  • Annual labor savings over $1.5M (50% ↓).
  • Annual infrastructure savings over $400K (30% ↓).
  • Allowed AAO to increase supported workloads by 2K% with no increase in staffing which has resulted in a 16K% increase in application traffic.
  • 90% of security vulnerabilities are now caught prior to deployment. Previously these were all caught post deployment.
  • 80% reduction in the development lifecycle on average from first commit to production deployment.

Once AAO was fully transitioned to a DevSecOps approach on private cloud infrastructure, TekFive began conducting pilots with AWS, Azure and GCP to incorporate public cloud services into the existing infrastructure to supplement and enhance AAO’s offerings. This resulted in services from all three public clouds being available for AAO customers and AWS and Azure also now provide active/active, active/passive and/or cloud bursting capabilities for the NASA agency because of these efforts.

TEKFIVE is an outstanding company to partner with; individuals who guide teams, set visions, drive innovation, and deliver high quality services. They are the leaders/experts, transitioning NASA’s Agency Applications Office 100+ applications to a containerized platform and transformed the many development/deployment processes to a standard/repeatable DevSecOps. Just an outstanding group to work with.

Ron NewbyEnterprise Services (Retired), NASA AAO