Part 2 - Is DevOps the Wild West? - DevOps Dilemmas
By: Troy Hagerty
In Part 1 of our DevOps series, we took a look at some non-technical, high-level issues that we at ARORA Solutions have discovered within our day-to-day operations talking to clients and through our conversation with our contacts in the Development world.
This time, we’re going for a deeper dive into the dense forest of DevOps and tackle some of the biggest dilemmas we have observed both during our audit process and first-hand experience from a DevOps perspective. We compare these directly with requirements in the brand spanking new release of ISO/IEC 27001:2022 version released October 2022.
Dilemma 1 – Lack of Security Oversight and Resource Constraints
In our observations, we were quite alarmed during the audit process due to a few discoveries: While there were some ad-hoc checks and balances throughout the Software Development Life Cycle (SDLC), there didn’t appear to be truly solidified processes to ensure proper implementation of security controls in the SDLC. This is mainly due to capacity constraints within organizations. Most DevOps teams have highly skilled individuals responsible for critical functions, and thus requiring collaboration amongst close-knit teams. Therefore, only limited security oversight is available, due to the highly complex nature of the code and architecture. Furthermore, resource constraints make having an additional “approval” layer both cost and resource prohibitive.
Newly released ISO/IEC 27001:2022 clearly requires secure development to take place within an organization (Annex A Control A.8.25).
Dilemma 2 – New Projects Incorporating Security
In several large organizations we assessed, new projects bypassed the Security teams and Change Advisory Board, altogether. Those projects did not have a security impact analysis or risk assessment attached to the project prior to implementation. Why does this continually happen?
Well, this leads us to a discussion on “Culture” in Dilemma 3. Additionally, the Change Management Process, SDLC Policies and Secure Development Policies and process are not clearly understood or communicated within the organization. In our understanding, Policy + Communication + Training + Security Controls = Secure Systems. Most organizations fail spectacularly in these areas.
Newly released ISO/IEC 27001:2022 clearly requires security to be incorporated into new projects (Annex A Control A.5.8) and change management (Annex A Control A.8.32).
Dilemma 3 – Culture of Openness and Collaboration vs Culture of Security
Development culture runs deep within all organizations, due to the historical nature of software development. Developers, as a whole, have often operated under this mindset: “freedom and collaboration leads to innovation”. This could include freedom of knowledge, freedom of technology by using open-source software, etc.
The depth of collaboration and freedom is on display on Github, Stack Overflow and Reddit. Chances are someone has already solved a problem and posted the working code one of those platforms. For the sake of efficiency and ease of access, developers enjoy a certain appreciate this freedom to share and utilize any other developer’s works. However, this culture doesn’t quite jive with the need to protect corporate intellectual property, privacy, and security of organizational assets.
While certainly not malicious in nature, that culture of openness could be exploited by those who wish to do harm. There have been examples, which we’ve seen, where company-owned code was being stored on a private personal Github and not adequately in control of the company. If an attacker were to obtain that code and analyze it, once they figure out how it operates, they can use that code to construct their own backdoor or to exploit weaknesses.
Newly released ISO/IEC 27001:2022 clearly requires access to source code be managed and restricted (Annex A Control A.8.4). ISO/IEC 27002:2022 provides even further guidance on limited editing abilities and access by technical and procedural controls.
Dilemma 4 – The Nature of Security in Deployment: Rolling Release vs Phased Release
There are two methodologies to consider when understanding software development. Each have different security implications. Companies can choose to adopt a rolling / continuous release methodology or a phased release methodology. While both ultimately support the same goal of ensuring that your program is continually improving itself, the execution is slightly different:
Rolling (Continuous) Release. Continuous release ensures that any time a new update is pushed to production, it automatically (or semi-automatically) is pushed out to the wild. By having changes be pushed in smaller increments in a frequent fashion, this results in faster deployment with fewer errors. Bugs in code are easy to rollback because they can be isolated and are mostly minor.
What are the security implications? Due to the speed and dynamic nature of rolling changes, the Change Management function supported by Security Teams does not have visibility into small but critical operational fixes.
Phased Release: Unlike continuous release, phased deployment is based upon more of a business rationale rather than a technical one. Phased release is directly tied to an organization’s business strategy, such as revenue or portfolio management. In other words, if it makes “business sense” to release the patch into deployment, they’ll update in large chunks instead of in small, frequent parts.
What are the security implications? In disaster recovery situation, it may take a long time to roll back the changes, causing system availability issues (“downtime”). Also, with large releases many variables are at play. Many dependent and interrelated processes are affected by the functionality of another. If one function goes down, due to a poor release, another may also going down. Numerous patches released at once make identification the root cause of an issue to take even longer to resolve.
Regardless of the deployment and release methodologies, SecOps should be involved in these new projects, change management and patching. Why? The security and quality of software is related to a company’s success. Poor quality development and uncontrolled changes can deeply interfere with business performance.
Newly released ISO/IEC 27001:2022 requires technical review of applications after operating platform changes (Annex A Control A.8.32). Security testing processes should also be defined and implemented Annex A Control A.8.29).
Dilemma 5 – DevOps and Security Teams Lack Skilled Developers and Security Engineers
The more complicated a system is, the narrower an individual’s focus becomes. There really aren’t sufficient layers of people who have adequate understanding of a specific component (code-level) to really know if it is properly functioning or secure. Within most Dev teams we encounter, only about 3 people know what is happening on an infrastructure level for a specific service.
The situation in these companies isn’t unique. Due to the ever-evolving complexity of various platforms, each with their own infrastructure and dependencies that need to be monitored, today’s controls with today’s skillset will quickly become obsolete tomorrow due to the ever-increasing speed of innovation.
The other half of the equation also appears to be the employers’ unattainable hiring expectations. Any job hunter in the cybersecurity field in recent years have seen their fair share of positions with unrealistic expectations of their potential candidates. Some positions are calling for multiple certifications, extended secondary education, multiple years of professional experience, coding experience in multiple languages, etc., all for a position that’s listed as “entry level” or “junior”.
Dilemma 6 – Harmonization of DevOps with international standards, IE ISO/IEC 27001:2022
Throughout this article, we have compared the ISO 27000 series to actual practices in DevOps. We have found some consistencies among these practices. However, most of the issues we have found relate to conflicting cultures. DevOps operates under the MO of freedom and collaboration. Whereas, Security operates under the assumption of process and control to ensure information, data and assets are protected.
Improvements in DevOps are clearly noted in our research and audit engagements. General improvements are needed in DevOps environments as it relates to:
technical control (access management, cryptography),
procedural controls (e.g. SDLC, project management, change management) and
basic understanding for the need of information security practices.
We think it is important to share with the world our experience with DevOps environments, with hopes of improving and securing our infrastructure, systems, data, privacy, and intellectual property.