DORA’s Research Program: 2022

Structural Equation Model

The DORA research program uses structural equation modeling (SEM), a predictive statistical modeling technique used to test relationships. We utilize PLS-PM (partial least squares path modeling) for our analysis, for several reasons: it does not require assumptions of normality in the data, it is will suited to exploratory and incremental research, and the analysis optimizes for prediction of the dependent variable (vs. testing for model fit of the data). All paths shown in the SEM figures are p < .05.

Survey Questions

Responses to the following questions were used in the analysis published in the 2022 Accelerate State of DevOps Report

Architecture

Think about how your team and work are structured. For the primary application or service you work on, please rate how strongly you agree or disagree with the following statements.

• My team can deploy and release our product or service on demand, independently of other services it depends upon
• On my team, we can make large-scale changes to the design of our system without creating significant work for other teams
• On my team, we can make large-scale changes to the design of our system without depending on other teams to make changes in their systems
• On my team, we perform deployments during normal business hours with negligible downtime
• To complete my own work, I don't need to communicate and coordinate with people outside my team
• We can do most of our testing on demand, without requiring an integrated test environment

Burnout

Sustainability of work is important, and burnout is an important indicator of this. Please answer a few questions about how your work has affected you. Please rate how strongly you agree or disagree with the following statements:

• I am indifferent or cynical about my work.
• I feel burned out from my work.
• I feel exhausted.
• I feel like I am ineffective in my work.
• My feelings about work negatively affect my life outside of work.

Cloud Computing

Please rate how strongly you agree or disagree with the following statements in relation to your use of cloud services.

• I can dynamically increase or decrease the cloud resources available for the service or product that I primarily support based on demand
• I can monitor or control the quantity and/or cost of cloud resources used by the service or product that I primarily support
• Once I have access, I can independently provision and configure the cloud resources and capabilities required for my product or service on demand without raising tickets or requiring human interaction
• The cloud my product or service runs on serves multiple teams and applications, with compute and infrastructure resources dynamically assigned and re-assigned based on demand

Continuous Delivery

For the primary application or service you work on, please rate how strongly you agree or disagree with each of the following statements.

• Fast feedback on the quality and deployability of the system is available to anyone on the team
• My team prioritizes keeping the software deployable over working on new features
• Our software is in a deployable state throughout its lifecycle
• We can deploy our system to production, or to end users, at any time, on demand
• When people get feedback that the system is not deployable (such as failing builds or tests), they make fixing these issues their highest priority

Continuous Integration

The next set of questions ask about committing code and building and deploying software at work. For the primary application or service you work on, please rate how strongly you agree or disagree with the following statements.

• Automated builds and tests are executed successfully every day
• Automated test failures will block a commit's progress through the pipeline
• Code commits result in a series of automated tests being run
• Code commits result in an automated build of the software

Demographics and Firmographics

• Approximately many people are on your immediate team?
• Do you identify as a member of an underrepresented group?
• Do you identify as a person with a disability? This includes disabilities associated with seeing, hearing, walking, remembering/concentrating, self-care, or communication.
• How many employees work at your organization?
• How many years of experience do you have in a role similar to your current role?
• Location
• Start Date
• User Language
• What is the principal industry of your organization?
• What is your gender?
• Which most closely describes your work role?
  Development or Engineering,DevOps or SRE,I am a C-level executive,I am a consultant, coach, or trainer,I am a student,I do not belong to any department,Information Security,IT Operations or Infrastructure,Manager,Network Operations,Other (please specify),Prefer not to answer,Product Management,Professional Services,Quality Engineering or Assurance,Release Engineering,Sales Engineering,Sales or Marketing,User experience or software analysis
• With which racial or ethnic groups do you identify? Select all that apply
  American Indian or Alaska Native, Asian, Black or African American, Hispanic, Latino, or Spanish origin, Middle Eastern or North African, Native Hawaiian or Pacific Islander, Other (please specify), Prefer not to respond, White

Documentation

The following questions ask about documentation. We define documentation as internal documentation for the services or applications you work on (e.g., manuals, readmes, code comments, etc.). End-users are not the audience for these documents. Please rate how strongly you agree or disagree with the following statements.

• I can rely on our technical documentation when I need to use or work with the services or applications I work on
• It is easy for me to understand our technical documentation
• It is easy to find the right technical document when I need to understand something about the services or applications I work on
• Most of the code for the services or applications I work with is documented
• Our technical documentation is well organized
• Technical documentation is updated as changes are made
• The technical documentation correctly reflects the capabilities and requirements of the primary service or application I work on
• When there's an incident or problem that needs troubleshooting, I reach for the documentation

Generative Organizational Culture

Think about your organization when you answer the following questions and rate how strongly you agree or disagree with the following statements. In my organization...

• Cross-functional collaboration is encouraged and rewarded
• Failures are treated primarily as opportunities to improve the system
• Information is actively sought
• Messengers are not punished when they deliver news of failures or other bad news
• New ideas are welcomed
• Responsibilities are shared

Infrastructure Platform

Please rate how strongly you agree or disagree with the following statement.

• The product or service I primarily work on was originally designed and architected to run in the cloud.

Organizational Performance

Select the option that indicates how your organization performed for the following goals over the past year.

• Achieving our organizational and mission goals
• Customer satisfaction
• Increased number of customers
• Operating efficiency
• Other measures that demonstrate to external parties that your organization achieve intended results
• Quality of products or services provided
• Quantity of products or services
• Relative market share for primary products
• Your organization's overall profitability
• Your organizations overall performance

Predictions

Making predictions isn't easy, but some people find it helps them better express their perspective. Please indicate how unlikely or likely you think the following events are for the primary service or application you work on.

• Leadership will sponsor one or more software security initiatives in the next year
• We will be compromised by a security vulnerability in the next year
• We will have a complete service/product outage in the next year
• We will have a significant performance degradation in the next year
• We will have an increased spending budget for new tools in the next year
• We will have an outage due to quota or capacity misunderstanding in the next 90 days
• We will increase the amount funding for getting new tools in the next year
• We will invest more to improve our build systems in the next year
• We will invest more to improve our deployment security and monitoring in the next year
• We will invest more to reduce risk for dependencies in the next year

Reliability

These next questions ask about reliability, and how you and your team think about it. For the primary application or service you work on, please rate how strongly you agree or disagree with each of the following statements.

• I know what its reliability actually was in the most recent period
• It has well-defined targets for reliability (such as Service Level Agreements / Service Level Objectives) that are clearly communicated among the team and to customers
• My team regularly reviews and revises reliability targets based on evidence
• Reliability reviews are performed throughout the development process for all major features on the applications I work on
• When we miss our reliability targets, we perform improvement work and/or re-prioritize

• How often do your services meet reliability expectations (i.e., SLOs)?

Security

How established are the following practices for the primary application or service you work on?

• Build definitions and configurations are defined in text files stored in a version control system
• Build metadata (e.g., dependencies, build process, build environment) about an artifact cannot be edited by build services users
• Build metadata (e.g., dependencies, build process, build environment) about an artifact documents all dependencies
• Build metadata (e.g., dependencies, build process, build environment) about an artifact includes all build parameters
• Build metadata (e.g., dependencies, build process, build environment) is available to the people who need it (e.g. via a central database), and is delivered in a format that they accept
• Build metadata (e.g., dependencies, build process, build environment) is either generated by the build service, or by a build-metadata generator that reads the build service
• Builds are fully defined through the build script and nothing else
• Builds are isolated; they cannot interfere with concurrent or subsequent builds
• Every change in a revision's history must be individually reviewed and approved by two trusted persons prior to submission
• Production releases are built by using a centralized CI/CD system, never on a developer’s workstation
• Revisions and their change history are preserved indefinitely
• The build metadata (e.g., dependencies, build process, build environment) about how an artifact was produced is signed by my build service
• When running builds, build steps are prevented from loading any build inputs dynamically (i.e., all required sources and dependencies are fetched upfront)

Please indicate the extent to which you agree with the following statements.

• At my company, the software-security protocol is seamlessly built into our development process
• At my company, we have a standardized process for addressing software security across projects
• I have access to the necessary tooling to execute security tests
• My organization has an effective method for addressing security threats
• My organization has an effective method for identifying security threats
• Our org has processes in place to identify and document all security requirements for the software our organization develops or acquires (including third-party and open source)
• Security requirements are reviewed at regular intervals (annually, or sooner if required)
• Security review is conducted for all major features on the applications I work on
• Security roles are integrated into our software development team
• Security tests are run early in the software development process, either by me or by another team
• We continuously engage in automated or manual code analysis and testing for all supported releases, in order to identify or confirm the presence of previously undetected vulnerabilities
• We have ongoing efforts to monitor information coming from public sources regarding possible vulnerabilities in the software we use and its third-party components

Software Delivery Performance

• For the primary application or service you work on, how long does it generally take to restore service when a service incident or a defect that impacts users occurs (e.g., unplanned outage, service impairment)?
• For the primary application or service you work on, how often does your organization deploy code to production or release it to end users?
• For the primary application or service you work on, what is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)?
• For the primary application or service you work on, what percentage of changes to production or released to users result in degraded service (e.g., lead to service impairment or service outage) and subsequently require remediation (e.g., require a hotfix, rollback, fix forward, patch)?

Team Characteristics

Think about your team (the people around you who report to the same manager or skip-level manager). To what extent do you agree or disagree with the following statements?

• Most of the people that were on this team 12 months ago are still on the team today

Trunk-Based Development

We are interested in the development practices and patterns you follow at work. Please rate how strongly you agree or disagree with the following statements.

• All developers on my team push code to trunk / main branch at least daily
• Branches and forks have very short lifetimes (less than a day) before being merged to the main branch
• There are fewer than three active branches on the application's code repo

Version Control

For the primary application or service you work on, please rate how strongly you agree or disagree with each of the following statements.

• Our application code is in a version control system
• Our application configurations are in a version control system
• Our scripts for automating build and configuration are in a version control system
• Our system configurations are in a version control system

Work Environment

• Approximately what percentage of your time is spent on rework or unplanned work?
• Currently, how inflexible or flexible is your company with regard to employee work arrangements (e.g., voluntary work from home, full-time remote work, hybrid schedules, etc.)?
• How likely are you to recommend your team as a place to work to a friend or colleague?