DORA | DORA Research Questions

Survey Questions

Responses to the following questions were used in the analysis published in the 2019 Accelerate State of DevOps Report.

Automated Testing

Automated test failures are likely to indicate a real defect.
I can get feedback from automated tests in less than ten minutes.
It is easy for developers to reproduce and fix acceptance test failures.
We have the test data necessary to run our automated tests easily at every step.
We regularly use data from previous test runs to improve the quality of our automated test suite.
When the automated tests pass, I am confident the software is releasable.

Availability

For the primary application or service you work on:

I know what its availability actually was in the most recent period.
It has well-defined targets for availability (such as Service Level Agreements / Service Level Objectives) that are clearly communicated among the team and to customers.
My team met or exceeded our target for availability in the most recent period.
When we miss our availability targets, we perform improvement work and/or re-prioritize.

Burnout

I am indifferent or cynical about my work.
I feel burned out from my work.
I feel exhausted.
I feel like I am ineffective in my work.
My feelings about work negatively affect my life outside of work.

Change Approvals

All significant changes must be approved by a senior manager prior to implementation.
For all the types of changes that I typically make, I know the steps it takes to go from “submitted” to “accepted” every time.
I am confident I can get changes through the approval process in a timely manner.
I have a clear understanding of the process to get changes approved for implementation.
My organization has a formal process to approve changes to applications or production systems prior to implementation or release.
Production changes must be approved by an external body (e.g., change approval board, manager, etc.) before implementation or deployment.

Cloud

I can dynamically increase or decrease the cloud resources available for the service or product that I primarily support based on demand.
I can monitor or control the quantity and/or cost of cloud resources used by the service or product that I primarily support.
Once I have access, I can independently provision and configure the cloud resources and capabilities required for my product or service on demand without raising tickets or requiring human interaction.
The cloud my product or service runs on serves multiple teams and applications, with compute and infrastructure resources dynamically assigned and re-assigned based on demand.
The service or product that I primarily work on is designed to be accessed from a broad range of devices (e.g. smartphones, tablets, laptops) over the network without the need for proprietary plug-ins or protocols.

Code Maintainability

It is easy for me to add new dependencies to my project.
It is easy for me to find examples in our codebase.
It is easy for me to migrate to a new version of a dependency.
It is easy for me to reuse other people's code.
It's easy for us to change code maintained by other teams if we need to.
My dependencies are stable and rarely break my code.

Continuous Delivery

Fast feedback on the quality and deployability of the system is available to anyone on the team.
My team prioritizes keeping the software deployable over working on new features.
Our software is in a deployable state throughout its lifecycle.
We can deploy our system to production, or to end users, at any time, on demand.
When people get feedback that the system is not deployable (such as failing builds or tests), they make fixing these issues their highest priority.

Continuous Integration

Automated builds and tests are executed successfully every day.
Code commits result in a series of automated tests being run.
Code commits result in an automated build of the software.

Culture

I can depend on my team to deliver high-quality results.
If I make a mistake on our team, it is not held against me.
It is not difficult to ask other members of our team for help.
It is safe to take a risk on our team.
Members of our team are able to bring up problems and tough issues.
Members of our team do not reject each other for being different.
My team has clear roles and responsibilities.
My team provides an environment where I can be innovative.
My unique skills and talents are valued on our team.
No one on our team would deliberately act in a way that undermines my work efforts.
Our team is able to resolve conflict when it arises.
There is a high level of trust within our team.
When there are conflicting opinions among members of my team, we treat each other with respect.

Disaster Recovery Test Frequency

Response options: Never Only at initial deployment Weekly Monthly Yearly Less than once a year N/A or I don't know

How often does your organization perform application failover to test for resiliency?
How often does your organization perform infrastructure (including datacenter) failover to test for resiliency?
How often does your organization perform simulations that disrupt production systems (including failure injection such as degrading network links, turning off routers, etc.)?

Loosely Coupled Architecture

My team can deploy and release our product or service on demand, independently of other services it depends upon.
On my team, we can make large-scale changes to the design of our system without creating significant work for other teams.
On my team, we can make large-scale changes to the design of our system without depending on other teams to make changes in their systems.
To complete my own work, I don't need to communicate and coordinate with people outside my team.
We can do most of our testing on demand, without requiring an integrated test environment.

Monitoring

My team has a tech solution in place for monitoring key business and systems metrics.
My team has a tech solution in place to report on system state as experienced by customers (e.g., “do my customers know if my system is down and have a bad experience?”).
My team has a tech solution in place to report on the overall health of systems (e.g, Are my systems functioning? Do my systems have sufficient resources available?).
My team has tooling in place that can help us with understanding and debugging our systems in production.

Organizational Performance

For each of the following performance indicators, how well did your organization meet its goals over the past year?

Response options: Performed well above goals Performed above goals Performed slightly above goals Met goals Performed slightly below goals Performed below goals Performed well below goals N/A or I don’t know

Achieving our organizational and mission goals
Customer satisfaction
Increased number of customers
Operating efficiency
Other measures that demonstrate to external parties that your organization achieve intended results
Quality of products or services provided
Quantity of products or services
Relative market share for primary products
Your organization's overall performance
Your organization's overall profitability

Productivity

I am often able to get into a good "flow", where I can get complex, time-consuming tasks completed with minimal distractions and interruptions.
I regularly reach a high level of productivity.

Search

Internal Search

I frequently search my organization's source code repositories when I have questions or am looking for code examples.
I often refer to internal knowledge bases or tools for help finding solutions when I work on a task.
I often search internal tools or code repositories for solutions to similar problems or examples when I'm working on a challenging problem.
I search internal docs when I have a question or a challenging problem.

External Search

I frequently check externally (e.g., Stack Overflow, Google, etc.) when I need to improve my knowledge on how to solve a problem.
I often search externally (e.g., Stack Overflow, Google, etc.) to find solutions to similar problems when I am working on a challenging problem.

Software Delivery Performance

For the primary application or service you work on...

How long does it generally take to restore service when a service incident or a defect that impacts users occurs (e.g., unplanned outage, service impairment)?

More than six months Between one month and six months Between one week and one month Between one day and one week Less than one day Less than one hour I don't know or not applicable
How often does your organization deploy code to production or release it to end users?

Fewer than once per six months Between once per month and once every six months Between once per week and once per month Between once per day and once per week Between once per hour and once per day On demand (multiple deploys per day) I don't know or not applicable
What is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)?

More than six months Between one month and six months Between one week and one month Between one day and one week Less than one day Less than one hour I don't know or not applicable
What percentage of changes to production or released to users result in degraded service (e.g., lead to service impairment or service outage) and subsequently require remediation (e.g., require a hotfix, rollback, fix forward, patch)?

0%-15% 16%-30% 31%-45% 46%-60% 61%-75% 76%-100% I don't know / NA

Tech Debt

[In my work during the past year,] I encountered code, scripts, configuration, or systems ...

That contained known bugs that went unfixed in favor of new feature development.
That had insufficient tests or test coverage.
That had not been cleaned up after the cancellation or abandonment of a project.
That had problems related to low code quality or design.
That no one on my team had expertise to understand.
That used obsolete technology.
That were incompletely or incorrectly migrated.
Where documentation and/or comments were missing, incomplete, or outdated.

Trunk-Based Development

All developers on my team push code to trunk/master at least daily.
Branches and forks have very short lifetimes (less than a day) before being merged to master.
There are fewer than three active branches on the application's code repo.

Useful, Easy-To-Use Tools

When deploying software through a CI/CD/test automation toolchain:

I find it easy to get the toolchain to do what I want it to do.
I find the toolchain to be easy to use.
Interacting with the toolchain does not require a lot of my mental effort.
My interaction with the toolchain is clear and understandable.

When deploying software through a CI/CD toolchain:

I find the CI/CD toolchain to be useful in my job.
Using the CI/CD toolchain enhances my effectiveness in my job.
Using the CI/CD toolchain improves my performance in my job.
Using the CI/CD toolchain in my job increases my productivity.

Work Recovery

I am able to cope effectively with work-related stress.
I am able to detach from work during non-work time (i.e., when I choose not to be working).

DORA Research: 2019