Impact of Generative AI in Software Development

Survey Questions

Responses to the following questions were used to inform the analysis published in the Impact of Generative AI in Software Development report.

Approval process

• For the primary application or service you work on, how long does it typically take to receive feedback from the approvals process (that is, how long does it take from when a code change is proposed to when it is approved for production use)?
  More than six months Between one month and six months Between one week and one month Between one day and one week Less than one day Less than one hour We do not receive feedback I don't know

Artificial Intelligence (AI)

• Over the last 3 months, approximately what percentage of your team's work is supported by AI?

For the primary application or service you work on, how much have you relied on AI for each of the following tasks over the last 3 months?

• Summarizing or accessing information
• Analyzing data (for example, logs)
• Analyzing security
• Code explanation
• Code reviews
• Debugging
• Migrating from one language to another
• Modernizing legacy codebases
• Optimizing code (for example, reducing the complexity of a function)
• Writing code blocks or functions (for example, auto-completion, auto-generation or auto-formatting)
• Writing documentation
• Writing tests

In the last 3 months, how frequently have you interacted with AI at work in each of the following:

• As an automated part of our tool chain (for example, our integration, build, or test services)
• Conversational AI interfaces (for example, chatbots in my IDE or on websites)
• External web interfaces (for example, external websites and documentation)
• Internal web interfaces (for example, internal websites, portals, and documentation)
• My development environment (your code editor or IDE)
• Other
• Within other tools and platforms I use for development, excluding my IDE

When you think about the following areas, what kind of impact do you think AI will have within the next [year | five years | ten years]. (respondents were randomly selected to answer for one of the three time horizons.)

• Society as a whole
• The environment (for example, ecosystems and climate)
• The performance of your organization
• The quality of your product
• The speed and stability of the software delivery process
• Your career

• For the primary application or service you work on, how much have you relied on AI over the last 3 months?
  Not at all relied on AI Minimally relied on AI Somewhat relied on AI Moderately relied on AI Extensively relied on AI I don't know or NA

• In the last 3 months, how has AI impacted your productivity at work?
  Extremely increased my productivity Moderately increased my productivity Slightly increased my productivity No impact on my productivity Slightly decreased my productivity Moderately decreased my productivity Extremely decreased my productivity I don’t know or NA

• In the last 3 months, to what extent has your organization shifted priorities to incorporate AI into its applications and services?
  Significant increase in AI prioritization Moderate increase in AI prioritization Slight increase in AI prioritization No change in AI prioritization Slight decrease in AI prioritization Moderate decrease in AI prioritization Significant decrease in AI prioritization I don’t know or NA

• In the last 3 months, how has AI impacted your ability to write code?
  Extremely improved my ability to write code Moderately improved my ability to write code Slightly improved my ability to write code No impact on my ability to write code Slightly worsened my ability to write code Moderately worsened my ability to write code Extremely worsened my ability to write code I don’t know or NA I don’t use AI to write code

• In the last 3 months, how much did you trust the quality of the output from AI-generated code as part of your development work?
  Not at all A little Somewhat A lot A great deal I don’t know or NA

• I trust my organization to be transparent with how they plan on using AI.
  Strongly agree Mostly agree Somewhat agree Neither agree nor disagree Somewhat disagree Mostly disagree Strongly disagree I don't know or NA

Burnout

• I am indifferent or cynical about my work.
• I feel burned out from my work.
• I feel like I am ineffective in my work.
• My feelings about work negatively affect my life outside of work.

Code complexity

• In the last 6 months, how much has your primary application or service been hindered by code complexity if at all?
  Not at all hindered Slightly hindered Moderately hindered Very hindered Extremely hindered I don’t know or NA

Code quality

• In the last 6 months, how satisfied or dissatisfied have you been with the quality of code underlying your primary service or application?
  Extremely satisfied I don’t know or NA Satisfied Slightly satisfied Neither satisfied or dissatisfied Slightly dissatisfied Dissatisfied Extremely dissatisfied

Code review time

• For the primary application or service you work on, how long does it generally take to complete the code review process?
  More than six months Between one month and six months Between one week and one month Between one day and one week Less than one day Less than one hour I don’t know We do not do code reviews

Continuous Delivery

For the primary application or service you work on, please rate how strongly you agree or disagree with each of the following statements

• A failing test will stop the team from deploying the system
• Fast feedback on the deployability of the system is available to anyone on the team
• My team prioritizes keeping the software deployable over working on new features
• Our software is in a deployable state throughout its lifecycle
• We can deploy our system to production, or to end users, at any time, on demand
• When people get feedback that the system is not deployable (such as failing builds or tests), they make fixing these issues their highest priority

Continuous Integration

Think about building and deploying software at work. For the primary application or service you work on, please rate how strongly you agree or disagree with the following statements.

• Code commits result in an automated build of the software.
• The application is built and tested automatically at least daily.

Cross-functional coordination

Many people work as part of a cross-functional team that may include project managers, product managers, test engineers, software developers, technical writers, or user experience (UX designers / researchers). Thinking about the cross-functional team you work on, how much do you agree with the following statement?

• Over the last 3 months, I have been able to effectively collaborate with cross-functional team members.
  Strongly agree Mostly agree Somewhat agree Neither agree nor disagree Somewhat disagree Mostly disagree Strongly disagree I don't know or NA

Culture

Think about your organization and rate how strongly you agree or disagree with the following statements. In my organization...

• Cross-functional collaboration is encouraged and rewarded
• Failures are treated primarily as opportunities to improve the system
• Information is actively sought
• Messengers are not punished when they deliver news of failures or other bad news
• New ideas are welcomed
• Responsibilities are shared

Documentation quality

Think about the documentation you use. We define documentation as internal documentation for the services or applications you work on (for example, manuals, readmes, code comments, etc.). End users are not the audience for these documents. Please rate how strongly you agree or disagree with the following statements.

• I can rely on our technical documentation when I need to use or work with the services or applications I work on.
• It is easy to find the right technical document when I need to understand something about the services or applications I work on.
• Technical documentation is updated as changes are made.
• When there’s an incident or problem that needs troubleshooting, I reach for the documentation.

Flow

• In the last 3 months, how often were you able to reach a high level of focus or achieve 'flow' during development tasks?
  Always Frequently Occasionally Rarely Never I don’t know or prefer not to answer

Job satisfaction

• How do you feel about your job as a whole?
  Extremely satisfied Satisfied Slightly satisfied Neither satisfied nor dissatisfied Slightly dissatisfied Dissatisfied Extremely dissatisfied I don't know or prefer not to answer

Organizational performance

For the following questions, we are interested in your organization's performance. If you are in a contract position, consider your current client to be your organization.

For each of the following performance indicators, how well did your organization meet its goals over the past year?

• Increased number of customers
• Relative market share for primary products
• Your organization’s overall performance
• Your organization’s overall profitability

For the following questions, we are interested in hearing about your organization's performance. If you are in a contract position, consider your current client to be your organization.

• Achievement of organizational and mission goals
• Customer satisfaction
• Operating efficiency
• Quality of products or services provided

Organizational practices

Rate how strongly you agree or disagree with the following statements.

• My organization actively encourages employees to incorporate AI tools into their workflow.
• My organization allows employees to take time during their work hours to learn to use AI tools.
• My organization has policies that allow employees to clearly understand how and when to use AI tools.
• My organization has policies that govern the adoption of AI-powered tools and services.
• My organization has proactively established safeguards against security and privacy breaches when utilizing AI.
• My organization has taken concrete steps to alleviate developers' worries about job displacement as a result of AI adoption.
• My organization is committed to the professional development of its employees.
• My organization is transparent about how AI is being used.
• My organization provides guidelines regarding how to mitigate security and privacy risks when engaging with AI
• My organization provides resources (courses, webinars, etc.) for employees to learn how to best incorporate AI into their workflows.
• My organization requires employees to complete trainings that provide no inside or outside value to their job roles.
• My organization's leadership has clear goals regarding how it expects AI to impact organizational performance

Organizational priorities

• How stable or unstable are the priorities of your organization?
  Extremely stable Very stable Somewhat stable Neither stable nor unstable Somewhat unstable Very unstable Extremely unstable I don’t know or NA

Product performance

The following question is about your primary service or application and how it performs for your end users. The primary application or service I work on...

• does well on performance metrics like latency
• does what it is supposed to do
• helps people accomplish what is important to them
• is usable and easy to navigate
• keeps users’ information safe
• reliably available for users

Productivity

Thinking about the work you do, please rate how strongly you agree or disagree with the following statements.

• I am able to do my work in the most effective way possible.
• I am productive at work.
• My work creates value.
• My work is aligned with my set of skills.

Software delivery performance

The next set of questions are about various outcomes of your work. For the primary application or service you work on...

• Approximately what percentage of changes to production or released to users result in degraded service (for example, lead to service impairment or service outage) and subsequently require remediation (for example, require a hotfix, rollback, fix forward or patch), if at all?
  (value between 0% and 100%) I don't know or NA
• Approximately what percentage of deployments in the last 6 months were not planned but were performed to address a user-facing bug in the application?
  (value between 0% and 100%) I don't know or NA
• How long does it generally take to restore service after a change to production or release to users results in degraded service (for example, lead to service impairment or service outage) and subsequently require remediation (for example, require a hotfix, rollback, fix forward, or patch)?
  More than six months Between one month and six months Between one week and one month Between one day and one week Less than one day Less than one hour I don’t know or NA
• How often does your organization deploy code to production or release it to end users?
  Fewer than once per six months Between once per month and once every 6 months Between once per week and once per month Between once per day and once per week Between once per hour and once per day On demand (multiple deploys per day) I don't know or NA
• What is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)?
  More than six months Between one month and six months Between one week and one month Between one day and one week Less than one day Less than one hour I don’t know or NA

Technical debt

• In the last 6 months, how much did technical debt inside your primary application or service hinder your productivity if at all?
  I don’t know or NA Not at all hindered Slightly hindered Moderately hindered Very hindered Extremely hindered

User-centricity

Thinking about the primary application or service you work on, to what extent do you agree with the following statements?

• Creating value for our users is our focus.
• The experience of our users is our top priority.
• We believe that focusing on the user is key to the success of the business.
• We have a clear understanding of what our users want to accomplish with our application or service.
• We leverage user feedback to continuously revisit and reprioritize features.