Suzie Prince, Head of Product at Atlaasian recently spoke with Abi Noda, CEO at DX, and Andrew Boyagi, Head of DevOps Evangelism at Atlassian, about how AI is changing software development. They shared valuable insights on the best practices for using generative AI (GenAI) tools on your developer teams, how to get value from developer portals, and the latest approaches for measuring developer productivity.
Here are five key takeaways from this conversation on strategies for leveraging AI to boost developer productivity and experience.
Set realistic expectations for GenAI tools
The promise of GenAI tools, such as automated code generation and pull request descriptions, has excited the developer community. These tools have the potential to reduce repetitive tasks and free up developers’ time for more critical work.
However, Abi and Andrew caution against overestimating their immediate impact. While these tools can alleviate some of the toil associated with development, they also require time for correcting inaccuracies. Organizations need to carefully assess the actual benefits and effectiveness of these tools within their specific contexts to ensure they deliver meaningful productivity improvements. To set realistic expectations and effectively assess the benefits of GenAI tools, organizations can follow these steps:
- Define clear objectives and goals such as reducing code review time or improving code quality.
- Establish baselines by gathering current performance data on metrics like code review time and deployment frequency.
- Start with small pilot groups and gradually expand to manage expectations and mitigate risks.
- Combine qualitative feedback (e.g., developer satisfaction surveys) with quantitative metrics (e.g., reduced code review time).
- Continuously monitor performance and gather feedback to make necessary adjustments.
- Break down the implementation into incremental goals to maintain realistic expectations.
- Provide training and ongoing support to help developers effectively use GenAI tools.
Invest in training and adoption strategies
Providing developers with GenAI tools is just the first step. To truly reap the benefits, organizations must invest in comprehensive training programs that help developers understand and effectively utilize these tools. Abi emphasized the importance of identifying specific use cases that align with the organization’s unique environment. For instance, one company might focus on streamlining pull requests, while another might prioritize knowledge curation and discovery. By identifying these use cases that are specific to your organization, developers can maximize the potential of GenAI tools, leading to more significant improvements in their workflows and overall productivity.
Use phased rollouts and pilot groups
A common pitfall that many organizations encounter is the rush to purchase and deploy AI tools en masse without adequate usage studies. Abi and Andrew recommend starting with pilot groups and phased rollouts. This approach allows organizations to study the benefits and identify the most effective use cases within smaller, controlled populations. By understanding the impact of the tools in these settings, organizations can make more informed decisions about broader adoption, reducing the risk of widespread implementation failures.
Quantitative data won’t give you the full story
Many companies introduce AI tools to make devs more productive, and focus on quantitative data like number of pull requests or deployment frequency to measure the increase in productivity. But effective measurement of developer productivity requires a blend of qualitative and quantitative data. While traditional DevOps metrics provide valuable insights, they don’t capture the full scope of developer work.
Abi suggests combining these metrics with qualitative feedback from developers through surveys and direct feedback mechanisms. This holistic approach ensures that the measurements are actually meaningful and actionable, helping organizations identify areas for improvement and track progress over time.
A prime example of how to incorporate qualitative insights into your productivity measurements can be found in Compass. In partnership with DX, Compass offers DevEx 360 surveys designed by the researchers behind DORA and SPACE metrics. These surveys can help identify top opportunities to improve developer job satisfaction and pinpoint factors slowing down development teams. By integrating surveys directly into their workflows, teams can gather comprehensive feedback that complements traditional metrics. This approach helps build a better developer experience by providing a more nuanced understanding of what drives productivity.
Dan Tao, Head of Engineering for DevOps products at Atlassian, also emphasized this during his session at Atlassian presents: Unleash. He explained the importance of understanding the limitations of DORA metrics. For instance, setting a goal to increase deployment frequency, without considering the underlying challenges, can backfire. He shared a compelling example: a team that shifted from weekly to daily deployments without addressing the fragility of their deployment process ended up more stressed and less productive. Instead of pushing for higher deployment frequency, focusing on the stability and success rate of deployments would have been more beneficial.
In this example, Goodhart’s Law comes into play, which states that when a measure becomes a target, it ceases to be a good measure. By setting a goal to move deployment frequency by a certain amount, the manager mistook the measure for what they actually cared about: shipping software.
Dev teams can create scorecards in Compass for their software components to track performance using custom metrics. This flexibility enables teams to define healthy levels for metrics like deployment duration and deployment success rate, ensuring they focus on what truly matters and not just arbitrary targets.
Beyond AI – leverage internal developer platforms to improve knowledge sharing
No single tool, even those enhanced with AI, can address all the unique engineering problems faced by an organization. Therefore, it’s essential for developer platforms to be extensible, allowing developers to customize and address their specific challenges.
Andrew highlighted the need for platforms that empower developers to overcome specific obstacles, enhancing their productivity and contributing to a more efficient and effective development environment. Internal Developer Platforms (IDPs) should also enable developers to integrate and leverage GenAI tools effectively as they are significant data sources.
So, how do you implement an IDP within your organization? Here are some actionable tips:
- Reduce cognitive load by using a component catalog to store all necessary information about software components, such as team ownership, contact channels, and dependency details. This centralized approach fosters a culture of self-service, reducing the need for information requests and simplifying onboarding.
- Automate compliance with standards by implementing dynamic scorecards to communicate and enforce standards at scale. This integration into the daily workflow ensures developers are aware of governance expectations without attending additional meetings or searching for standards.
- Simplify infrastructure setup by using templates to streamline cloud infrastructure and tool configurations according to company standards. This reduces setup time and potential errors, allowing developers to focus on high-value tasks.
Pair tools and processes to improve developer experience
While AI tools are revolutionizing developer workflows by automating repetitive tasks and reducing technical debt, they are not a standalone solution. To fully harness their benefits, organizations must set realistic expectations, use phased rollouts, and combine qualitative and quantitative data. Leveraging internal developer platforms like Compass, combined with strategic use of AI tools, can further streamline processes, and ensure a more efficient and effective development environment.
Learn more about Compass and Atlassian Intelligence!