How AI can—and can't—boost efficiency for programmers

McKinsey conducted an experiment with over 40 of its programmers, asking them to complete tasks of varying difficulty levels in a lab. They found that developers who used AI for a simple task, code documentation, finished their work in around half the time it took their peers who did not use AI. Artificial intelligence was far less effective at helping complete more complicated tasks, only saving developers about 10% of their time to finish their work compared with their colleagues who did not.

Another study, published by researchers at Microsoft, GitHub, and MIT, analyzed the impact of a particular AI tool (GitHub Copilot) on the productivity of freelance programmers. On average, programmers who had access to Copilot completed their work about 55% faster than those who did not.

However, not all of the workers benefited from AI assistance equally. The researchers found that less-experienced workers, older programmers (between 25 and 44), and those with heavier workloads were much more likely to benefit from using AI than their peers.

Despite AI's added value in expediting repetitive or mundane tasks, many programmers remain skeptical about its broader implications. JetBrains data indicates that 59% of developers have security concerns about generative AI, while 42% agree that its use presents ethical challenges.

Artificial intelligence tools can also be unreliable, suffering from the same confabulation problem as chatbots. AI's tendency to generate complications of facts that can be misleading or downright false might be why AI has yet to earn the trust of developers. Less than 3% of developers say they "highly trust" AI, while about 39% say they only "somewhat trust" AI, according to the StackOverflow survey.

Scores of new startups are trying to push the technology forward. Devin by Cognition Labs is one product piquing the interest of investors and engineers. Advertising itself as "the first AI software engineer," the company boasts that its software can beat other leading AI tools at meeting computer programming benchmarks and completing software engineering tasks with minimal supervision. However, the product hasn't launched yet, and because it relies on the same technology that powers ChatGPT, it could share many of the same weaknesses regarding complex problem-solving.

Despite suggestions that AI will trounce human capacity, software developers are generally optimistic about AI's impact on their work. According to the JetBrains survey, 60% of programmers believe AI coding tools will dramatically alter the job market for software development, while just over half think AI will boost demand for human programmers. Only a fraction, 13%, believe AI will completely replace human developers.

The technology behind many artificial intelligence tools is advancing at lightning speed. Though speculation on the future of AI abounds, there are still more questions than answers about what's next. For now, though, computers are augmenting human intelligence—not replacing it.

Story editing by Alizah Salario. Copy editing by Kristen Wegrzyn.