Maximising AI Benefits in Software Engineering: In-Depth Strategic Overview

Terence Chi-Shen Tao, a renowned Professor of Mathematics and evangelist for machine-assisted mathematics, posed this question: How should the mathematical community respond to artificial intelligence?
He explores this question directly, bypassing the hype debate, limited proprietary data, undisclosed costs, variables and incentives, lack of transparency, and the moving goalposts.

What surprised me most is realising how seamlessly his questions and conclusions translate to Software Engineering:

• How should the Tech and Software community respond to artificial intelligence?
  
• If AI becomes capable, to some degree, of autonomously creating software, what should the Tech and Software community value and prioritise?

Tao explores these questions assuming that AI will become capable, to varying degrees, of autonomously creating software (whereas the original question refers to: doing mathematical research work).

Applying Terence Tao’s thinking to Software Engineering is extremely interesting because nobody else in our community seems to be having this conversation. It is possible and meaningful because Computer Science and AI are branches of mathematics.
This is Tao’s presentation I am referring to (my thanks to Marco Abis for sharing this presentation with me):

      Mathematics in the age of AI
      Public lecture, International Congress of Mathematicians 2026
      Terence Tao
      University of California, Los Angeles
      July 24, 2026
      
      -  Video presentation: https://www.youtube.com/watch?v=M0--ZH1lOzg
      -  Slides: https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.pdf
      -  Paper: https://arxiv.org/pdf/2608.16753

Look at it, and:

Some of the challenges mentioned

Here are several challenges Terence Tao highlights, already translated for software engineering. Many tech organisations adopting AI to augment their SDLC will immediately recognise these issues. They illustrate just how naturally Tao’s mathematical insights map to software engineering, revealing how these problems stem directly from how AI models currently work:

• Shift from Scarcity to Overabundance: we are moving from a problem of scarcity of code created to an overabundance of AI-generated code
  
• Code Review Indigestion: this overabundance leads to developers’ fatigue and a waning interest in reviewing generated code
  
• Opaque Problem-Solving Process: the AI models’ problem-solving process is inherently different from how humans reason, think, solve problems, and communicate their solution, and it is also opaque. Because of it, developers struggle to understand the code generated and the questions, decisions, topics, and explanations handed over to them.
  
• Lack of Natural Friction: AI models present simple, straightforward points and extremely complicated issues in similar ways. By failing to signal which are the difficult and important parts, glossing over critical details of the complicated issues, while over-explaining the simple points
  
• …

Some of the solutions suggested

To overcome these challenges, Tao suggests several solutions, translated here for software engineering:

Code digestion:
Shift the emphasis from code generation toward code “digestion” intended as review, codification, and sharing of relevant system and domain learning and knowledge emerging from the generated code and the related decisions taken.

Evolving workflows:
Experiment with and develop new workflows and infrastructure to complement the traditional ones.

Preserving the Human aspect of work:
Focus on the human aspect of software engineering, particularly in education and training, where limiting the use of AI tools is critical to support the development of core skills.

Software Engineering goals, objectives, values worth pursuing

Tao identifies the above challenges and solutions related to the goal of solving open math problems. However, he highlights the need to run the same analysis across all the goals, objectives, and values of mathematics, after identifying them. Because:

• Diverging Goals: over-optimisation of AI around each individual goal of mathematics may cause the many (previously largely aligned) goals to diverge from each other.
  
• Misaligned Incentives: generative AI’s ungrounded nature combined with the commercial drivers of the AI companies makes the aggressive pursuit of one goal as a target can undermine the other goals and the broader mission.

He adds that the mathematicians’ community needs to come together to have open and honest discussions about both AI capability and our goals and values. This equally applies to the Tech and Software community. See the example below:

Proposed Rules for the AI Era

To protect the mathematical ecosystem, Tao suggests several ideas.
Below is my summary, ready to be translated for Software Engineering.

1. Mandatory Disclosure: Authors must transparently declare the use of AI.
2. Human Responsibility: Authors retain full responsibility for correctness.
3. The Expert-Understanding Gate: If an author cannot give a clear, expert-level explanation of the proof, it shouldn’t be accepted.
4. Reconsider Incentives: If AI can produce mathematics at enormous scale, the community should place less weight on raw publication volume and priority, and more on insight, understanding, exposition, synthesis, and lasting contribution.
5. Prioritise Proof Digestion: The community should value the work of understanding, simplifying, explaining, connecting, and contextualising AI-generated mathematics, not merely producing more proofs.
6. Canonicalization: Important results should be developed into natural formulations and proofs that can be incorporated into the broader mathematical knowledge base, rather than remaining isolated machine-generated artefacts.
7. Preserving Training Grounds: The community may need to declare certain problem sets “off-limits” to AI solvers so student mathematicians still have meaningful challenges on which to develop their skills.
8. Proactive Governance: Mathematicians must take the initiative to define acceptable tool usage rather than letting tech companies or external actors set the rules.

Conclusions

Terence Chi-Shen Tao’s insights translated to Software Engineering point to this broad conclusion:

Our Tech and Software community need to engage in open and honest discussions about our implicit and explicit goals, objectives and values, prioritise them, and define how AI capabilities should serve those priorities while helping our work and organisations.

Terence Chi-Shen Tao’s analysis on the use of AI for solving open math problems, translated to using AI for creating production-grade software, suggests overcoming these challenges listed below as the next step to improve AI’s benefits:

– Opaque Problem-Solving Process: the black-box nature of the AI’s problem-solving process makes AI-generated code and decisions difficult to understand

– Lack of Natural Friction: the indistinguishable way in which simple and complex questions, decisions, topics, and explanations are handed over to the developer makes them difficult to understand

– From Scarcity to Overabundance: we have shifted from a shortage of code created to an overabundance of AI-generated code

– Review Indigestion: this overproduction creates a new bottleneck and requires a new focus to overcome it

– AI-Augmented Development Methodology: an updated methodology is needed for discussing and thinking about evolving our workflows, practices and infrastructure

– Knowledge Codification & Sharing: the knowledge and learning from the hard-to-understand AI-generated code and solutions need to be captured and shared

– Education & Training: the overall impact of AI on the education and training of tech and software professionals must be evaluated, and the use of AI tools in education should be limited to support the development of core skills.

– Human Responsibility: defining the role and responsibility of the human developer, and what should not be delegated to AI.

These two stark realities make the work and conversation around these emerging key challenges difficult:
– Hype isn’t helping: the current discourse, driven by social media trends and vendor hype, fails to address this need.
– Incentives are misaligned: we cannot rely on AI companies or external actors to set the rules for us; their financial incentives and lack of transparency do not necessarily align with our priorities.

See Also

Post: The case for an AI-Augmented Development methodology

Ship Production-grade Software Faster with AI. No AI slop.

Drive Value Faster with AI. No AI bloat.

See how we can help across the full AI adoption lifecycle.

AI Services