Posts

AI capabilities are improving exponentially 1, with recent benchmarks showing models outperforming human developers. Yet, we still struggle to construct large-scale, high-quality software using AI. Attributing this solely to a lack of specific training (e.g., DevOps) overlooks the fundamental issue. The core challenge is alignment: ensuring the AI executes the user’s true intent. As task duration increases, alignment becomes increasingly fragile. A microscopic initial deviation results in a massive miss over a long distance.

In his "Bitter Lesson"1, 2025 Turing Award winner Richard Sutton argues that human-designed heuristics help when resources are scarce but hinder progress when resources become abundant. Methods that win at small scale often lose badly at large scale. The transformer architecture and scaling laws have made this pattern hard to ignore. Yet core SE techniques built on classical algorithms—fuzzing, synthesis, and static analysis—have not fully absorbed these lessons. They rarely assume access to truly large-scale compute.

Through the RLVR (Reinforcement Learning with Verifiable Reward) methodology, we have been able to create reasoning models with overwhelming performance in domains where automatic verification is possible. However, this is not a panacea. Depending on what rules are used to assign rewards, LLMs can exploit these rules, leading to "reward hacking." From a safety perspective, reward hacking manifests as follows: LLMs insert exception handling code literally everywhere in order to generate safe code.