I have only one data point:
I am an embedded software developer. Recently, I had to attach and debug a stepper motor with a serial interface to an embedded control system. A bog standard task - basically, you need to initialize the motor, send it a home / referencing command, tell it to which position it has to go, and wait until it is there. Luckily for science, I had to do practically the same seven years ago with a lab system at a research institute. And in the current job, the senior engineer responsible for the motor interface is a heavy proponent of AI tools and uses these whenever he can.
Oh, and there are a few more pesky little differences:
- In the first job, I had to write the driver myself, as I had no working code, but some examples in Python and Java.
- I had only a partial protocol from the vendor and had to reverse-engineer important parts myself, as the vendor preferred to supply a closed-source windows driver.
- in the second job, we already had working though a bit old C++98 code written by somebody who had left the company, for an earlier iteration of that embedded system, which was in production for about 8 years.
- in the second case, the responsible engineer relied heavily on AI
- he told me that in multi-threaded C++ code, you don’t need locks for shared access, because the AI didn’t tell him. I had to educate him a bit about undefined behaviour in C++.
In both cases, the result had to be reliable, ad it was part of expensive and heavy machinery with high cost of failures.
The outcome? The task took less than four weeks in the first job, and over six months in the second job. In the first case, the result was very reliable. In the second, it is still not fully reliable.
You can point out that the second was a legacy system, which is more difficult to evolve. But that’s the point - AI does not “understand” legacy systems at all, and worse its use brings down and inhibits communication and knowledge transfer.
At best, you can conclude that AI is no substitute at all for a lack of knowledge and working institutional processes.
At best, you can conclude that AI is no substitute at all for a lack of knowledge and working institutional processes.
I think AI can stand in for a lack of knowledge and working institutional processes. I think they’re a pretty good match.
Have you tried DeepWiki?
It basically lets an agent loose on your codebase, and it documents the intricacies of your legacy code, and in my experience, it has been very accurate.
I don’t even vibe code, I just ask questions about my legacy codebase and I code manually.
Static code analysis existed before the “AI” era.
Of course, but this is auto documentation without necessarily relying on the programmer properly documenting their code, or even writing self-documenting code (both of which everyone should be doing anyway)
Holy shit does this resonate!
I spend hours each week hand-holding, manipulating, and sometimes fighting with LLMs. We are encouraged to use them at work, but it’s always a coin toss. Will it nail the task at hand on the first attempt? Or will it fall down a black hole of bad assumptions and getting stuck in stupid loops?
And it’s not due to ‘bad prompts’ as the kool aid drinkers frequently claim. I’ve had enough practice with them now, and I’ve had others review them and give a thumbs up.
‘Botsitting’… I like that. Very apt.
We are encouraged to use them at work, but it’s always a coin toss. Will it nail the task at hand on the first attempt?
A digital slot machine.
Addictive by design.
And moreover, it makes you believe that it did the work, while in reality you do all the difficult stuff and on top take extra responsibility for it. Like self-checkout supermarket counters.
