Understanding has always been the bottleneck. Sometimes AI helps with it like explaining things pretty well with diagrams. However, in general I agree that more code is being generated per developer and it's difficult to keep up with the phase of new changes and understand it.
We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation.
Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.
My main gripe is with Claude deciding to make 200 lines of code in a PR I need to review, instead of 3 lines of code somebody who understands the original algorithm/intent would do. And coworkers just YOLOing changes without understanding them. Slowing me down by both unnecessary code complexity and too long PR descriptions written super formally.
A PR with a minimal title and empty description should be refused at submission. If the human is so disinterested that they're using LLM generated code and then can't explain the purpose, that human should be prevent from making the PR. Working as a solo dev, it is very easy to be lazy like that, and I'm as guilty as anyone. Working in teams with actual reviews should absolutely have much more strict policies of what is considered a valid PR
This is the biggest issue I have with current state of affairs. It's not there yet. Because of that, extra work is needed to get them to work that otherwise would not need to be spent. Everyone is shouting from the roof tops about how great things are while suppressing these types of issues.
We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back
Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent.
I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.
The difference is an LLM can convert a stream of consciousness into well-formed prose for approximately free; I assume ‘provide some context’ means ‘brain dump’ in the OP
This has unfortunately not been my experience at all. Often LLMs miss or get wrong subtle details when I don't do the pre-work to organize my thoughts well ahead of time (at which point it's unclear how much value they're providing).
The PR descriptions are pretty universally disliked. We have centralized tooling that manages the prompts for that, I’m sure they’ve tried tuning it but maybe there’s more they could do.
Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.
It’s definitely possible to get much better output with prompting. I know, because when I’m faced with a “standard” PR description full of technical clutter, I can paste the link to Claude and ask “ELI5 what the problem actually is, any important context, what changed, and why that solves the problem.” And most of the time it converts it into something pretty good and readable.
The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags.
Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model.
(In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.)
LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.
understanding a different modality of model interaction gave me proper insight into the specific problem. In visual models, even if the model understands the concept of face, or hand, or whatever, it doesn't know how to de-dupe a statement like "count the number of faces" until you give it a countable reference frame, so it can internally, place a box around a face and give that a coordinate, and then it can collect all the coordinates, and suddenly it's counting face in a picture.
The same thing happens in code. Things we're happily shifting from context to context, the model itself isn't doing. When it reads file1 for the main() clause, it will easily read file2's main() clause as the same. It'll internally merge these.
So if you do want to work with these models to achieve complex tasks, you basically do have to go reverse centaur and bend the code base to it's blindness. You can't use the same function names across the code base; each one needs to be dstinguishable; same thing with variables that represent seperate entity relationships.
You do that, and it suddenly because a whole lot smarter.
It doesn’t matter, once you found the bottleneck there is a new one. Seems we changed the supposed bottleneck of writing code (as if it ever were, the world was producing far too much code before LLMs were even a thing) with about ten or so new ones, was it a good trade?
Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.
Understanding was always the bottleneck. The way LLMs speed up your work is by letting you get code without taking the time to understand it. If you want to understand your code, LLMs are a net loss.
If you want to move faster with LLMs, you need to act like a manager and stop caring about what the LLM did. You just need to do the manual testing and make sure it works.
For me the solution has been to throw away the code I don’t understand. I let the agent write the code, and if when I read it it seems unclear or needs a lot of explanation from the agent, I just throw it away and start over, or do it by myself.
While the tips are good to handle the volume, I still think this sets code owner on a dangerous path.
AI have limitation and hallucinate. Complex code will be explained in hallucinated way. At some point AI will be unable to write more because the arch has become too complex or the volume of code will be to high.
The article I would like to read would suggest how to force LLM to architect the code like a solid tower instead of a pile of unstable mud.
I feel if you still architect the code and guide the LLM, it will do a pretty good job. Maybe one day the LLM will be able to do all the system architecture but that’s probably still quite some time out. I don’t even know if that’s possible considering different business needs and other factors that aren’t technical.
I've been using Geoffrey's /explain-diff skill in my replace-github-with-tailor-fit-personal-software journey, and I'm liking it. I recommend at least giving it a try.
Understanding has always been the bottleneck. That's why LLMs aren't actually helpful: they speed up the part which is easy (typing characters into your editor), but are neutral or even harmful on the part which is hard (understanding the problem and how best to solve it).
It seems like humans have a limited "understanding budget" but LLMs force us to spend that understanding on waaaaay more code and projects than ever before.
LLMs usually points to the most idiotic future trajectory on my work, and I have to curse it inorder to let it keep up with my refined understanding.
But what else would one expect from a probabilistic weighted next token predictor, other than to conduct probabilistic search which are 99.99% deadends.
But LLMs can pave the way towards constructing resilient and correct architecture which can be iterated fast by a human.
Architecture and determinism is where my money is in.
Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.
They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.
We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back
That's pretty much what a PR description is.
I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.
But yeah, most probably don’t.
Have you tried writing in AGENTS.md or whatever to exactly explain what you like/dislike about the PR descriptions?
Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.
The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags.
Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model.
(In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.)
LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.
The same thing happens in code. Things we're happily shifting from context to context, the model itself isn't doing. When it reads file1 for the main() clause, it will easily read file2's main() clause as the same. It'll internally merge these.
So if you do want to work with these models to achieve complex tasks, you basically do have to go reverse centaur and bend the code base to it's blindness. You can't use the same function names across the code base; each one needs to be dstinguishable; same thing with variables that represent seperate entity relationships.
You do that, and it suddenly because a whole lot smarter.
Where is the bottleneck? WHERE?? Tell me! No evidence needed, just lay it on, man to man, thought-leader to thought-leader!
This is my new chat-up line at networking events.
Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.
If you want to move faster with LLMs, you need to act like a manager and stop caring about what the LLM did. You just need to do the manual testing and make sure it works.
AI have limitation and hallucinate. Complex code will be explained in hallucinated way. At some point AI will be unable to write more because the arch has become too complex or the volume of code will be to high.
The article I would like to read would suggest how to force LLM to architect the code like a solid tower instead of a pile of unstable mud.
About a year ago most people were still typing code. Having an agent do ALL code was crazy.
Within a year or two years at most, a lot of people will stop trying to understand code. The onus will shift to testing and QAing.
LLMs usually points to the most idiotic future trajectory on my work, and I have to curse it inorder to let it keep up with my refined understanding.
But what else would one expect from a probabilistic weighted next token predictor, other than to conduct probabilistic search which are 99.99% deadends.
But LLMs can pave the way towards constructing resilient and correct architecture which can be iterated fast by a human.
Architecture and determinism is where my money is in.