Rendered at 12:29:10 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
avinoth 11 hours ago [-]
This is like one of Google’s infrastructure blog. Google scale things are only relevant and appropriate for them. We had lots of bitter lessons from smaller engineering teams following the process with the justification that “this is how Google does it”.
Similarly, Anthropic can afford to do this because they have virtually unlimited tokens and labs to train models for their specific usecase. The rest of us need code comprehension lest we become dependent on continuous supply of tokens (which is a state these labs badly want to achieve)
So, unless you work at a lab, or have unlimited tokens guaranteed through the life of your company, it would do both the company and you good to continue to read the code and not let these 30k PRs per engineer be the norm.
An excerpt from the article makes my point
> Nobody on the team keeps a complete mental model of Claude Code in their head, and we don't think anyone needs to. When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today.
thr0w 36 minutes ago [-]
> they ask Claude to write an explainer for exactly that slice
If their Claude is the same as our Claude, then this is a joke. Claude’s writing style is insidiously mind numbing. It takes constant pushing to get it to explain things in digestible English.
panarky 32 minutes ago [-]
v5.5 eliminated much of the impenetrable prose of earlier versions.
ban_this_man 30 minutes ago [-]
[dead]
sadengineer22 15 minutes ago [-]
I have heard from someone who works at Anthropic that they will have postmortems that sometimes devolve into "my claude says X" vs "well my claude says Y". Maybe this is the future of software engineering, but it seems pretty fucking sad.
strken 3 minutes ago [-]
This is basically just being a manager in the before times. "My engineer said X!" "Well, mine said Y!"
Which is fine, whatever, I guess this is how I pay the mortgage off instead of having any clue what the hell Claude is actually writing. Hopefully I can semi-retire soon and go study dunnarts or take up welding or something else that's more rewarding than this nonsense.
thatsabadlook 11 hours ago [-]
Even then do we really have evidence that what they are doing is reasonable or practical for them at all? They are edging on like 3 9s uptime, what leaked of Claude code looked like soup, tons of security issues, they can't build a sandbox to save their company, and last I heard they aren't even profitable without adjusting their numbers for arbitrary things.
Think I'm good on taking advice from companies that aren't making money.
avinoth 11 hours ago [-]
Exactly. What they’re doing, repeatedly, is setting a narrative that this is how software engineering should be. Always be reliant on tokens. Don’t worry about uptime or errors - agents can address it, don’t worry about security - agents can address it, don’t worry about code comprehension - agents will always be there for you.
samuell 3 hours ago [-]
Nailed it
ban_this_man 1 hours ago [-]
[dead]
panarky 27 minutes ago [-]
If they're cash-flow positive with 75% of compute going to research and training, what do you think will happen to cash flow when they shift to 50-50 training-inference?
Because "pacing the frontier" is more about flipping on the money printer switch than it is about safety.
Rapzid 8 hours ago [-]
I would love to ask them why this scale is even necessary for a conceptually simple CLI app like Claude Code, but they wouldn't be able to answer because they don't even understand the scope of it apparently.
thatsabadlook 2 hours ago [-]
It's absolutely insane that they can't keep track of a CLI application that makes shell calls. This is so far from rocket science even mentioning the term is misleading.
ban_this_man 26 minutes ago [-]
[dead]
unddoch 3 hours ago [-]
It needs to gather a bunch of inputs and feed into the remote GPUs correctly and as efficiently as possible. Lots of room to optimize under the constraints that users love cheap tokens, demand a fair billing structure, and want responsiveness and low latency.
jbjbjbjb 3 hours ago [-]
I don’t think you can infer the scale for Claude Code alone. They said it doubled but not what the starting point was. The 30k figure is for the entire Anthropic monorepo.
tayo42 6 hours ago [-]
Is Claude code just the cli or does it include a bunch or backend services supporting?
scottyeager 6 hours ago [-]
It's also a desktop app, integrated into the mobile app, and it has cloud hosted session capabilities too. Plugs into Slack and Github, maybe others. So definitely there's backend stuff as well, beyond the inference.
ban_this_man 25 minutes ago [-]
[dead]
cheema33 6 hours ago [-]
> Nobody on the team keeps a complete mental model of Claude Code in their head.
For sufficiently large codebases in large organization that is how it has been with humans, who come and go. In many respects, What Anthropic is doing is a massive upgrade from the status quo.
ares623 5 hours ago [-]
Really? Hundreds of billions of dollars in debt is an improvement? The product that this code is supporting can't even pay for its debt.
Jesus. Give me 1 billion dollars and I'll keep your entire codebase in my head + personal notes for you until I die. Hell, I'll even force my descendants to do the same. Now _that's_ an improvement.
bossyTeacher 1 minutes ago [-]
[flagged]
pjmlp 1 minutes ago [-]
Do they still have humans left on the team? /s
mk12 15 hours ago [-]
I saw someone say the other day (in response to another blog post being well written) that Anthropic employees have a rule against using AI for writing/external comms? That seems clearly not the case here, normal people don't write like this:
> Most of the path is automated; a person's attention goes where the risk is
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
jazzyjackson 15 hours ago [-]
100%, although if people are getting 90% of their interaction from a robot, one may start to pick up some vocal tics and colloquialisms from machine world. a kind of robo-pidgin.
TeMPOraL 3 hours ago [-]
Irony being, robots are averaging our best speech patterns. So we're really just syncing humanity's speech patterns, with LLMs acting as proxy.
levkk 14 hours ago [-]
Yuuuup. Which I guess makes it even harder to detect what's AI and what's not.
cjkhowes 9 hours ago [-]
I've definitely picked up some weird claudeisms. I even know the em-dash keyboard shortcut now.
wffurr 27 minutes ago [-]
Macs make it really easy to type a certain few special characters -- em-dash among them. I'm typing on a Windows laptop and I know there's alt+numpad codes but this laptop doesn't have a numpad and I can never remember those. I used to know a few many years ago, but I've forgotten all of them.
Access to a wider array of special characters is also one of the really nice things about touch keyboards on mobile or tablet. Keyboards are great for fast, accurate typing but not for going beyond the basic character set.
nickspacek 4 hours ago [-]
I used to use semi-colons and parenthesis in my semi-formal writing, but now I find myself reaching for em-dash instead. I'm not sure if my tone has changed as well but LLM-writing has probably influenced me more than I realize.
wffurr 29 minutes ago [-]
If you spend all day reading Claude-written "explainers" of your own codebase, I can imagine the "house style" would rub off on you.
> That seems clearly not the case here, normal people don't write like this:
I think when you repeatedly expose and immerse yourself to such a degree in reading and writing language, it brushes off on oneself. So I could also absolutely believe someone from Anthropic wrote this.
jansport123 15 hours ago [-]
this definitely seemed like something claude generated, i know because i frequently do all my research in chat and then ask it to generate a doc based on what we talked about and when i read what it generates the verbosity, the style it writes in makes me delete the whole thing. I would never subject readers to that.
atty 12 hours ago [-]
It felt weird reading it because every once in awhile there were some well written sentences and coherent paragraphs, then it fell back into Claudisms.
> This is not a cryptographic break, new attack, or weakness in the ECDSA curves.
myth2018 12 hours ago [-]
As a non-native, I see how susceptible I am to absorbing such style. There's no way at all for me to perceive anything wrong with the examples you present
mk12 9 hours ago [-]
It's a bit more subtle than things like "it's not X, it's Y". And even that isn't "wrong", it's just the way AI talks, and therefore an indicator that the author offloaded their thinking to an agent. I've read so much Claude output that it's easy for me to recognize its voice, the same way I could recognize a friend's voice in writing. The examples I quoted are a weird kind of circuitous sentence structure with a certain shape and rhythm that scream Claude to me. Here is how I might rewrite them in human speak:
> Most of the path is automated; a person's attention goes where the risk is
--> We automate as much as possible, focusing human attention on risky parts
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
--> We run reviewer evals against every incident
But that's just my interpretation and my voice. I can't know for sure what the authors would have said if they took the time to write it themselves.
versteegen 3 hours ago [-]
Personally, a year or two ago writing like that did not really come across as particular bad or AI generated to me, although it was uncommon for humans to write phrases like that. Writing in an unusual way is not inherently bad, it can make your text interesting... but it is less direct and clear than alternatives. But the style becomes unpleasant if you keep seeing the same patterns over and over such as "X'ed against all of it"
wannabe44 9 hours ago [-]
It feels like it's trying to sell you something in every sentence.
userbinator 4 hours ago [-]
What came first, "Claudish" or those who trained it? I wonder if Anthropic's employees originally wrote like that. It's definitely irritating to read, and IMHO more so than the ESL-grade English which was quite common before AI.
MisterMunchkin 3 hours ago [-]
Claudeslop didn’t come from humans, it’s an artifact of running RL on slop outputs. If you keep feeding the slop output back into the model and then training on it, it amplifies its own unique weird vocabulary and behaviours.
damontal 9 hours ago [-]
The word “gate” was used.
15 hours ago [-]
ta93754829 1 hours ago [-]
load bearing comment
status_quo69 15 hours ago [-]
No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I really don't get the tendency here from Anthropic to describe what's most likely churn? A high pressure amount of changes/PRs a day on a TUI to wrap a while loop? I'm not going to be a reductionist here so I'm going to cut it off there, I'm positive that the complexity of handling multiple terminal emulators is something that needs some handling (even if doing their own rendering engine was a self made problem). Yes I've vibe coded my own harness from scratch without writing any of it, I still know how it works.
I'm also not sure that they addressed point 2 in any actual capacity? They say "the system verifies itself" which is cool and is something that automated testing has done for decades, but they eschew the mental model of how the system works as a whole and prefer instead to have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
N.B. Changed "1000 PRs" to "A high pressure amount" to be vague since the article is vague here on what CC undergoes.
jonahx 15 hours ago [-]
> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...
There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.
Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
patrickdavey 14 hours ago [-]
"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
daveguy 14 hours ago [-]
Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.
dmd 12 hours ago [-]
I'm guessing you've only ever worked at startups?
Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.
mqus 2 minutes ago [-]
Sure that happens, but over _years_. LLMs can make this happen over _days_.
hnedeotes 4 hours ago [-]
This is mostly a self made and kept problem. I also worked in codebases where the current developer(s) didn't know why things were the way they were and would take days to dig into why some part wasn't working - but the issue was, instead of taking the time to write tests and document, even if only when doing these point in time fixes, they just did the ad-hoc thing and solved whatever issue there was at the time without any worry whatsoever.
At some point I was tasked with improving the queries/retrieval times and started converting parts of the code and not only was it possible to write tests that would convey the actual meaning of the code and crystallise it but I started finding bugs in the implementation that no-one realized were even bugs because they no longer knew what it should do when queries grew to certain complexity and the search space was in the hundreds of millions of rows that thinking through them just with napkin calcs wasn't cutting it (specially when the bugs were subtle and the results still allowed the program to work "mostly" correctly).
hxugufjfjf 7 hours ago [-]
And this holds true for most of the Internet. There is no one person who truly understands how it all works: DNS, routing, and all the other things we rely on to get our systems to work.
leonidasrup 7 hours ago [-]
Network engineers and TCP/IP stack implementers have pretty good understanding of how the Internet works.
But I have see quite terrible finance software, patches over patches hold together by a duct tape. Also finance software has to implement large complexity of tax law, which is very large and always changing. It also doesn't help if the company developing finance software sees software development as a cost center and wants always to push the costs down (especially short term costs).
csallen 14 hours ago [-]
But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
evolve-maz 8 hours ago [-]
If anthropic is making as many PRs as stated in articles, then rolling back is unlikely an option. More likely it'd be a new commit to fix it, and with a feature flag to preserve the "wrong" behavior (even if wrong, since people will probably depend on that function since its original behaviour).
Those sorts of changes are really rough to manage. Maybe still manageable by LLMs but we'll find out.
myegorov 6 hours ago [-]
Is the Hugging Face incident from July really a problem? Or Australian healthcare portal breach? Who let their guard down unable to detect or roll back the change for days on end?
jdlshore 12 hours ago [-]
> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
csallen 11 hours ago [-]
> I see no reason to believe that people are going to be thoughtful about LLM-generated code.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
dieortin 13 hours ago [-]
> can roll back changes and investigate them
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
csallen 11 hours ago [-]
I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
what 7 hours ago [-]
> Just like people are more careful when carrying a baby than when carrying a pillow
lol, I assume you’re not a parent then. Or at best a new parent.
hxugufjfjf 7 hours ago [-]
What do you mean, exactly?
TeMPOraL 3 hours ago [-]
They're talking about operational benefits of having backup kids.
pofferfish 11 hours ago [-]
I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.
customguy 2 hours ago [-]
"their code" == code they actually write, that originates with them, "their system" == anything, from gentoo to something off-the-shelf
So the real question would be, how many people designing / implementing the boot process of any given system don't understand what what's going on.
marcindulak 5 hours ago [-]
Here is an explicit example of the additional lack of coherence due to LLMs in the Anthropic Claude Code public repo itself: the human maintainer trusts LLM's incorrect understanding of the current state of code in the repo.
> "Issues here are auto-closed by a bot when they've had no activity for a while and are missing information needed to reproduce (usually a version and steps, or a /feedback ID). Nothing is triaged solely by counting"
Therefore, I wonder whether the internal review process described in the Anthropic's blog for internal repos also results in such issues, and whether Anthropic has a internal way to measure and mitigate that. The blog post says:
> "When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today."
The Claude Code public repo problem shows that LLMs don't always "reflect the code as it is today", and can be a source of additional misunderstanding.
pofferfish 11 hours ago [-]
Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.
What you are saying is leaning on 2 premises that seem to not hold water
1. Code written by people is always more readable and maintainable than LLM code
2. Code written by an LLM is always less readable and maintainable
Can you really make those statements in all intellectual honesty?
SR2Z 14 hours ago [-]
I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.
antihipocrat 14 hours ago [-]
I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.
cmrdporcupine 12 hours ago [-]
> you know it's at least coherent enough that someone understood it
30 years working in this industry and I'd say that's the exception rather than the rule. When it comes to large systems.
monkpit 12 hours ago [-]
Yes, [citation needed]
aturek 8 hours ago [-]
> You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
I love this phrase, and I think it captures the difference between the "60 years of legacy code" mentioned elsewhere here, and the vibe-coded monstrosity that we all know we're going to be encountering in the next 5 years. The fact that there were human brains with intent making all the little (short-sighted, myopic, ignorant, etc) changes in the legacy code means that you DO have at least some shot of gaining enough context to make safer changes.
There is no real intent in LLMs and particularly in blind vibe coding, so you'll end up reading stuff that works, or appears to work, but there was no human brain behind the change for you to model and thus predict.
TeMPOraL 3 hours ago [-]
This just means we can't take the (often bad) shortcut of guessing what the code does because we feel we can predict what the author was thinking. Instead, we have to treat existing code as a natural sciences exercise - use the same methods we apply to the world around us, which has its logic and structure, but no human intent behind it.
peab 14 hours ago [-]
most large software companies are already too large for anyone to have a complete mental model of.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
tripleee 9 hours ago [-]
I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
skippyboxedhero 7 hours ago [-]
You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.
It is not completely useless as a tool but there is still a huge gap in most real-world applications (I work in an org with multiple thousands of devs and I am top 5 AI user consistently, I have used Claude Code since pre-release). If you are building something like an SQL database which has decades of logic tests built by humans then it can work. If you do not have this (i.e. 99.99% of the time) then you still need to be able to issue prompts that give context. And if you start without that context, it will produce shit code in extremely basic scenarios. In particular, humans are far better at producing code that is easy to change (in practice, you need to create unintuitive, non-human rules to govern this i.e. creating many artificial boundaries which would be completely unnecessary for humans).
I am not sure whether the models just capped out in terms of coding or whether they are just trained by people who have never worked on enterprise software but the gap isn't closing. It has just been filled by people who lack the knowledge to identify when they are being bullshitted (the prevalence of consultants suggests that this markets is large).
I have worked on legacy stuff before. In most cases, there is a kernel of reason and logic that can be recovered. I have seen pure AI projects and they are the worst grade of TCS-authored abandonware. They can often do one specific thing but are basically unmaintainable outside of that.
bpye 5 hours ago [-]
> You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.
I frequently find this in LLM generated comments, both when using an LLM myself or when reviewing code that someone else used an LLM for. I find it is often the most obvious indication that nobody really read through what was generated.
rwmj 2 hours ago [-]
We had one that hallucinated benchmarks, with graphs and everything! (The code didn't actually work.)
tored 4 hours ago [-]
Yes, it has become quite common that Claude’s text feedback, e.g. Claude found an issue in the existing code that requires a decision and Claude tries to explain it, it is a paragraph of bullshit words stringed together.
Then you investigate it and it is often something relatively simple that could have been explained with short sentence.
bhaak 6 hours ago [-]
I vibecoded a project without looking at the code as an experiment, and I had the same experience as you. It really drives home the point that, left to their own devices, LLMs produce abysmal code.
If you give them no constraints, they’ll produce code that just works, and it’ll look exactly like that.
We already know the solution to this problem. Software engineering. Put the LLM through the same grinder as any software dev and out constraints into place. Tell the LLM that the code design is bad and that it should improve it.
In my case, I had to go through several rounds of refactoring before the code at least resembled a good codebase.
A fun fact is that the refactorings didn’t break any functionality. Only adding new features broke preexisting behavior.
ithkuil 7 hours ago [-]
I'm trying to learn how to write software with agents where I keep control over the design while yielding control over the implementation.
This is not very different from what already happens to architects that delegate implementation to teams.
The challenge Is to find a way to encode the architecture in a way that both you and the agents can understand and agree on.
I'm still learning how to do this effectively, but my initial experience is quite positive. Not because agents are great but because humans are bad; the degree of coherence I've seen agents following is bigger than many human teams I've led in my career.
My benchmark is a quite big project for which the architecture is already well known, namely the reimplementation of the OpenVMS OS (kernel and user-land). There are many traps and many opportunities for the agents to just go ahead and diverge from the goals, so I found it a good playground toearn how reign them in but let them be effective at "drawing the rest of the owl"
majormajor 8 hours ago [-]
One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
jonahx 14 hours ago [-]
Right, that's what I alluded to in my last sentence.
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
dasil003 14 hours ago [-]
Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.
So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.
Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.
skippyboxedhero 7 hours ago [-]
One problem is managing expectations from non-technical people. You can ship faster but you are also producing tech debt at that same faster rate, and possibly even faster as the gaps in your initial implementation/prompts are filled by Claude deciding to do random shit.
In most orgs, there is also a gap between leadership, even tech leadership and people actually using LLMs every day. And it is challenging for actual devs who are being told that they should be going much faster and the reality which is more mixed. Pushing back on people who are convinced that Claude is God is going to be tough if you are a junior.
I don't feel the job has really changed, I don't write code as much but that time goes into working out the structure...which is what I was doing anyway. But it has changed the general environment, how work is planned, expectations, etc.
curt15 3 hours ago [-]
> There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug.
At this point, why even bother with source code? Source code is designed for humans to comprehend. If humans are no longer the intended audience, why not have machines output and maintain some more efficient intermediate representation of computer programs?
gieksosz 6 hours ago [-]
I fixed many bugs over the last years with help of claude code in an unfamiliar codebase. it is never just Claude alone so we don’t rely on AI understanding 100%. It is whatever AI can understand plus what I do. If you use AI right your understanding will also grow over time.
12 hours ago [-]
dimbletimbers 13 hours ago [-]
> But.... That's not knowledge?
Thank you! This is a huge sticking point for me. There is no knowing the code without experiencing the code. If nothing else, you have to read the tests to know they verify what you think they do. Even then, you might know it does what you want, but can’t know it doesn’t do anything it shouldn’t.
There are no shortcuts to true understanding.
giancarlostoro 11 hours ago [-]
When you get to the point that you are doing 1000 PRs I have to seriously wonder what even are you building, I hear about this a lot from a number of companies, but their “too many commits / too many PRs” problem never seems to give anyone from the outside the feeling they are innovating at groundbreaking speeds.
7 hours ago [-]
Terr_ 15 hours ago [-]
> wherein the machines are treating them like pets
I know some people will groan hearing the phrase again, but: "Reverse centaur." A centaur is a synergistic combination of two things, but now those things are in the wrong position: The human is the computer's tool.
Naturally, how "wrong" it is tends to depend on whether you ask the people who do all the work or the people who own all the computer-capital.
frereubu 15 hours ago [-]
My favourite quote at the moment: "May not man himself become a sort of parasite upon the machines? An affectionate machine-tickling aphid?" Samuel Butler, Erewhon, 1872
hgh 15 hours ago [-]
I think it's just a different focus on what matters.
There's clearly enormous (human) work that goes in to building and tuning their models.
The software they develop around their model is, relatively, trivial. It's not clear that enormous amounts of (human) energy need to go in to that code.
It seems largely a function of applying human energy towards the things that are marginally most important to generating you money and putting you in front of competitors.
xyzsparetimexyz 15 hours ago [-]
>The software they develop around their model is, relatively, trivial
30,000 changes a day is not my idea of trivial.
californical 14 hours ago [-]
I mean a harness around a model is pretty trivial, I’ve also built one (heavily relying on LLMs to build it).
With a few thousand lines of code over the course of a week or so of my limited evening energy, I have agentic loops, ide integration, and a limited selection of custom tools which are all very safe (no unintended changes), along with session management and awareness of git.
It works great, is customizable, and allows integration with any provider.
If they need 30k changes per day, it’s not due to the nature of the problem!
chicagobuss 7 hours ago [-]
I think claude code is just way, way, way more than "just a harness" at this point.
You have to remember all of the surrounding features it works with now, like the infra behind the cloud agents, or how remote control works, the mobile apps, their connectors, team skills and mcp servers, on and on....
xyzsparetimexyz 4 hours ago [-]
Do I need any of that? Pi works just fine
xyzsparetimexyz 14 hours ago [-]
Oh, I mean I can't disagree there. But they've dug themselves into a hole where they're implementing custom ray tracing in the game engine that claude code runs on or whatever.
californical 14 hours ago [-]
> implementing custom ray tracing in the game engine that claude code runs on
Haha, I don’t doubt it! Pretty damning overall for long-term maintainability of vibe-coded projects, even as the models are great at a lot of individual tasks
status_quo69 14 hours ago [-]
I mean, sure we can assume that (although the Anthropic article from the other day say that claude mostly builds Claude) but this is the Claude Code team which, I again am assuming so I could be wrong, focuses on the TUI/CLI interface to those models, not necessarily the models themselves. That coupled with the fact that this is how a large portion of their clientele interact with their moat, it would be odd to me if they didn't apply significant work towards this thing to keep the hype or productivity train moving
markbao 15 hours ago [-]
> I really don't get the tendency here from Anthropic to describe what's most likely churn? 1000 "PRs" a day on a TUI to wrap a while loop?
From the article, the 30,000 changes a month are in the monorepo, and Claude Code is in a separate repo. So it’s not the TUI that has 1,000 PRs a day.
I could easily see 1,000 PRs a day on the backend. Big tech was doing more than that pre-AI.
zeroonetwothree 10 hours ago [-]
When I worked at Facebook (pre LLM) we were around 50-100k per day. I imagine it's much higher now (quantity, probably not quality).
status_quo69 15 hours ago [-]
Fair enough, I did invent the 1000 a day because the article was vague, I'll edit my submission accordingly to also be vague
fouc 6 hours ago [-]
> No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I guess it wouldn't be surprising for people prone to AI psychosis to be attracted to working at the frontier labs, resulting companies that are heavily in the throes of mass AI psychosis.
Being tongue-in-cheek here.
j45 10 hours ago [-]
The Anthropic post the other day mentioned part of what helps Claude Code unlock some of this is the sheer ability to use more tokens than the rest to get something going and efficient.
tdhz77 12 hours ago [-]
This is a good answer
ls-a 14 hours ago [-]
[dead]
koehr 32 minutes ago [-]
It's funny to me that they use Claude Code as advertisement for their coding models. This software is mediocre at best, full of useless bloat and really only good enough for the use case (unless, the use case is advertising their models to vibe coders, then it's pretty good and I'm not the target audience)
_heimdall 22 minutes ago [-]
Its unclear to me how at least Anthropic hasn't been doing what amounts to recursive self improvement for a while now.
They heavily use Claude for both code generation and code review of Claude's own code and related tooling. Is that not functionally Claude running recursive loops to improve the next version of Claude?
TNWin 11 hours ago [-]
Anthropic has amazing models. But they should really stop commenting on software practices. Most of what they say seem out of touch with what's happening in software shops. A lot of it seems hype driven. And the entire message looks to be written by AI.
cheema33 6 hours ago [-]
> But they should really stop commenting on software practices.
I disagree. Many of us have an emotional attachment to the way we do things and have difficulty letting go. Anthropic is showing a different way of doing things. And I am not going to judge them too harshly if the final output is good. I use Claude Code every day. And it is good. Ahead of the pack.
dimator 5 hours ago [-]
You're failing to differentiate what Claude Code means here. The models, no one argues that they're not good. If you mean Claude code the local agent, there are now dozens of smaller, faster, lighter harnesses that do everything the Claude tui does and more.
Yes, this is a different way of doing things, and it's clearly working for them, but there is no direct line between that and a "ahead of the pack" product.
qnleigh 5 hours ago [-]
Given the internal models that they have access to, I'm more worried about software companies being out of touch with the next generation of models will enable. Anecdotally I've seen that it takes a while for an organization to adjust to new LLM capabilities, even if everyone had access.
pokstad 9 hours ago [-]
It’s all marketing nonsense. Along with the “AI is dangerous” and “we’re all going to lose our jobs” messaging they do, it’s just to sell the services.
koolba 16 hours ago [-]
> Low - Docs, copy, internal-only features, additive changes behind a flag
> None required once automated review and security checks are green
How anybody can let an LLM write docs and not even read them is pure insanity.
brational 11 hours ago [-]
It’s writing the docs for other agents obviously
movedx01 3 hours ago [-]
> We don't have a formal process for this. Nobody on the team keeps a complete mental model of Claude Code in their head, and we don't think anyone needs to.
Nobody on the team keeps the mental model of Claude Code as it is today in their head, so nobody on the team is able to reason and decide about Claude Code in the future, is that right? So no long term architecture strategy can exist as no current architecture understanding exists. While it seemingly does work for a system which is a TUI, would the same be true for a complex distributed system today?
baq 3 hours ago [-]
'Nobody has a complete mental model of <insert name of system>' has very much been the norm since the invention of the two week's notice. What's actually new is that an architecture strategy can now be cheaply reverse engineered out of any system, adjusted where needed and immediately applied to the development process in days or hours since 'the team' is now a single dude or dudette with a fleet of robots which can see the whole code base together with its durable arch docs at once.
stuaxo 3 hours ago [-]
It doesn't do all that much - so nobody being able to fit it in their head is not a good sign.
vb-8448 14 hours ago [-]
No human can stay on par with 30k pull requests per day ... they simply don't have control on what's going on and rely on AI itself to do the job.
PS: I really want to know what is the churn rate (and the cost per LOC) here. My gut is saying the cost per LOC is insane, and they are basically rewriting over and over the same code.
cush 13 hours ago [-]
30k PRs is such a meaningless figure though, you can't make anything of it. It entirely depends what's living in that repo. Adding localized strings, configs, things that change frequently, can easily move numbers like this by orders of magnitude. How big are the PRs? Maybe there's an auto-bug-fixer agent that runs around the codebase making thousands of single-line fixes per day. Who knows. The number means nothing in isolation. We used to scoff at bad managers for measuring the performance of engineers using lines of code, now we're using the same figures to judge AI. We've become the bad managers.
vb-8448 3 hours ago [-]
Actually LOC is a decent measurement, it becomes a bad measurement when you use it in the wrong way, like any other measurement.
Agree that 30k PR alone doesn't mean much, but there aren't other details on what's inside. Based on I saw on how LLM programming works, it's hard to believe that there are single-line fixes. If I have to bet I'd bet on 200 LOC on average, that is 6M/day. Even if only a small portion reach production, it's still insane.
grey-area 16 hours ago [-]
At 30k commits a day, is there anything that can meaningfully be called review?
stillpointlab 16 hours ago [-]
I'm thinking about high volume mass manufacturing as an example here. I like those "How it is made" type YouTube videos. You see a production line that has some kind of QA process where cameras + automated systems do spot checks at high speeds and these crazy actuators are punching out bad examples. These processes often also involve spot checks where some statistical sample is selected and viewed carefully.
I don't think we've gotten to that level of reliable review with code, but I don't see any reason we can't continue moving in that direction.
fipar 15 hours ago [-]
I really like those videos too, but a very big difference is, in manufacturing QA, the system outputs a very high number of copies of a very small number of possible types, so spot checks work fine from an operations research point of view, as the variance in output has a known upper bound.
For code, I think things go in the opposite direction: the system outputs artifacts that are all different among themselves (even more so now that we no longer distribute software via physical copies. Well, bits are always physical, but I mean we no longer ship floppies/CDs/DVDs).
That said, at least in my experience, code review has never been a direct part of factory-line-type QA. That QA is typically done by treating software as a black box and just verifying that it complies with a spec (often times based on a very loose definition of "spec" ...), and this is something that works equally well regardless of whether a human or a machine wrote the software. The type of QA that is done (among other things) by code reviews is checking whether the software's design is such that it will be economically feasible to maintain and extend it, for example.
The one part the code reviews takes a part of (but is in no way error-free) and that you just can't do with black-box testing (or it is much harder to do) is to look for things like backdoors. As our industry moves more towards "agents write software and tests validate that it works," I think a lot about "Reflections on Trusting Trust" and the new and wide attack surface added by people no longer reading the code shipped on their behalf.
grey-area 16 hours ago [-]
That’s a category error. Software isn’t stamping out the same part again and again; if it were review would be simple. Analogies often mislead.
Of course Anthropic would love you to think you are heading towards that future so we have to bear in mind their livelihood depends on LLMs being the future of work and 30k changes a day being normal.
But why would we want 30k changes a day in a given product, what possible justification could there be?
stillpointlab 15 hours ago [-]
I understand the skepticism and I can't claim I am right. But some points come to mind.
First, I think we overestimate how different code is from other code. We even joke about CRUD apps and how cookie-cutter they are. A lot of changes in those apps are: add some new features to a data access layer (SQL, ORM, whatever), add new routes, add new UI surface, etc. There is an entire industry around "Design Patterns" and people live by the DRY axioms. Things like Ruby on Rails exists because rebuilding these systems over and over is wasteful. I feel we are just moving further down that same path.
Second, your question of "why would we want" is totally backwards. Progress doesn't follow your desires. Progress doesn't require justification. Sometimes we adapt the environment to our needs, other times we adapt to a changing environment. Which case we are currently in is still undecided and you are free to fight for the future you prefer. But demanding an industry justify itself before it is allowed to change fails in the face of evolution. If it works, we will justify it in hindsight.
g-b-r 14 hours ago [-]
The second paragraph hardly makes any sense, was it the output of an LLM?
stillpointlab.com , "Get more from your LLM"
jbvlkt 14 hours ago [-]
Exactly. 30k changes a day look like brute force. No one wants that except the one charging for generating those changes.
7 hours ago [-]
0c3ca83 15 hours ago [-]
Those manufacturing videos are of someone making the exact same thing over and over. Months and years of testing go into ensuring that there is no change in the system, and continuous monitoring is put in place so that if any changes slip in, it can be undone. The design is set in stone, as it should be, and variations are bugs. There are months to years of cheap prototypes, iteration, design changes, pipeline tuning, and verification to ensure that nothing will need to change for the next million items.
If you want a software analogy to those devices getting churned off the assembly lines, it's the TCP streams coming out of your network card, with crazy checksums punching out bad packets.
We're orders of magnitude better at reliability with code, but we're regressing on that metric rather quickly. And the comment above deeply misunderstands both manufacturing and software.
cadamsdotcom 12 hours ago [-]
The repetitive piece - the piece that needs that process-control - is moving from shipping code to its creation.
That's new. All analogies to 2022 software engineering need to be thrown out.
Software construction is now more like an industrial process, with the artisanal work shifting towards both
1. aligning what's built with what's needed (talking to customers!)
as well as
2. implementing and tuning your software construction process (ie. hi - we're Anthropic. We're burying ourselves in a mountain of code but it's okay because the fix is a second mountain of code, try some tokens the first hit's free!)
Back to my original point...
Writing code is now on the production line; making quality output demands industrial grade/scale verification (some involving humans of course).. which is a fascinating engineering challenge, and it's brand new!
0c3ca83 12 hours ago [-]
Sure, I'd you want the exact same code created every time, I can automate that with cp. It's perfectly reliable, and checksums can catch the few times it fails.
Verification means nothing for industrial processes when the desired outcome is a moving target, so the first part of an industrial process is to develop a fixed target and aggressively remove variation.
Moving the target around is a research problem, before you feed the result into an industrial process.
Software solved industrialization so well we even forgot that we go from research to industrial with `npm` deployments or whatever your package manager of choice is.
cadamsdotcom 9 hours ago [-]
You should go learn about test driven development and test first development in general.
It absolutely can be industrialized: the agent writes tests that conform to your requirements; the harness makes sure those tests pass before work can stop.
Have a look at https://codeleash.dev/docs/tdd-guard for my (dated but still interesting) take on this. Claude Code hooks are really useful to do the enforcement.
It's open source so no need to read my crappy generated docs - you can point your agent at the repo and ask it questions.
grey-area 7 hours ago [-]
Stop trying to use analogies to reason. Analogies are for persuasion and colour, not for thinking.
The process of anthropic, turning gold into shit software, is hardly something to emulate. What is the evidence software should now be produced in an industrial way? Because arthropic (the ones selling you the shovels) says so?
jeremyjh 15 hours ago [-]
Those machines are not doing design work. They automate production of designs that had long development and testing lifecycles. Code is design. The compiler is the automated production process.
cush 13 hours ago [-]
Honestly systems like this resemble real engineering more than manual human review does
swaptr 15 hours ago [-]
I dont believe it, its FOMO marketing to me.
mirekrusin 15 hours ago [-]
It's conducting/steering/oversight/supervision, not reviewing.
dieortin 13 hours ago [-]
> One colleague said slowing down does lower your error rate, but mostly because you're doing less of everything
That’s not how rates work
dozerly 11 hours ago [-]
They aren’t claiming that the doing less by itself changes the rate in this sentence. They are saying that the rate changes when you slow down, probably due to having more time to focus, which in my opinion is likely.
Rapzid 7 hours ago [-]
Haha, yeah.. Also with most real world, vs LLM FOMO hype fantasy land, consumer software has two very real tiers of bugs; regressions, and bugs in new features.
Regressions are usually the absolute worst.
Up to now we've just been eating shit on the DX side as far as regressions in Codex and Claude Code because everything is all "go-go-go OMG I don't want to fall behind" and also.. Access to new models is tied to harness updates.
But a ton of industries have users that are not so willing to eat the regressions in the name of more features and changes than they can keep up with.
Edit: Also moving too fast.. You may have been less likely to "get it right" than ever before leading to design churn. Again, depending on industry and users, they may not be very tolerant to releasing the 5th complete overhaul of some feature in so many weeks..
cebert 41 minutes ago [-]
Anthropic is doing some fascinating work. However, I find their communication style frustrating. It’s clear to me their employees have near unlimited token budgets. When Anthropic team members share their internal processes, I feel they don’t appreciate the constraints that many of us face in our work with budgets.
RGS1811 16 hours ago [-]
How is the ability of a human to understand the code via hands-on familiarity not a direct dependency of their ability to do manual code review? I don't get it. Having claude write an explainer (something I do a lot!) doesn't get you to the point where you can confidently assess a complex diff. And you've funneled your priors on what the diff says and does through the same system that already reviewed and approved it. This article isn't bad but it doesn't give enough weight to the human context-building part of the problem.
mahogany 11 hours ago [-]
I think there is no magic trick here. The answer is simply that humans will no longer understand the code.
Although, following Anthropic’s ethos and taking this to its conclusion, I’m not even sure why they need human reviewers at all. What, they bring in a human to look at a diff of an “important” section of code that was written by agents and has changed a dozen times since last week through 12 PRs of churn? Obviously they don’t know what’s going on. So what’s a human to do? Do what the other approvers (if any) did: fire up a Claude to command it “verify this change has no regression; check carefully for edge cases”, then come back after a half hour of High Thinking to see “Change is clean (with some non-blocking findings below, in order of significance)” and click Approve. Not sure what the purpose of the human is in such a process.
jazzyjackson 15 hours ago [-]
Yes it used to be said, the code is not the asset, the people who know what the code does and how to change it are the asset.
Now we’re just going to outsource our capacity to design and review because it’s faster that way, as if the thing that was wrong with the software industry was that releases weren’t being pushed out often enough.
YmFzZTY0ZGFkZHk 16 hours ago [-]
And so the wildly unprofitable AI company explains how it token maxes it's code and review process.
Authors should really try and view this article from the perspective of someone who has to think about cost to use the models.
evolve-maz 7 hours ago [-]
I look at it as a giant experiment. Any research tends to be very investment heavy, and people are given a lot more leeway to try things which are wildly unprofitable.
I'm glad they're doing this, because it lets me see an extreme of LLM development.
markbao 15 hours ago [-]
We spend a few thousand dollars a month in AI code review costs.
Compare that to the cost of human code review which can take up a third of an engineer’s time. You’re looking at $50-100K a year per engineer fully loaded in code review costs.
If AI assists in code review (not even replaces – we still do human review ourselves), you could spend thousands a month per engineer and still come out ahead.
Ironic that this comment talks about cost without weighing the numbers.
mohsinakram 1 hours ago [-]
[flagged]
enraged_camel 16 hours ago [-]
[flagged]
tedd4u 15 hours ago [-]
"Anthropic told investors that it will profit on a measure called adjusted operating income.
"According to the report, adjusted operating income excludes stock-based compensation, which is not unusual for technology companies. However, the report also quotes its gross margins - above 80% - exclude revenue shared with distribution partners, as well as the cost of training its models."
I wonder how they account for free credits given in this spend. We got given US$5k free API credits from them a couple months ago and even accounting for our Claude Code subscription spend, I think we're still ahead overall - they haven't made any money from us! And they just gave us another few hundred bucks a month of API credit as part of our subscription.
15 hours ago [-]
rhdunn 16 hours ago [-]
How so? Can you link to the numbers?
Antropic's revenue is up [1], [2], [3] and if you calculate adjusted operating income (by excluding most expenses like training costs) then by that measure it is profitable. However, if you factor in all the numbers it is operating at a loss.
Source? Only reports I’ve seen was they were profitable for one month in last quarter due to xAI discount of renting out colossus . Be interested to learn otherwise.
lbrito 14 hours ago [-]
That's factually incorrect, and shouldn't be allowed here.
After investments. Investments they can cut if they need to. Their daily operation is very profitable, they just decided to use that to bring cash in and go on the crazy growing trajectory, instead of taking it slow and steady. But if AI progress suddenly stops, they can just scale investments back and became profitable immediately. Calling them unprofitable is highly misleading. It's not like they are an early venture company still trying to figure out their product-market fit, with totally inflated numbers based merely on a theoretical number of potential customers (while having close to zero paying ones).
YmFzZTY0ZGFkZHk 13 hours ago [-]
I'm not arguing that their product is not valuable or that it doesn't have customers.
I'm stating that their current business model, prices, and behavior they are trying to instill is not profitable. For example, token maxing every code change is great for Anthropic and terrible advise for Anthropic customers.
The moment they stop training, they become irrelevant. The moment they become irrelevant their valuation crashes down to normal valuation ranges for a company with their revenue and future growth
surgical_fire 16 hours ago [-]
We're talking about being actually profitable, not fake profitable.
tracerbulletx 15 hours ago [-]
“I think that, every time you see the word EBITDA, you should substitute the words "bullshit earnings.”
― Charles T. Munger
And what they're reporting is adjusted even further than EBITDA
adammarples 16 hours ago [-]
Yes when you subtract expenses and capex
throwaway613746 13 hours ago [-]
[dead]
misschresser 2 hours ago [-]
This was an exhausting post. both the writing style and the thought that this is what bleeding edge currently looks like.
Some of the most capable engineers in the world high on their own supply… I have no doubt they think this is aspirational but even as I know LLMs will forever change how we code, I’m not entirely sure 30,000 PRs for a coding agent is a volume that makes any sense.
mstaoru 15 hours ago [-]
Before all came to this, I've been promoting reviews at $dayjob not as gatekeeping but as method to exchange information and build more team spirit. We are an internal service team working by "projects", which makes us a very disjointed team with barely any communication. (Part of it is also German work culture.)
And now this. Suffice to say, my efforts did not bear fruit. I feel more isolated than ever before.
gaal_dornick 12 hours ago [-]
From the article: "merge with at most one human in the loop, who can be the author". Curious to see how this handles enterprise compliance requirements and insider threats?
- How was this approved for SOC2 compliance? Our SOC2 reviewer explicitly mentioned that all code needs to reviewed by at least one other human before it goes to production. Claude Code says they are SOC2 compliant (https://code.claude.com/docs/en/security).
- This assumes all engineers will not do anything malicious. What is preventing a Claude engineer from spying on their ex's or some other companies Claude searches by pushing malicious code? Recently an OpenAI engineer was caught planting a recording device at their own office.
llama052 12 hours ago [-]
I’m sure it’s not compliant. We have the same requirement for soc2. However the audits are pretty light and a company big enough can afford to have an auditor who looks past those things.
c0rruptbytes 10 hours ago [-]
i think SOC2 is about audibility more than anything - my guess - and i’m not an internal auditor or anything - is soc2 needs auditing for human changes, but a machine is just a “system behaving on its own”
vatsachak 15 hours ago [-]
Mfw tui needs a billion prs per day.
Opus 5.5 is obviously stellar but come on what are you doing lol
pokstad 10 hours ago [-]
They need to add tamagotchi pets to the TUI so you stay awake while approving commands.
Rapzid 6 hours ago [-]
Staff software engineers(because all the seniors and below have already been sent to work at the soylent green factories) will have AI monitoring software installed on their computers.
If it detects they aren't paying attention, fall asleep, or haven't approved enough PRs per hour; off the work at the factory.
vavkamil 17 hours ago [-]
> Security gets its own automated review too. ... You either fix it or close the thread with a short reason, which is how you clear a false positive.
I hope this is only for low-risk false positives. Dismissing a critical finding as a false positive should still involve AppSec.
You could improve revert rates while simultaneously introducing more security vulnerabilities.
And the linked "claude-code-security-review" explicitly states in the README
> This action is not hardened against prompt injection attacks
That's a significant limitation for an AI security review control, no?
maxgashkov 13 hours ago [-]
I'd be more interested to read on what 10% of the code is still getting written by humans and how it doesn't become a bottleneck at the pace they described.
lifeisloving 13 hours ago [-]
Important/hard things are usually bottlenecks, its a good thing and usually by design. People should have more self imposed bottlenecks.
What are llm freaks in such a hurry for? Its like everyone is manically racing towards a finish line without realizing the finish line is you out of a job. Its like everyone is excited to be completely obsolete and poor.
maxgashkov 12 hours ago [-]
My interest is mostly from the management perspective -- in 2025 I have seen a prime example when a heavily vibe-coding team chugged out half-baked products that other teams later had to deal with and this was not pretty and created all sorts of interpersonal issues; the same I fear may be happening here as well.
I'm looking at Claude Code and they chug releases daily and sometimes more often then this, if someone has to stop and code a critical path by hand or smth I'm wondering how this is handled.
--
On a separate topic, the "llm freaks" -- I know you intended this as an insult, but I'm actually proud to be called a freak again, and I thought this is the place where freaks are discussing things.
lifeisloving 11 hours ago [-]
I didnt really mean it towards you, I meant it more generally but I can see how it came off that way.
While Im rooting for Anthropic to collapse under the weight of slop and their own contradictions. There's a leaked interview for Anthropic from earlier this year (I think) for their inference team. The interviewer seemed very competent and the quality of the questions were good.
This article is more likely a marketing gimmick in service of shelling shovels (for devs to dig their own grave) "hey look guys we automate everything with claude, you can too, just give us all your money!"
visarga 16 hours ago [-]
All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.
dominotw 15 hours ago [-]
i am tired of listening about other ppls agent setups. Everyone thinks they've discovered something cool and unique but They are all the same.
Its like telling other ppl about your dreams. Not interesting.
kaashif 14 hours ago [-]
I used to keep a dream journal because I thought it might be interesting to recall all of my dreams.
Eventually I did, but it was hours and hours of effectively random stuff happening that I had to write down or dictate with speech to text.
It was a waste of time and I'm now glad I forget almost all of my dreams. My point is even my own dreams are uninteresting.
I'm not that interested in my own agent setup either - usually the tools become better on their own after a few months and render it all obsolete.
insin 6 hours ago [-]
Sir, have you heard of our lord and saviour, Markdown?
nzjrs 10 hours ago [-]
So true.
acedTrex 15 hours ago [-]
For real, my eyes completely glaze over, there is nothing interesting in it.
zeroonetwothree 10 hours ago [-]
Much like the output of agents themselves
crtk 5 hours ago [-]
Does anyone ever read what code these models produce? 50% or more is just crap: meaningless tests, convoluted code, a lot of indirection, unreadable with a lot of bugs. I am not surprised that they merge as much as they do, they just dont know what the code does but see it is somewhat working. Most people who are impressed by it, I am sure, are just shit coders themselves ngl
chrisjj 4 hours ago [-]
> Most people who are impressed by it, I am sure, are just shit coders themselves
The less you understand code, the more you're impressed by coding chatbots - in part because you're less likely to read the bot's code, and if you do, less likely to realise how poor it is. Code gets judged good simply because it exists and costed little to create.
dylanhouli 17 hours ago [-]
Here I am still unable to stop myself from doing line-by-line code review. I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) by doing it that I feel it would be careless not to.
stillpointlab 16 hours ago [-]
My advice is to start a totally vibe coded side project. You will never gain confidence until you experiment, find the failure cases, learn to deal with them, etc.
Choose something you want to do but don't really care that much about. Be more ambitious than you think you could be, have a high bar on the outcome but purposefully choose to completely ignore the actual code. Make it write tests, dashboards, monitoring - whatever you need other than code to feel confident.
Now is the cheapest time it will ever be to experiment.
72deluxe 13 hours ago [-]
I did this, but now I have a useful piece of software that I have zero clue regarding how it works. My only chance of "maintaining" it is giving it to another LLM to explain.
I am happy that I have a useful tool, but it simply isn't maintainable without a machine or me investing much time to understand it.
I don't see how the former is a sensible future for humanity. I am happy to say I won't be releasing this software I've created (I will find it useful myself), but deferring all understanding to a black box seems a poor future for brain activity.
stillpointlab 12 hours ago [-]
One thing I do is ask the LLM to write me a walkthrough of the code. LLMs are pretty good at generating documentation like HTML slideshows or whatever else.
I mean, I've had useful software that I don't understand how it works where I am stuck hoping some small company continues to maintain it. And I've had some of those companies go out of business and leave me with useless software that I eventually have to replace once it no longer works on my machine.
Having the code that I can dump into an LLM to understand myself or have it alter it seems better than that.
chrisjj 4 hours ago [-]
> One thing I do is ask the LLM to write me a walkthrough of the code.
I tried that. ChatGPT's result was laden with loud, clear and wrong fabications which I might have believed had I not examined the code.
Unreliable program be unreliable.
arcanemachiner 11 hours ago [-]
Mermaid is a great output format for this type of information. Flowcharts, sequence diagrams, etc.
zeroonetwothree 10 hours ago [-]
I've been working on a large (~10MLOC) mostly vibe-coded project for months now. It's actually quite frustrating:
* The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself. Yes, I can ask it to look elsewhere for similar bugs, but even that's not super reliable.
* The user-facing text it generates is just as awkward as ever. I have to rewrite all of it, and sometimes it slips through. It's also hard to write text when you don't know what the code is actually doing.
* It doesn't have a good model of what kind of information is relevant to humans. It's just as likely to produce a paragraph of random jargon as a clear explanation of what's actually going on. It shows excessive detail for things that don't matter, and not enough detail for things that do. For example, it doesn't understand that $0 can logically mean "null" in terms of how humans think about things (e.g. "This item is free" vs. "This item is $0").
I kind of wish I hadn't gone down this path but now it's far too late, I can't really manually maintain this amount of slop. I'm quite sad if this is going to be the end state of this profession.
chrisjj 4 hours ago [-]
> The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself.
I see the same in e.g. simple CSS. I ask ChatGPT to apply styling X to aĺl instances of say a UI control type. Then to another. Later I ask for a tweak of X. Bot updates some but not all instances. It didn't create a single class for all. No intelligence seen.
The most remarkable fails are simpler though. E.g. at version 303 I ask "Revert to version 299". I get 299 with 303's edits. Bot explains it interpreted "Revert" to mean something no intelligent programmer would.
And most importantly, these are just the ones I directly notice. To see the total, I have to read through and understand the most repulsive code I've seen in my life.
The only way to be satisfied with bot code is by having exceptionally low expectations. And never ever let yourself be kidded that your helper is more than a stochastic parrot gorged on stackoverflow etc.
mattm 14 hours ago [-]
I have done this. It's exactly the reason I went back to doing code review. I don't look at the code with a fine toothed comb, but it does help me catch bad patterns early.
stillpointlab 14 hours ago [-]
I guess the question is: did the project you worked on fail? As in, you couldn't achieve the outcome you were trying to achieve? Or at some point did you sneak a look at the code and recoil in revulsion? What was the failure state?
Was this with the latest models (Fable/Opus 5.5 or equivalent) or an older model in a 3rd party harness?
Are you in the camp of: "if at first you don't succeed, never ever try again".
hn_acct-1029393 14 hours ago [-]
Not parent, but project I was involved in was shipped very quickly, had many bugs, and burned a lot of customer trust.
This was a few months ago, so obviously not with the latest models, but:
1. That argument is evergreen. You will always be able to make it, and it doesn’t really add much. It is impossible to refute.
2. Gone back to - when writing code - having a strict mental modal and good control over the codebase. A mix of hand coding (to establish the mental modal) and prompting (for boilerplate). It’s not quite as fast, but the quality is much higher when you mix in using LLMs in quality enhancing capacities (review, debug, etc). This is with the latest model. Hands off still atrophies over time (even the boilerplate), and in some cases is very inefficient (as in wasting my time, nothing to be said about tokens).
> Are you in the camp of: "if at first you don't succeed, never ever try again".
This is great in some domains, but “don’t make the same mistake twice” is much more important in mine. I don’t have the luxury to ship low quality, especially when LLMs make even higher quality so much easier.
stillpointlab 13 hours ago [-]
I think people are moving the goal post here. My post suggested starting a vibe coded side project, one where failure is low stakes and has no consequences. You are talking about shipping a production system to customers.
If you cannot tolerate failure then don't experiment. Don't mix the two.
xboxnolifes 13 hours ago [-]
I've done this, and always get subpar results. I cant get the LLM to make the minor tweaks I want to see, and now I have no mental model of the code to make the fix myself. Sometimes they arent minor things.
xyzsparetimexyz 14 hours ago [-]
Ridiculous suggestion. If someone already takes issue with ai generated code (as well they should, it often is _still_ pretty bad), getting them to write more of it is useless.
Additionally, do you really believe that 'Now is the cheapest time it will ever be to experiment'? Why so? Because of token prices? Not a good omen if so.
stillpointlab 14 hours ago [-]
I'm not sure what is ridiculous about suggesting people take a stance of openness and curiousness when confronted with new situations. I genuinely don't understand what people have to lose by building some throw-away experiments without reading the code.
I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
xyzsparetimexyz 14 hours ago [-]
>I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
God I hope you're right. Maybe then this era of vibe slopping out photoshop clones that'll be abandoned within a week (while advertising them all over social media) will come to a close.
chrisjj 4 hours ago [-]
> 'Now is the cheapest time it will ever be to experiment'?
First hit is free ;)
troupo 16 hours ago [-]
[flagged]
dang 16 hours ago [-]
Please don't do this here.
(You're welcome to make your substantive points thoughtfully, of course.)
Have you reviewed his posting history? He's may be welcome to, but he's incapable of.
He sure is smugly impressed with himself for giving you a taste of your own medicine, though. He knows you are but what is he? ;)
xyzsparetimexyz 14 hours ago [-]
[flagged]
stillpointlab 16 hours ago [-]
Issue: I am still unable to stop myself from doing line-by-line code review
Response: Try experimenting in a low stakes environment
From Google: "Exposure therapy is a psychological treatment designed to help people confront their fears, break cycles of avoidance, and reduce anxiety."
g-b-r 16 hours ago [-]
The thing is that's not an issue in the slightest, it doesn't need therapy!!
It's those who don't do it who'd need a talk..!
stillpointlab 15 hours ago [-]
I think that depends on where you work. If your company demands that all code is reviewed then anyone who doesn't do that should be spoken to.
However, I suspect there are teams that are making different choices. In the most obvious example, I can do whatever I want on my personal projects (I hope you agree). And if I started a business and hire in programmers, I have some expectation on defining the culture, making hiring decisions based on my preferences, etc.
Whether or not such teams succeed will be a matter of fact, which I do not believe has been decided at this time.
9dev 15 hours ago [-]
I hate to say this, but you're now living in a world where the value of artisanal code is rapidly approaching zero. If you still cling to beautifully written code as your core qualification, you're having a rough future ahead of you.
And I say this with the bitterness of someone having lost a big part of what made me enjoy this line of work.
That's not wrong right now, we'll see in the future.
But I have no interest in working as a vibecoder, there are many other jobs that I'd enjoy more
troupo 15 hours ago [-]
No. The issue is: "I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) during review"
How does "low-stakes environment" fix that?
stillpointlab 15 hours ago [-]
Because one may learn that those fears are overblown. Or they may learn that they cannot build any kind of software without reviewing every line of code.
The person is saying "I am still unable to stop" and that is most likely because they feel a sense of responsibility. If they choose a project where there is no responsibility, where it literally doesn't matter if the project fails, they may feel able to stop. And if they manage to make that project work, if they can succeed, then they will build up confidence.
It seems better to be able to work both on projects where they review all code as well as projects where they do not review any code. In fact, keeping their job in the long term may (or may not) require having both skills.
daveguy 14 hours ago [-]
yolo'ing a code complexity generator doesn't make it suddenly stop being a code complexity generator.
troupo 5 hours ago [-]
> one may learn that those fears are overblown.
They are not overblown. LLMs do all the things the person described.
> Or they may learn that they cannot build any kind of software without reviewing every line of code.
We already had bad horrendous software before the onslaught of LLM-slop. Why contribute to that?
> And if they manage to make that project work, if they can succeed, then they will build up confidence.
Confidence in what? It doesn't fix the underlying issue. And the underlying issue isn't "I can't stop" but "redundant code, duplicate code, overly complex architecture, etc." that LLMs regularly produce.
> It seems better to be able to work both on projects where they review all code as well as projects where they do not review any code.
"If I close my eyes maybe this goes away"
jazzyjackson 15 hours ago [-]
I guess they’re just saying “you’re holding it wrong”
Last time I chose to vibe code, I did it in a language I didn’t know so that I wasn’t tempted to try and comprehend the choices the robot made. I ended up with the application I wanted.
Caveat: it was code I would never have to touch again, I ran a script to generate the 140,000 static html pages that the rails app provided and then just uploaded them all to R2 buckets (million reads per month on free tier)
troupo 5 hours ago [-]
> I guess they’re just saying “you’re holding it wrong”
No, they are saying "who cares if all those issues with code are true, just don't review it"
> Caveat:
It's not really a caveat. I also have throw-away code. But we're not talking about throw--away code. We're talking about the systemic issues in LLM-produced code, which are completely ignored and the answer is always "who cares, close your eyes and don't review the code".
BobbyJo 16 hours ago [-]
I think the intended takeaway is to learn to solve those problems without line by line review. Start by being hands off on a low stakes project, and see how the model fails. Then, come up with, and try, solutions that aren't you doing archeology.
troupo 15 hours ago [-]
This somehow assumes that the person above doesn't already do all the magic the people say work 100% of the time, and finds these issues. How does running into literally the same issues in a low-stakes project help?
BobbyJo 15 hours ago [-]
I think it's a good assumption. Their comment implies they aren't harnessing the coding agent very strongly given the failure modes they mention.
Do you think it's better to experiment with high stakes projects? Do you think it'll be easier to avoid defaulting to line by line auditing of a project that is high stakes?
troupo 5 hours ago [-]
> Their comment implies they aren't harnessing the coding agent very strongly
Ah yes. "all the magic the people say work 100% of the time"
> Do you think it's better to experiment with high stakes projects?
This again assumes that people working on high stakes projects only work on one single project.
This again assumes that people finding issues with LLM-generated code have not tried every magic recipe under the sun from people claiming their hair growth elixir is 100% a working solution.
This again assumes that people experiencing something different from the others are inevitably stupid.
etc. etc.
benoau 17 hours ago [-]
This is why I like GitHub Copilot's autocompletion, I'm right there guiding and policing the code as it appears.
fantasizr 11 hours ago [-]
if you take yourself out of the loop it will hurt your skills and career. Engineering blogs used to have some best practices you emulate, but the big ai companies you cannot mimic without their infrastructure
g-b-r 17 hours ago [-]
You're unable to stop being responsible? Tragic!
spiderice 16 hours ago [-]
Is it responsible to move slowly when your competitors are moving quickly? Depends on how your competitors perform. If they perform well, then your "responsibility" suddenly becomes your extinction.
tripleee 16 hours ago [-]
Apple's iPhone Duo seems like it'll be a huge hit.. how many years after the competitors?
They've built an entire strategy around moving slow and doing it better
g-b-r 16 hours ago [-]
If they move quickly by shipping crap they're just running towards a wall, the responsible thing is to not follow them.
xyzsparetimexyz 14 hours ago [-]
fomo nonsense. not everyone has to jump off the cliff just because you are.
15 hours ago [-]
dyauspitr 17 hours ago [-]
No one cares about good code. It’s a means to an end. Your code doesn’t need to look pretty or be well documented or any of that stuff anymore because people are not looking at it. The only thing people care about is performant and error free code.
tripleee 17 hours ago [-]
People saying this are telling on themselves that they have no idea what quality code is. It's not about looking pretty or nice variable names.
Code quality matters to AI just as much as it did to people
7 hours ago [-]
bbctr1 17 hours ago [-]
Do you have any data backing this statement? Or is it just your experience?
bpeng2000 16 hours ago [-]
Personal experience:
AI tends to add new wheels for a new feature in a large codebase, because finding, understanding, and generalizing the existing code to integrate a new feature will likely break existing features, cost more effort (and token) to implement and test. Human reviewers also tend to like these code since they are clean, easy to review, and get the job done, especially when they are not familiar with existing codebase.
The issue is that wheels like these accumulate in the codebase, and they inevitably entangle. When an upstream specification changes, when a new feature that works with many existing features are introduced, all these wheels need to be revisited, and in many cases they are not because the new test cases do not cover all the paths. The code base got larger and more complex, and contains more bugs than the unit tests can cover, especially when unit tests are often "bent" to fit the wrong behavior unnoticed.
The worst part about this is that if you aren't watching it, the AI will "solve" this with a thousand assert_3_plus_5_equals_8() test cases that just further clutter the workspace and the context window. Churn baby churn.
Rudybega 16 hours ago [-]
I haven't had this issue since like last year with frontier models. What are you using?
troupo 15 hours ago [-]
All the models do this. I'm literally right now watching Opus 5.5 duplicating code even after telling it to fix duplication.
tripleee 15 hours ago [-]
-
SmirkingRevenge 16 hours ago [-]
The models are pretty good at finding code duplication and eliminating it though, and it's usually part of my AI review process at this point
If you let it go buck wild on a massive change, there are those times where it sort of paints itself into a corner but rewrites are cheap (for now), so it's just not that big of a deal, imho
Funnily enough, the models learned from GGC and LLVM, yet, cannot produce something half-decent.
hn_acct-1029393 13 hours ago [-]
I’ve had multiple occurrences (with the latest models) of AI reading a poorly named db column and assuming it was one thing when it was another that resulted in a problem. It does not know when to stop or when to continue with determining what is true.
gwynforthewyn 17 hours ago [-]
Literally 50 years of engineering history and culture cares about whether code is good. There are libraries worth of books about it, not as a means to an app but for its own beauty. Just read “The Elements of Programming Style”, for example.
Do what you want, write apps with AI and don’t review code, it doesn’t impact me. The claim that no one cares about good code is demonstrably false, and that’s all I’m objecting to.
Quothling 7 hours ago [-]
This is true but a lot of that history and culture has not been capable of producing software which has better quality than what came before. Around here banks are spending hundreds of millions trying to exit mainframe systems, and they've been failing for the better part of a decade. This despite having hundreds of people working on "modern" software paradigms in "modern" languages.
I do think the capabilities of AI depends a lot on how you work. If you're doing very low abstraction, fail fast defensive programming in something like Rust then it's fairly good at writing code with high quality. If you're working in some OOP language with tons of freedom, then my experience is that it's usually not better (or worse) than most human programmers, but certainly much worse than some.
I've got decades of experience in enterprise development and I've also been an external examiner for CS students. We have people with no coding back ground vibe coding internal tools in Python which are terrible, but not really worse than what we've previously bought from some of the cheaper coding houses around here. At the same time we have AI writing internal SDK's and libraries in Rust which will frankly write better code than I do on a thursday afternoon after too many meetings and too little sleep because the children kept me awake all week.
Good code can mean a lot of things though. If you're deep into clean SOLID DRY OOP programming I suspect that an AI will produce unreadable horrible code, much worse than most humans. On the flip-side the AI probably knows the difference between a generator and looping over a List. It probably also knows more about JVM optimization than most of those banking programmers trying to replace COBOL with JAVA.'
I doubt Antropic is a good sample though. Bun is written in Rust and it's a single purpose software, which seem to be an optimal environment for LLM's.
g-b-r 17 hours ago [-]
> it doesn’t impact me
so long as his software isn't used for anything that can affect you
nerdo 17 hours ago [-]
Not true, LLM’s will struggle with badly organized and designed large codebases just the same, though likely with less errors and profanities.
copperx 16 hours ago [-]
The biggest concern is that AI can doxx you and find where you live if your code sucks. If that doesn't scare you, nothing will.
weakfish 15 hours ago [-]
> No one cares about good code. It’s a means to an end.
I do, my coworkers do, my friends do, etc etc.
swat535 15 hours ago [-]
I think a more generous interpretation of the parent comment would have been: "No one [in business] cares about good code" .
Your end users certainly don't either, heck they'll gladly sit there and scroll through their phone until the 50GB React app loads.
weakfish 11 hours ago [-]
That’s a fair interpretation, thanks for the clarification!
xyzsparetimexyz 14 hours ago [-]
I don't care about the opinions of end users with poor taste as much as I care about the opinions of end users with good taste.
g-b-r 17 hours ago [-]
Good code is a means to the end of the code being error (and vulnerability) free, maintainable and performant.
And if people are not looking it, those people don't care about their customers.
andrekandre 11 hours ago [-]
> those people don't care about their customers
if it was any other industry like medicine, food, or construction etc, could they get away with just shipping products where they have no idea whats in it?
where things like "it tastes good to me" or "cement didn't crumble when i hit it with a hammer" is good enough to ship we'd be inundated with lawsuits and customer claims... its a total joke where the software industry is headed
g-b-r 7 hours ago [-]
Absolutely
drewg123 16 hours ago [-]
The way you phrase it makes me realize its almost like a GPL contamination. Eg, once you pull in GPL code, your project is infected and can never use a BSD (or other GPL incompatible) license.
Similarly, once you pull in terrible AI slop code, the project becomes unmaintainable by humans and you need to maintain it with AI.
gwynforthewyn 16 hours ago [-]
There’s a little irony about the example you picked of the GPL, just because in the US the output of an LLM cannot be covered under copyright. The GPL works as an assignment of rights under copyright.
This is, of course, tangential to your point.
troupo 16 hours ago [-]
Code that isn't duplicated in 15 places, that doesn't read the whole database to fetch just 1 record, that doesn't do 15 database reads where one would suffice, that uses an optimized algorithm instead of an exponential one recreated badly from scratch... all that is also good code, and is just as important in LLM-slop era.
xyzsparetimexyz 14 hours ago [-]
No code is error free. All code will need a fix of some kind eventually. When it does, it's better that the internals are not a total mess. In the same way that you don't want the insides of your car to be total alien nonsense either.
hyperpape 17 hours ago [-]
I'm somewhat surprised they would say Claude is writing 90% of their code at this point. Did they get tired of reporting new milestones, or has there been a plateau?
pokstad 10 hours ago [-]
[dead]
lbrito 14 hours ago [-]
Token peddler says best way to do something is using more tokens.
This is not HN-worthy
ativzzz 16 hours ago [-]
It looks like they are specifically optimizing for incidents/reverts and shipping faster
> The goal hasn't changed: less human time per change, and no more reverts than before.
Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target. Is this then still a good measure? I guess Revenue is a more direct measure, but we don't have access to that one.
enraged_camel 16 hours ago [-]
From my perspective as a power user, the product has improved substantially over the past six months. It has gained new features that are actually useful, lots of UI/UX/perf improvements, and bug fixes as well.
New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.
All in all, I'd say their approach is working.
zeroonetwothree 10 hours ago [-]
I'm curious what has improved so much. To me Opus 5.5 is very good but so was 4.6. I try to forget the versions in between...
lavezzi 11 hours ago [-]
not even one tangible example?
UberMouse 9 hours ago [-]
The Workflow engine, the Mod system. I've tried to explore other coding harnesses and nothing is as powerful as claude code (barring investing a lot of effort into custom plugins for Pi or something). Other harnesses have workflow engines, but none are as powerful as Claude Codes.
UberMouse 9 hours ago [-]
And lets not forget the `claude agents` view and the backgrounding system. A++
enraged_camel 8 hours ago [-]
I really like the the new ability for agents to start and manage other top-level agents. Makes it super easy to have one agent as the orchestrator for a feature and have it launch new sessions, answer questions from those agents, review their work and build merge trains. This was possible before, but required a fair amount of manual work. Now the experience feels first-class and it's really nice.
drdexebtjl 9 hours ago [-]
They are in for a surprise when they become publicly traded and have auditors requiring actual humans signing off on any code that touches financial data.
rileymichael 16 hours ago [-]
what are people using for automated code review these days? the claude code review product originally mentioned it cost "$15-25 per review" (which is absurd), but it seems like they've removed that from their docs with the launch yesterday. we've looked at various products in the space (coderabbit, greptile, etc.) but have gripes with each.
honestly i'm thinking it might be easier (and cheaper) to launch our own agent from GH webhooks and trigger `/code-review`..
NorthSouthNorth 15 hours ago [-]
We've had great success with a custom gh action on a self hosted runner. It sets up codex and all the same stuff a dev has on their machine:
So dbs, seed files, etc, and then just a review prompt and some skills and a few scripts.
It triggers on a new published PR or if tagged @bot-name. It's way better than the other services we've tried (which was admittedly ages ago) because you can get it to work exactly how you want it and it's dead simple.
With Codex, for the time being at least, you can use a subscription, so it's basically free if you're already paying for one. But on API pricing it's still not too bad as it doesn't produce that much output.
I set this all up in about half a day. Codex does 99% of the heavy lifting.
tills13 12 hours ago [-]
this: gh actions using an opencode harness (so we can use cheap models through OpenCode Zen). With the right model, and the right, scoped access, it's just a prompt away.
bugbot was costing us $1k per month for a small team. EVEN IF we decide to go with a paid or say something like GPT Sol for our custom harness it'll cost us maybe $100. We've already implemented all of bugbot's interesting features like cross-PR memory.
the cool thing is we can leverage this for things that bugbot cannot do. Just today I added a security-reviewer action reusing almost all of the code that we wrote for the code reviewer.
PJDK 14 hours ago [-]
I've set up a claude code on a vm with a bunch of skills that can use git and the Azure Dev Ops MCP via a service user, the regular build agent pops a record in a db that serves as a queue/review and a bash script picks things up and fires off claude code in headless mode.
It's pretty effective and lets you do nice stuff like compare the code to the tickets, reply to comments and stuff like that.
walthamstow 16 hours ago [-]
Using a different lab's model to review the code to the one that wrote it. Claude model plans/codes, OpenAI model reviews, or vice versa, or DeepSeek or GLM or whatever.
a3nz3g 13 hours ago [-]
I do it this way too. I think having the same lab's model review the code in a fresh session also helps, so I use both: I generally start a new Claude Code and a Codex and have them review the code with a focus on 3 things: 1. bugs, 2. cut corners, 3. code style.
This generally works for me. I think a solid spec is the key for the models to review against. I ask them to follow the spec, and if the spec doesn't say, follow first principles.
Also, if there is a user experience change, they need to notify me.
How about your pattern?
warkdarrior 16 hours ago [-]
> the claude code review product originally mentioned it cost "$15-25 per review" (which is absurd)
If an engineer paid $150,000 a year reviews some code manually for about 30 minutes, the cost of their time is about $36.
rileymichael 16 hours ago [-]
in a vacuum, sure. the average pull request will have multiple commits once opened (i.e. to address feedback or perhaps it was opened too early), each triggering a review. it's very easy to rack up $100+ per pr (and an eng may open a dozen prs in a single day)
shukla1289 6 hours ago [-]
[flagged]
skillstackhq 8 hours ago [-]
[flagged]
aryehof 5 hours ago [-]
> Code review does two jobs:
1. Find and fix issues before they ship
2. Build shared context and codebase understanding across the team
So code review has become newspeak for “testing” and “education”?
Well not really testing. We the users are the testers and it falls to lucky us to really discover and find the errors, edge cases and security issues. The consequence of the rush to continually release…
Is the AI advance in the field of software testing nothing more than “now an AI agent can do a code review?”.
mcbishop 7 hours ago [-]
> At Anthropic, our jobs have moved from reading every diff to the decisions machines are bad at:
> ...Architecture that sets a pattern others will copy...
I'm happy that even Anthropic agrees this is best left to humans.
jwpapi 16 hours ago [-]
What I’m more worried about is feature identification and how to monitor them. How can you get the product to the next level?
Honestly I’m not worried about code anymore in my codebase I’ve not had bug problems for a while and incidents get fixed automatically.
I monitor new features with an agent who checks recordings.
Most actually good features need still 20-30 iterations to design the userflow to actually be interactive, but then there is also another challenge to translate usage patterns into features and how to present functionality with the least amount of friction as possible.
How to efficiently run monitors without wasting your token spends. A best 'monitor' best practice would be nice.
Still this article was very helpful and gave me a lot of new ideas.
stymaar 17 hours ago [-]
Poorly I'm afraid, given how many new bugs make their way to the claude code CLI every update.
compounding-err 15 hours ago [-]
I think this is the future. Im more bullish than bearish. The only issue is that eventually human verification can no longer be possible and we will have to live under AI-verification assumptions....
KasianFranks 14 hours ago [-]
We're already there with our pipelines and suspect a few more out there more concerned with measuring outcomes at this point.
bob1029 5 hours ago [-]
> Claude Code lives in its own repository, separate from the monorepo
I have some bad news about that "monorepo" designation.
stkdump 5 hours ago [-]
That's not news though. In fact almost no company beyond maybe 10 people that has a large central repository has no other smaller repos with other smaller production projects. They still always call it monorepo, because it approximately still behaves like one.
willtemperley 4 hours ago [-]
I think this is exactly why the current "attack of the vibe-clones" will fail.
Vibe-coding complex software can work, but it's going to take a really disciplined team to manage this properly, like in this article.
btbuildem 12 hours ago [-]
It makes sense for Anthropic to pitch "more code = more better" but as anyone who's been in this game long enough knows, "less is more" often enough. And good luck coercing an AI coding agent to solve a problem by removing things.
Personally, I find Claude's degenerate urge to just spew uncontrollable amounts of code quite counterproductive. Yes I get it, you get paid by the token. I'd like to see more reasoning, more understanding, and less generating.
Truth is, we are still very very far from actual AI software engineering. Some people are working on useful tooling, and (predictably, like anything good and useful) it is taking a long time. LLMs are great, and hopefully they will help us accelerate building the tools that bring this profession to the next level.
30k PRs in a month (or whatever) is not the brag they make it out to be.
gonzalohm 17 hours ago [-]
That explains why the UI keeps changing.
How is this a proper product? Every time you open the desktop app the buttons may be on a completely random place
enraged_camel 16 hours ago [-]
The UI has improved substantially over the past six months. New capabilities are prominently featured and less-used ones are de-emphasized or removed. Every product goes through this type of evolution. The difference with Claude Code is that at the speed they iterate, it happens a lot faster and is therefore more noticeable.
gonzalohm 15 hours ago [-]
You can call it however you want, but in the end it is just poor engineering. Imaging building a bridge that needs construction every week
enraged_camel 12 hours ago [-]
Software is very different than other engineering disciplines. It is meant to be malleable. I think you need a better analogy.
andrekandre 11 hours ago [-]
customers don't like big changes to workflows they get used to; adding a new feature or a new button or menu item isn't a big deal but rearranging things so often start to feel random and shows little respect for the customer (or signs of any cohesive product design)
gonzalohm 10 hours ago [-]
Is it, though? Imagine NASA software having to change in-flight every day...
enraged_camel 9 hours ago [-]
Claude Code is not in-flight software though.
gonzalohm 8 hours ago [-]
That's what I mean, it's a piece of badly engineered software
pianopatrick 14 hours ago [-]
I am happy people are testing these things out. I am also just going to wait a bit longer before really adopting things like this, I don't really trust it yet. A year ago this looked very different, and next year my guess is things will look different as well.
rddbs 12 hours ago [-]
Genuine question I’ve been wanting to ask folks working on different types of things: how are your teams doing agent-assisted code reviews? Not automated reviews, but what techniques and tools are humans using to keep up with the larger volume of PRs that seem to trend larger and larger?
I think a lot of people like myself still want to feel in the loop but I don’t have the right tools to stay on top of the (generally good, productive, and still thoughtful) contributions from peers anymore.
pokstad 9 hours ago [-]
Using agents for a first pass is a good way to speed up the review cycle. This way contributors can get near immediate feedback before a human needs to weigh in. This doesn’t replace the human review, but augments it. Each time your agent misses something, update your code review guidelines.
thisisauserid 17 hours ago [-]
Remind me to invest more in the clouds they are burning those tokens in.
mixtureoftakes 17 hours ago [-]
30 thousand changes merged per day? no. like just no. what are we doing? I'm trying to understand
devin 17 hours ago [-]
There was a post some months back called "Lines of Code Got a Better Publicist" or something along those lines. This is just more of that. I am not impressed by 30 thousand "changes" per day in the slightest without significant added context. When I logged in and prompted Claude today, do I feel like I got +600,000 change value from when I logged in a month ago?
grey-area 16 hours ago [-]
I’m horrified by it. There is no possible justification for it but it does explain their buggy products.
Rapzid 5 hours ago [-]
This might be one of those things were everyone on the team takes a 30 minute lunch instead of 60 and just nobody clocks out to get the 1 hour extra over time.. And everyone just kinda ";) ;) ;) Working from our desk while eating lunch ;) ;) ;)"
JackSlateur 17 hours ago [-]
I, too, can write and merge 30000 changes per day. Even 300000 if I've up to.
Of course, just like "line of code" is a worthless metric, "change merge per day" is;
for i in {1..10000000}
do
echo "I am working !" >> README.md
git add README.md
git commit -s -m "feat: working with AI"
done
(automerged PR, JOB IS DONE)
dfunckt 7 hours ago [-]
Absolutely, at this rate most software would be complete in a few hours. I may lack imagination but wtf are you changing? Endlessly refactoring? That’s spinning your wheels renamed as productivity.
JamesSwift 15 hours ago [-]
lol right? So youre telling me they shipping 150,000 features this week? Or, maybe, they whack-a-moled 149,995 perf or bug regressions and shipping 5 new things. :shrug:
aacid 17 hours ago [-]
The premise is clear, is one agent can do what developer can, but quicker, why not speed up the whole workflow to match it?
ActionHank 17 hours ago [-]
Have you used Claude Code?
Have you seen the benefit of the churn of 1000 PRs a day?
leoqa 17 hours ago [-]
It’s only about 10 per engineer. I’m not at Anthropic and before 2026 I was regularly hitting 1400 commits / year on my GitHub. If agents made me 2x faster (they have, review is still slow; I currently have 40 PRs open) that’d be the same rate.
IneffablePigeon 15 hours ago [-]
> 40 PRs open
Ah, so you've privatised the benefits and externalised the costs.
_verandaguy 17 hours ago [-]
Right, but why do things this fast to begin with, especially for a product that's already publicly available?
I don't want to use a product that shifts under me as I use it several times a day. I want a product that actually moves slowly and works very reliably.
As an end user: why would I care about how many hundreds of releases a month are being pushed if maybe 1% of them have features I care about?
Speed is not the product.
sp4cec0wb0y 17 hours ago [-]
More code output isn't a metric of success per se. Goodhart's law.
Andrex 17 hours ago [-]
Humans don't operate at AI speeds. We're making the car go from 0 to 5000 mph in 1 second and the drivers are turning to goo in the seat.
xyzsparetimexyz 14 hours ago [-]
haha I do like that analogy.
crooked-v 17 hours ago [-]
Well, that explains the Github repo with 5000 issues, maybe about a third of which are random regressions of previous behavior.
devin 17 hours ago [-]
I snorted laughing at this. "We are experiencing tremendous growth" and it's just a codebase with 1,000 agents endlessly adding markdown documentation for every 1 line change.
Rapzid 6 hours ago [-]
Codex had over 15k issues last I checked including legit months old issues with the sandboxing/bubblewrap config.
Human contributions to issues, comments or submissions, were drowning in a sea of AI vomit. Tons of the issues had those AI issue submissions with the classic confidence of having identified the problem, but that smell I know a lot of us recognize of it not actually knowing what the problem is at all because it hadn't failed to fix it 5 times yet.
AlexErrant 17 hours ago [-]
I just learned that manually invoked skills MUST be the _first_ thing in the prompt, e.g. you can't have `disable-model-invocation: true` and say "Hi, /grill-me on X".
Worse, everything after the /grill-me is injected a second time into the prompt as a <command-args> so your prompt is literally added to the context TWICE. (first time in the original user prompt.)
I swear Anthropic is intentionally bloating your context so you burn tokens on useless bullshit like this.
svachalek 16 hours ago [-]
Oh /loop was the crowning achievement. Basically guarantees you'll run a prompt at max context without caching over and over.
steveklabnik 12 hours ago [-]
I have done exactly this as recently as yesterday, so I’m not sure why it doesn’t work for you.
AlexErrant 10 hours ago [-]
To be clear: you can certainly do "Hi, /grill-me on X" anytime you want. You can even write "hi, /grill-me and use /search-docs", even though the /search-docs skill absolutely doesn't work. But the Claude Code _harness_ won't autoload the skill into context, so claude-the-model has to go "hmm, where can I find that skill, when I call the Skill tool with that name it no worky, so now I have to search the filesystem for that skill...".
The <command-args> I know because I just audited my raw Claude transcripts via ccglass. You can verify by just asking Haiku "hey, /grill-me does this skill autoload for you?" and then "/grill-me hey, how many times does this text appear in your context?"
You can trivially doublecheck me. The above prompts won't even eat 1% of a 5 hr on 20$ plan.
I believe their peak month was >13k issues opened, but worth bearing in mind: "This issue has been automatically locked since it was closed and has not had any activity for 7 days.".
layman51 17 hours ago [-]
I was running into something similar where I had it working on a single Python file. Maybe part of the issue is that I had memory turned off, and the conversation might have spilled into two conversations. I wasn't using version control either, but since it was a single file with about 1000 lines of code, it didn't seem too hard to eventually get all of my features back.
K0balt 4 hours ago [-]
The furious pace of Ux “optimisation” that we see across the board (and especially at AI companies) is a hidden anti pattern. The time and frustration of learning a new ux is an acceptable burden when it frames real new capabilities or workflows. It’s not good to rework your ux user experience every week just because you can.
Not only that, but AI is categorically terrible at ux review. I don’t mean aesthetics, I mean attention flow, workflow in general and what the user anticipates as a logical path to feature discovery and exploration. Increasingly, features are hidden behind workflows that meet a single use case but fail to be universally logical or in many cases, even accessible at all outside of the test case.
So, people… use AI in code review. It’s actually pretty great at it if you do it right, and you can ship better software with fewer bugs than ever before. But please, don’t pressure your ux guy to “10x” with AI. In fact, now that you are “10x”ing your development cycle, you are probably going to need to hire -more- ux developers if you don’t want your product to feel like slop even if it’s gold under the hood.
beardbandit 11 hours ago [-]
I'm astounded that I kinda ended up building the same agentic workflow they are talking about in this article and distributed it amongst the org (eng and product/designers).
On the other hand, I would have expected Anthropic to have a drastically more elegant SDLC.
nathanqueme 14 hours ago [-]
I built a library to have a way to check Claude Code's work against API docs, and asked it to write many tests on top, but even then I found Claude Code would paraphrase and drift from the docs. I also found that it passed its own smoke tests but failed when testing against the real APIs, so reviewing Claude Code's code is tricky.
alasano 14 hours ago [-]
I simply don't want to review AI code in depth.
I used to do it but now I only review code organization and structure.
Mostly I define the product behavior I want and discuss the technical implementation beforehand.
Once that's in a spec I let it build in my durable implementation engine. Automated verification, review rounds from different agent types across different models, issues triaged into meaningful clusters and flagged for false positives before finally being fixed.
That loops until the code is up to spec.
There is no special secret, LLMs need repetition and multiple passes.
They'll one shot bigger and bigger things in their default harnesses but they'll be filled with issues no matter what.
petersumskas 5 hours ago [-]
The writing in that blog has the distinct whiff of Claudish to me!
matltc 10 hours ago [-]
Interested to know how they handle ordering changes. Sounds like with that many changes, you're bound to hit lots of conflicts.
senderista 13 hours ago [-]
If their only metrics are revert/incident rate then they're missing the long-term, cumulative impact of compounding complexity due to poor architectural decisions.
sigotirandolas 3 hours ago [-]
My thought as well, I think this approach only works for them because:
* They can afford frequent CC incidents since their value is having the best models, and 95%+ uptime is still OK-ish.
* The core functionality of CC (the harness "agentic loop") is simple, an if that works then for most purposes CC works.
* CC just stores a bunch of config and history files, there isn't any financial or irreversible safety loss if they mess it up once in a while.
Once you work on something where a minor slipup can result in a huge real world mess, then I don't see how their approach can work.
julesrms 5 hours ago [-]
30k commits per day on a console app?? Are these all one-line changes?
I've written a harness myself and fair enough, it's more complicated than you might expect.. but surely at this point the claude CLI is pretty stable, and this must mainly be a ridiculous amount of test churn.
Or maybe their team of clankers is just creating busy-work to make the humans think they're essential. That's not unheard of in the human behaviours that they're trained on.
typesanitizer 17 hours ago [-]
FWIW, Pangram identifies several sections as 100% AI-written.
zeroonetwothree 10 hours ago [-]
I would have been surprised were this not the case
luisgvv 17 hours ago [-]
And yet they can't fix the janky bugs when you scroll or show/hide chain of though, commands executed, configuring plug-ins from the options menu etc.
nakul_ 17 hours ago [-]
I wonder how they maintain the quality of tests for such a massive codebase and frequent commits.
owebmaster 17 hours ago [-]
That's the trick: they don't
corentin88 16 hours ago [-]
Claude Code is great, but do you really push that much (new) code every day?
12 hours ago [-]
swingboy 14 hours ago [-]
Sorry, but how the hell does a TUI harness have 30,000 changes _a day_?!
steveklabnik 12 hours ago [-]
The 30,000 number is across their entire monorepo, which they say in the article Claude Code does not live in.
foolfoolz 16 hours ago [-]
everything is shipped with auto reviews behind a flag. then you need a human approver to launch
good model
jstummbillig 17 hours ago [-]
That's a shockingly low number.
t-writescode 17 hours ago [-]
1000/day in what sounds like a single repo is a low number?
jstummbillig 17 hours ago [-]
I am not sure what it is you are saying. What I am saying: I would have supposed that Claude writes 99.x% of Claude's Code code, and that ~no code at Anthropic is written by humans anymore.
sandeepkd 17 hours ago [-]
I agree finding out who is writing that 10% of the code in the AI company is important. Maybe time to identity them and put them on PIP since they are slowing down everyone else and giving a bad name to the AI company
shadow28 14 hours ago [-]
The 30,000 number in the blog is the number of changes landing per day, not per month!
ActionHank 17 hours ago [-]
to achieve what?
Honestly not sure that there are that many effective changes a day given how little complexity there is to their public facing product.
oofbey 17 hours ago [-]
No. 90% sounds like a low number. That implies something else is writing 10% of the code? People? Other models? What on earth is writing that other 10%?
bdcravens 14 hours ago [-]
They use Codex?
kettlez 16 hours ago [-]
If the entire body of this article isn't just an "LGTM" then I know they're lying.
smetannik 13 hours ago [-]
spoiler: they don't
JackSlateur 17 hours ago [-]
90% ?
A random twitter said that, 6 month ago, it was 100% !
DECLINE ! SELL THE STOCK OPTIONS ! (wait, there are none, confused!)
ChrisArchitect 17 hours ago [-]
Title is: How the Claude Code team does code review
13 hours ago [-]
pang_claw 18 minutes ago [-]
[flagged]
miroljub 14 hours ago [-]
I don't care how they do it.
I vibe coded my harness that is superior to CC in functionality, and it still doesn't flicker.
I don't need to hear their bullshit.
surgical_fire 16 hours ago [-]
Anthropic's solution to anything is to simply burn more tokens. Because of course.
Anyway, flagging as misinformation.
rwieruch 7 hours ago [-]
[dead]
favori995749721 9 hours ago [-]
[flagged]
gary1381 15 hours ago [-]
[flagged]
drewbitt 9 hours ago [-]
[flagged]
r2ob 11 hours ago [-]
[flagged]
0c3ca83 13 hours ago [-]
tl;dr: We don't, claude does. Make of that what you will.
vektormemory 14 hours ago [-]
We have to test many different api's and models as we have over 600+ in our harness now, Sonnet 5.5 via cloud code provides the highest quality results.
Over the last few years we have added mcp tools, memory graph, md files and skills, automated everything to the point that on desktop the system runs itself, besides proceed/approved/explain.
We have 100's of PRs/jobs/projects and Claude builds down the list automated and only asks a few questions point form if needed.
Similarly, Anthropic can afford to do this because they have virtually unlimited tokens and labs to train models for their specific usecase. The rest of us need code comprehension lest we become dependent on continuous supply of tokens (which is a state these labs badly want to achieve)
So, unless you work at a lab, or have unlimited tokens guaranteed through the life of your company, it would do both the company and you good to continue to read the code and not let these 30k PRs per engineer be the norm.
An excerpt from the article makes my point
> Nobody on the team keeps a complete mental model of Claude Code in their head, and we don't think anyone needs to. When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today.
If their Claude is the same as our Claude, then this is a joke. Claude’s writing style is insidiously mind numbing. It takes constant pushing to get it to explain things in digestible English.
Which is fine, whatever, I guess this is how I pay the mortgage off instead of having any clue what the hell Claude is actually writing. Hopefully I can semi-retire soon and go study dunnarts or take up welding or something else that's more rewarding than this nonsense.
Think I'm good on taking advice from companies that aren't making money.
Because "pacing the frontier" is more about flipping on the money printer switch than it is about safety.
For sufficiently large codebases in large organization that is how it has been with humans, who come and go. In many respects, What Anthropic is doing is a massive upgrade from the status quo.
Jesus. Give me 1 billion dollars and I'll keep your entire codebase in my head + personal notes for you until I die. Hell, I'll even force my descendants to do the same. Now _that's_ an improvement.
> Most of the path is automated; a person's attention goes where the risk is
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
Access to a wider array of special characters is also one of the really nice things about touch keyboards on mobile or tablet. Keyboards are great for fast, accurate typing but not for going beyond the basic character set.
I think when you repeatedly expose and immerse yourself to such a degree in reading and writing language, it brushes off on oneself. So I could also absolutely believe someone from Anthropic wrote this.
> This is not a cryptographic break, new attack, or weakness in the ECDSA curves.
> Most of the path is automated; a person's attention goes where the risk is
--> We automate as much as possible, focusing human attention on risky parts
> The eval grows with every incident, and every change to the reviewer is scored against all of it.
--> We run reviewer evals against every incident
But that's just my interpretation and my voice. I can't know for sure what the authors would have said if they took the time to write it themselves.
I really don't get the tendency here from Anthropic to describe what's most likely churn? A high pressure amount of changes/PRs a day on a TUI to wrap a while loop? I'm not going to be a reductionist here so I'm going to cut it off there, I'm positive that the complexity of handling multiple terminal emulators is something that needs some handling (even if doing their own rendering engine was a self made problem). Yes I've vibe coded my own harness from scratch without writing any of it, I still know how it works.
I'm also not sure that they addressed point 2 in any actual capacity? They say "the system verifies itself" which is cool and is something that automated testing has done for decades, but they eschew the mental model of how the system works as a whole and prefer instead to have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
N.B. Changed "1000 PRs" to "A high pressure amount" to be vague since the article is vague here on what CC undergoes.
This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...
There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.
Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.
At some point I was tasked with improving the queries/retrieval times and started converting parts of the code and not only was it possible to write tests that would convey the actual meaning of the code and crystallise it but I started finding bugs in the implementation that no-one realized were even bugs because they no longer knew what it should do when queries grew to certain complexity and the search space was in the hundreds of millions of rows that thinking through them just with napkin calcs wasn't cutting it (specially when the bugs were subtle and the results still allowed the program to work "mostly" correctly).
But I have see quite terrible finance software, patches over patches hold together by a duct tape. Also finance software has to implement large complexity of tax law, which is very large and always changing. It also doesn't help if the company developing finance software sees software development as a cost center and wants always to push the costs down (especially short term costs).
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
Those sorts of changes are really rough to manage. Maybe still manageable by LLMs but we'll find out.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
lol, I assume you’re not a parent then. Or at best a new parent.
So the real question would be, how many people designing / implementing the boot process of any given system don't understand what what's going on.
The comment https://github.com/anthropics/claude-code/issues/30407#issue..., written by Claude Code, and posted by the human GitHub account says:
> "Issues here are auto-closed by a bot when they've had no activity for a while and are missing information needed to reproduce (usually a version and steps, or a /feedback ID). Nothing is triaged solely by counting"
In reality all issues in this repo are subject to time-based auto-closing, unless they collect a number of upvotes. See details at https://github.com/anthropics/claude-code/issues/87647.
Therefore, I wonder whether the internal review process described in the Anthropic's blog for internal repos also results in such issues, and whether Anthropic has a internal way to measure and mitigate that. The blog post says:
> "When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today."
The Claude Code public repo problem shows that LLMs don't always "reflect the code as it is today", and can be a source of additional misunderstanding.
What you are saying is leaning on 2 premises that seem to not hold water 1. Code written by people is always more readable and maintainable than LLM code 2. Code written by an LLM is always less readable and maintainable
Can you really make those statements in all intellectual honesty?
30 years working in this industry and I'd say that's the exception rather than the rule. When it comes to large systems.
I love this phrase, and I think it captures the difference between the "60 years of legacy code" mentioned elsewhere here, and the vibe-coded monstrosity that we all know we're going to be encountering in the next 5 years. The fact that there were human brains with intent making all the little (short-sighted, myopic, ignorant, etc) changes in the legacy code means that you DO have at least some shot of gaining enough context to make safer changes.
There is no real intent in LLMs and particularly in blind vibe coding, so you'll end up reading stuff that works, or appears to work, but there was no human brain behind the change for you to model and thus predict.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
It is not completely useless as a tool but there is still a huge gap in most real-world applications (I work in an org with multiple thousands of devs and I am top 5 AI user consistently, I have used Claude Code since pre-release). If you are building something like an SQL database which has decades of logic tests built by humans then it can work. If you do not have this (i.e. 99.99% of the time) then you still need to be able to issue prompts that give context. And if you start without that context, it will produce shit code in extremely basic scenarios. In particular, humans are far better at producing code that is easy to change (in practice, you need to create unintuitive, non-human rules to govern this i.e. creating many artificial boundaries which would be completely unnecessary for humans).
I am not sure whether the models just capped out in terms of coding or whether they are just trained by people who have never worked on enterprise software but the gap isn't closing. It has just been filled by people who lack the knowledge to identify when they are being bullshitted (the prevalence of consultants suggests that this markets is large).
I have worked on legacy stuff before. In most cases, there is a kernel of reason and logic that can be recovered. I have seen pure AI projects and they are the worst grade of TCS-authored abandonware. They can often do one specific thing but are basically unmaintainable outside of that.
I frequently find this in LLM generated comments, both when using an LLM myself or when reviewing code that someone else used an LLM for. I find it is often the most obvious indication that nobody really read through what was generated.
Then you investigate it and it is often something relatively simple that could have been explained with short sentence.
If you give them no constraints, they’ll produce code that just works, and it’ll look exactly like that.
We already know the solution to this problem. Software engineering. Put the LLM through the same grinder as any software dev and out constraints into place. Tell the LLM that the code design is bad and that it should improve it.
In my case, I had to go through several rounds of refactoring before the code at least resembled a good codebase.
A fun fact is that the refactorings didn’t break any functionality. Only adding new features broke preexisting behavior.
This is not very different from what already happens to architects that delegate implementation to teams.
The challenge Is to find a way to encode the architecture in a way that both you and the agents can understand and agree on.
I'm still learning how to do this effectively, but my initial experience is quite positive. Not because agents are great but because humans are bad; the degree of coherence I've seen agents following is bigger than many human teams I've led in my career.
My benchmark is a quite big project for which the architecture is already well known, namely the reimplementation of the OpenVMS OS (kernel and user-land). There are many traps and many opportunities for the agents to just go ahead and diverge from the goals, so I found it a good playground toearn how reign them in but let them be effective at "drawing the rest of the owl"
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.
Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.
In most orgs, there is also a gap between leadership, even tech leadership and people actually using LLMs every day. And it is challenging for actual devs who are being told that they should be going much faster and the reality which is more mixed. Pushing back on people who are convinced that Claude is God is going to be tough if you are a junior.
I don't feel the job has really changed, I don't write code as much but that time goes into working out the structure...which is what I was doing anyway. But it has changed the general environment, how work is planned, expectations, etc.
At this point, why even bother with source code? Source code is designed for humans to comprehend. If humans are no longer the intended audience, why not have machines output and maintain some more efficient intermediate representation of computer programs?
Thank you! This is a huge sticking point for me. There is no knowing the code without experiencing the code. If nothing else, you have to read the tests to know they verify what you think they do. Even then, you might know it does what you want, but can’t know it doesn’t do anything it shouldn’t.
There are no shortcuts to true understanding.
I know some people will groan hearing the phrase again, but: "Reverse centaur." A centaur is a synergistic combination of two things, but now those things are in the wrong position: The human is the computer's tool.
Naturally, how "wrong" it is tends to depend on whether you ask the people who do all the work or the people who own all the computer-capital.
There's clearly enormous (human) work that goes in to building and tuning their models.
The software they develop around their model is, relatively, trivial. It's not clear that enormous amounts of (human) energy need to go in to that code.
It seems largely a function of applying human energy towards the things that are marginally most important to generating you money and putting you in front of competitors.
30,000 changes a day is not my idea of trivial.
With a few thousand lines of code over the course of a week or so of my limited evening energy, I have agentic loops, ide integration, and a limited selection of custom tools which are all very safe (no unintended changes), along with session management and awareness of git.
It works great, is customizable, and allows integration with any provider.
If they need 30k changes per day, it’s not due to the nature of the problem!
You have to remember all of the surrounding features it works with now, like the infra behind the cloud agents, or how remote control works, the mobile apps, their connectors, team skills and mcp servers, on and on....
Haha, I don’t doubt it! Pretty damning overall for long-term maintainability of vibe-coded projects, even as the models are great at a lot of individual tasks
From the article, the 30,000 changes a month are in the monorepo, and Claude Code is in a separate repo. So it’s not the TUI that has 1,000 PRs a day.
I could easily see 1,000 PRs a day on the backend. Big tech was doing more than that pre-AI.
I guess it wouldn't be surprising for people prone to AI psychosis to be attracted to working at the frontier labs, resulting companies that are heavily in the throes of mass AI psychosis.
Being tongue-in-cheek here.
They heavily use Claude for both code generation and code review of Claude's own code and related tooling. Is that not functionally Claude running recursive loops to improve the next version of Claude?
I disagree. Many of us have an emotional attachment to the way we do things and have difficulty letting go. Anthropic is showing a different way of doing things. And I am not going to judge them too harshly if the final output is good. I use Claude Code every day. And it is good. Ahead of the pack.
Yes, this is a different way of doing things, and it's clearly working for them, but there is no direct line between that and a "ahead of the pack" product.
> None required once automated review and security checks are green
How anybody can let an LLM write docs and not even read them is pure insanity.
Nobody on the team keeps the mental model of Claude Code as it is today in their head, so nobody on the team is able to reason and decide about Claude Code in the future, is that right? So no long term architecture strategy can exist as no current architecture understanding exists. While it seemingly does work for a system which is a TUI, would the same be true for a complex distributed system today?
PS: I really want to know what is the churn rate (and the cost per LOC) here. My gut is saying the cost per LOC is insane, and they are basically rewriting over and over the same code.
Agree that 30k PR alone doesn't mean much, but there aren't other details on what's inside. Based on I saw on how LLM programming works, it's hard to believe that there are single-line fixes. If I have to bet I'd bet on 200 LOC on average, that is 6M/day. Even if only a small portion reach production, it's still insane.
I don't think we've gotten to that level of reliable review with code, but I don't see any reason we can't continue moving in that direction.
For code, I think things go in the opposite direction: the system outputs artifacts that are all different among themselves (even more so now that we no longer distribute software via physical copies. Well, bits are always physical, but I mean we no longer ship floppies/CDs/DVDs).
That said, at least in my experience, code review has never been a direct part of factory-line-type QA. That QA is typically done by treating software as a black box and just verifying that it complies with a spec (often times based on a very loose definition of "spec" ...), and this is something that works equally well regardless of whether a human or a machine wrote the software. The type of QA that is done (among other things) by code reviews is checking whether the software's design is such that it will be economically feasible to maintain and extend it, for example.
The one part the code reviews takes a part of (but is in no way error-free) and that you just can't do with black-box testing (or it is much harder to do) is to look for things like backdoors. As our industry moves more towards "agents write software and tests validate that it works," I think a lot about "Reflections on Trusting Trust" and the new and wide attack surface added by people no longer reading the code shipped on their behalf.
Of course Anthropic would love you to think you are heading towards that future so we have to bear in mind their livelihood depends on LLMs being the future of work and 30k changes a day being normal.
But why would we want 30k changes a day in a given product, what possible justification could there be?
First, I think we overestimate how different code is from other code. We even joke about CRUD apps and how cookie-cutter they are. A lot of changes in those apps are: add some new features to a data access layer (SQL, ORM, whatever), add new routes, add new UI surface, etc. There is an entire industry around "Design Patterns" and people live by the DRY axioms. Things like Ruby on Rails exists because rebuilding these systems over and over is wasteful. I feel we are just moving further down that same path.
Second, your question of "why would we want" is totally backwards. Progress doesn't follow your desires. Progress doesn't require justification. Sometimes we adapt the environment to our needs, other times we adapt to a changing environment. Which case we are currently in is still undecided and you are free to fight for the future you prefer. But demanding an industry justify itself before it is allowed to change fails in the face of evolution. If it works, we will justify it in hindsight.
stillpointlab.com , "Get more from your LLM"
If you want a software analogy to those devices getting churned off the assembly lines, it's the TCP streams coming out of your network card, with crazy checksums punching out bad packets.
We're orders of magnitude better at reliability with code, but we're regressing on that metric rather quickly. And the comment above deeply misunderstands both manufacturing and software.
That's new. All analogies to 2022 software engineering need to be thrown out.
Software construction is now more like an industrial process, with the artisanal work shifting towards both
1. aligning what's built with what's needed (talking to customers!)
as well as
2. implementing and tuning your software construction process (ie. hi - we're Anthropic. We're burying ourselves in a mountain of code but it's okay because the fix is a second mountain of code, try some tokens the first hit's free!)
Back to my original point...
Writing code is now on the production line; making quality output demands industrial grade/scale verification (some involving humans of course).. which is a fascinating engineering challenge, and it's brand new!
Verification means nothing for industrial processes when the desired outcome is a moving target, so the first part of an industrial process is to develop a fixed target and aggressively remove variation.
Moving the target around is a research problem, before you feed the result into an industrial process.
Software solved industrialization so well we even forgot that we go from research to industrial with `npm` deployments or whatever your package manager of choice is.
It absolutely can be industrialized: the agent writes tests that conform to your requirements; the harness makes sure those tests pass before work can stop.
Have a look at https://codeleash.dev/docs/tdd-guard for my (dated but still interesting) take on this. Claude Code hooks are really useful to do the enforcement.
It's open source so no need to read my crappy generated docs - you can point your agent at the repo and ask it questions.
The process of anthropic, turning gold into shit software, is hardly something to emulate. What is the evidence software should now be produced in an industrial way? Because arthropic (the ones selling you the shovels) says so?
That’s not how rates work
Regressions are usually the absolute worst.
Up to now we've just been eating shit on the DX side as far as regressions in Codex and Claude Code because everything is all "go-go-go OMG I don't want to fall behind" and also.. Access to new models is tied to harness updates.
But a ton of industries have users that are not so willing to eat the regressions in the name of more features and changes than they can keep up with.
Edit: Also moving too fast.. You may have been less likely to "get it right" than ever before leading to design churn. Again, depending on industry and users, they may not be very tolerant to releasing the 5th complete overhaul of some feature in so many weeks..
Although, following Anthropic’s ethos and taking this to its conclusion, I’m not even sure why they need human reviewers at all. What, they bring in a human to look at a diff of an “important” section of code that was written by agents and has changed a dozen times since last week through 12 PRs of churn? Obviously they don’t know what’s going on. So what’s a human to do? Do what the other approvers (if any) did: fire up a Claude to command it “verify this change has no regression; check carefully for edge cases”, then come back after a half hour of High Thinking to see “Change is clean (with some non-blocking findings below, in order of significance)” and click Approve. Not sure what the purpose of the human is in such a process.
Now we’re just going to outsource our capacity to design and review because it’s faster that way, as if the thing that was wrong with the software industry was that releases weren’t being pushed out often enough.
Authors should really try and view this article from the perspective of someone who has to think about cost to use the models.
I'm glad they're doing this, because it lets me see an extreme of LLM development.
Compare that to the cost of human code review which can take up a third of an engineer’s time. You’re looking at $50-100K a year per engineer fully loaded in code review costs.
If AI assists in code review (not even replaces – we still do human review ourselves), you could spend thousands a month per engineer and still come out ahead.
Ironic that this comment talks about cost without weighing the numbers.
"According to the report, adjusted operating income excludes stock-based compensation, which is not unusual for technology companies. However, the report also quotes its gross margins - above 80% - exclude revenue shared with distribution partners, as well as the cost of training its models."
From https://www.morningstar.com/news/marketwatch/2026091414/the-...
Antropic's revenue is up [1], [2], [3] and if you calculate adjusted operating income (by excluding most expenses like training costs) then by that measure it is profitable. However, if you factor in all the numbers it is operating at a loss.
[1] https://finance.yahoo.com/technology/ai/articles/anthropic-v...
[2] https://fortune.com/2026/09/29/anthropic-leaked-ipo-prospect...
[3] https://www.cnbc.com/2026/10/15/anthropics-third-quarter-rev...
https://youtu.be/T-oXyXwD6sE?si=tauzXVyGmQ6MhI_5
I'm stating that their current business model, prices, and behavior they are trying to instill is not profitable. For example, token maxing every code change is great for Anthropic and terrible advise for Anthropic customers.
The moment they stop training, they become irrelevant. The moment they become irrelevant their valuation crashes down to normal valuation ranges for a company with their revenue and future growth
And what they're reporting is adjusted even further than EBITDA
Some of the most capable engineers in the world high on their own supply… I have no doubt they think this is aspirational but even as I know LLMs will forever change how we code, I’m not entirely sure 30,000 PRs for a coding agent is a volume that makes any sense.
And now this. Suffice to say, my efforts did not bear fruit. I feel more isolated than ever before.
- How was this approved for SOC2 compliance? Our SOC2 reviewer explicitly mentioned that all code needs to reviewed by at least one other human before it goes to production. Claude Code says they are SOC2 compliant (https://code.claude.com/docs/en/security).
- This assumes all engineers will not do anything malicious. What is preventing a Claude engineer from spying on their ex's or some other companies Claude searches by pushing malicious code? Recently an OpenAI engineer was caught planting a recording device at their own office.
Opus 5.5 is obviously stellar but come on what are you doing lol
If it detects they aren't paying attention, fall asleep, or haven't approved enough PRs per hour; off the work at the factory.
I hope this is only for low-risk false positives. Dismissing a critical finding as a false positive should still involve AppSec.
You could improve revert rates while simultaneously introducing more security vulnerabilities.
And the linked "claude-code-security-review" explicitly states in the README
> This action is not hardened against prompt injection attacks
That's a significant limitation for an AI security review control, no?
What are llm freaks in such a hurry for? Its like everyone is manically racing towards a finish line without realizing the finish line is you out of a job. Its like everyone is excited to be completely obsolete and poor.
I'm looking at Claude Code and they chug releases daily and sometimes more often then this, if someone has to stop and code a critical path by hand or smth I'm wondering how this is handled.
--
On a separate topic, the "llm freaks" -- I know you intended this as an insult, but I'm actually proud to be called a freak again, and I thought this is the place where freaks are discussing things.
While Im rooting for Anthropic to collapse under the weight of slop and their own contradictions. There's a leaked interview for Anthropic from earlier this year (I think) for their inference team. The interviewer seemed very competent and the quality of the questions were good.
This article is more likely a marketing gimmick in service of shelling shovels (for devs to dig their own grave) "hey look guys we automate everything with claude, you can too, just give us all your money!"
Its like telling other ppl about your dreams. Not interesting.
Eventually I did, but it was hours and hours of effectively random stuff happening that I had to write down or dictate with speech to text.
It was a waste of time and I'm now glad I forget almost all of my dreams. My point is even my own dreams are uninteresting.
I'm not that interested in my own agent setup either - usually the tools become better on their own after a few months and render it all obsolete.
The less you understand code, the more you're impressed by coding chatbots - in part because you're less likely to read the bot's code, and if you do, less likely to realise how poor it is. Code gets judged good simply because it exists and costed little to create.
Choose something you want to do but don't really care that much about. Be more ambitious than you think you could be, have a high bar on the outcome but purposefully choose to completely ignore the actual code. Make it write tests, dashboards, monitoring - whatever you need other than code to feel confident.
Now is the cheapest time it will ever be to experiment.
I am happy that I have a useful tool, but it simply isn't maintainable without a machine or me investing much time to understand it.
I don't see how the former is a sensible future for humanity. I am happy to say I won't be releasing this software I've created (I will find it useful myself), but deferring all understanding to a black box seems a poor future for brain activity.
I mean, I've had useful software that I don't understand how it works where I am stuck hoping some small company continues to maintain it. And I've had some of those companies go out of business and leave me with useless software that I eventually have to replace once it no longer works on my machine.
Having the code that I can dump into an LLM to understand myself or have it alter it seems better than that.
I tried that. ChatGPT's result was laden with loud, clear and wrong fabications which I might have believed had I not examined the code.
Unreliable program be unreliable.
* The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself. Yes, I can ask it to look elsewhere for similar bugs, but even that's not super reliable.
* The user-facing text it generates is just as awkward as ever. I have to rewrite all of it, and sometimes it slips through. It's also hard to write text when you don't know what the code is actually doing.
* It doesn't have a good model of what kind of information is relevant to humans. It's just as likely to produce a paragraph of random jargon as a clear explanation of what's actually going on. It shows excessive detail for things that don't matter, and not enough detail for things that do. For example, it doesn't understand that $0 can logically mean "null" in terms of how humans think about things (e.g. "This item is free" vs. "This item is $0").
I kind of wish I hadn't gone down this path but now it's far too late, I can't really manually maintain this amount of slop. I'm quite sad if this is going to be the end state of this profession.
I see the same in e.g. simple CSS. I ask ChatGPT to apply styling X to aĺl instances of say a UI control type. Then to another. Later I ask for a tweak of X. Bot updates some but not all instances. It didn't create a single class for all. No intelligence seen.
The most remarkable fails are simpler though. E.g. at version 303 I ask "Revert to version 299". I get 299 with 303's edits. Bot explains it interpreted "Revert" to mean something no intelligent programmer would.
And most importantly, these are just the ones I directly notice. To see the total, I have to read through and understand the most repulsive code I've seen in my life.
The only way to be satisfied with bot code is by having exceptionally low expectations. And never ever let yourself be kidded that your helper is more than a stochastic parrot gorged on stackoverflow etc.
Was this with the latest models (Fable/Opus 5.5 or equivalent) or an older model in a 3rd party harness?
Are you in the camp of: "if at first you don't succeed, never ever try again".
This was a few months ago, so obviously not with the latest models, but:
1. That argument is evergreen. You will always be able to make it, and it doesn’t really add much. It is impossible to refute.
2. Gone back to - when writing code - having a strict mental modal and good control over the codebase. A mix of hand coding (to establish the mental modal) and prompting (for boilerplate). It’s not quite as fast, but the quality is much higher when you mix in using LLMs in quality enhancing capacities (review, debug, etc). This is with the latest model. Hands off still atrophies over time (even the boilerplate), and in some cases is very inefficient (as in wasting my time, nothing to be said about tokens).
> Are you in the camp of: "if at first you don't succeed, never ever try again".
This is great in some domains, but “don’t make the same mistake twice” is much more important in mine. I don’t have the luxury to ship low quality, especially when LLMs make even higher quality so much easier.
If you cannot tolerate failure then don't experiment. Don't mix the two.
Additionally, do you really believe that 'Now is the cheapest time it will ever be to experiment'? Why so? Because of token prices? Not a good omen if so.
I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
God I hope you're right. Maybe then this era of vibe slopping out photoshop clones that'll be abandoned within a week (while advertising them all over social media) will come to a close.
First hit is free ;)
(You're welcome to make your substantive points thoughtfully, of course.)
https://news.ycombinator.com/newsguidelines.html
He sure is smugly impressed with himself for giving you a taste of your own medicine, though. He knows you are but what is he? ;)
Response: Try experimenting in a low stakes environment
From Google: "Exposure therapy is a psychological treatment designed to help people confront their fears, break cycles of avoidance, and reduce anxiety."
It's those who don't do it who'd need a talk..!
However, I suspect there are teams that are making different choices. In the most obvious example, I can do whatever I want on my personal projects (I hope you agree). And if I started a business and hire in programmers, I have some expectation on defining the culture, making hiring decisions based on my preferences, etc.
Whether or not such teams succeed will be a matter of fact, which I do not believe has been decided at this time.
And I say this with the bitterness of someone having lost a big part of what made me enjoy this line of work.
But I have no interest in working as a vibecoder, there are many other jobs that I'd enjoy more
How does "low-stakes environment" fix that?
The person is saying "I am still unable to stop" and that is most likely because they feel a sense of responsibility. If they choose a project where there is no responsibility, where it literally doesn't matter if the project fails, they may feel able to stop. And if they manage to make that project work, if they can succeed, then they will build up confidence.
It seems better to be able to work both on projects where they review all code as well as projects where they do not review any code. In fact, keeping their job in the long term may (or may not) require having both skills.
They are not overblown. LLMs do all the things the person described.
> Or they may learn that they cannot build any kind of software without reviewing every line of code.
We already had bad horrendous software before the onslaught of LLM-slop. Why contribute to that?
> And if they manage to make that project work, if they can succeed, then they will build up confidence.
Confidence in what? It doesn't fix the underlying issue. And the underlying issue isn't "I can't stop" but "redundant code, duplicate code, overly complex architecture, etc." that LLMs regularly produce.
> It seems better to be able to work both on projects where they review all code as well as projects where they do not review any code.
"If I close my eyes maybe this goes away"
Last time I chose to vibe code, I did it in a language I didn’t know so that I wasn’t tempted to try and comprehend the choices the robot made. I ended up with the application I wanted.
Caveat: it was code I would never have to touch again, I ran a script to generate the 140,000 static html pages that the rails app provided and then just uploaded them all to R2 buckets (million reads per month on free tier)
No, they are saying "who cares if all those issues with code are true, just don't review it"
> Caveat:
It's not really a caveat. I also have throw-away code. But we're not talking about throw--away code. We're talking about the systemic issues in LLM-produced code, which are completely ignored and the answer is always "who cares, close your eyes and don't review the code".
Do you think it's better to experiment with high stakes projects? Do you think it'll be easier to avoid defaulting to line by line auditing of a project that is high stakes?
Ah yes. "all the magic the people say work 100% of the time"
> Do you think it's better to experiment with high stakes projects?
This again assumes that people working on high stakes projects only work on one single project.
This again assumes that people finding issues with LLM-generated code have not tried every magic recipe under the sun from people claiming their hair growth elixir is 100% a working solution.
This again assumes that people experiencing something different from the others are inevitably stupid.
etc. etc.
They've built an entire strategy around moving slow and doing it better
Code quality matters to AI just as much as it did to people
AI tends to add new wheels for a new feature in a large codebase, because finding, understanding, and generalizing the existing code to integrate a new feature will likely break existing features, cost more effort (and token) to implement and test. Human reviewers also tend to like these code since they are clean, easy to review, and get the job done, especially when they are not familiar with existing codebase.
The issue is that wheels like these accumulate in the codebase, and they inevitably entangle. When an upstream specification changes, when a new feature that works with many existing features are introduced, all these wheels need to be revisited, and in many cases they are not because the new test cases do not cover all the paths. The code base got larger and more complex, and contains more bugs than the unit tests can cover, especially when unit tests are often "bent" to fit the wrong behavior unnoticed.
https://news.ycombinator.com/item?id=49867306
The worst part about this is that if you aren't watching it, the AI will "solve" this with a thousand assert_3_plus_5_equals_8() test cases that just further clutter the workspace and the context window. Churn baby churn.
If you let it go buck wild on a massive change, there are those times where it sort of paints itself into a corner but rewrites are cheap (for now), so it's just not that big of a deal, imho
Funnily enough, the models learned from GGC and LLVM, yet, cannot produce something half-decent.
Do what you want, write apps with AI and don’t review code, it doesn’t impact me. The claim that no one cares about good code is demonstrably false, and that’s all I’m objecting to.
I do think the capabilities of AI depends a lot on how you work. If you're doing very low abstraction, fail fast defensive programming in something like Rust then it's fairly good at writing code with high quality. If you're working in some OOP language with tons of freedom, then my experience is that it's usually not better (or worse) than most human programmers, but certainly much worse than some.
I've got decades of experience in enterprise development and I've also been an external examiner for CS students. We have people with no coding back ground vibe coding internal tools in Python which are terrible, but not really worse than what we've previously bought from some of the cheaper coding houses around here. At the same time we have AI writing internal SDK's and libraries in Rust which will frankly write better code than I do on a thursday afternoon after too many meetings and too little sleep because the children kept me awake all week.
Good code can mean a lot of things though. If you're deep into clean SOLID DRY OOP programming I suspect that an AI will produce unreadable horrible code, much worse than most humans. On the flip-side the AI probably knows the difference between a generator and looping over a List. It probably also knows more about JVM optimization than most of those banking programmers trying to replace COBOL with JAVA.'
I doubt Antropic is a good sample though. Bun is written in Rust and it's a single purpose software, which seem to be an optimal environment for LLM's.
so long as his software isn't used for anything that can affect you
I do, my coworkers do, my friends do, etc etc.
Your end users certainly don't either, heck they'll gladly sit there and scroll through their phone until the 50GB React app loads.
And if people are not looking it, those people don't care about their customers.
where things like "it tastes good to me" or "cement didn't crumble when i hit it with a hammer" is good enough to ship we'd be inundated with lawsuits and customer claims... its a total joke where the software industry is headed
Similarly, once you pull in terrible AI slop code, the project becomes unmaintainable by humans and you need to maintain it with AI.
This is, of course, tangential to your point.
This is not HN-worthy
> The goal hasn't changed: less human time per change, and no more reverts than before.
Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target. Is this then still a good measure? I guess Revenue is a more direct measure, but we don't have access to that one.
New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.
All in all, I'd say their approach is working.
honestly i'm thinking it might be easier (and cheaper) to launch our own agent from GH webhooks and trigger `/code-review`..
So dbs, seed files, etc, and then just a review prompt and some skills and a few scripts.
It triggers on a new published PR or if tagged @bot-name. It's way better than the other services we've tried (which was admittedly ages ago) because you can get it to work exactly how you want it and it's dead simple.
With Codex, for the time being at least, you can use a subscription, so it's basically free if you're already paying for one. But on API pricing it's still not too bad as it doesn't produce that much output.
I set this all up in about half a day. Codex does 99% of the heavy lifting.
bugbot was costing us $1k per month for a small team. EVEN IF we decide to go with a paid or say something like GPT Sol for our custom harness it'll cost us maybe $100. We've already implemented all of bugbot's interesting features like cross-PR memory.
the cool thing is we can leverage this for things that bugbot cannot do. Just today I added a security-reviewer action reusing almost all of the code that we wrote for the code reviewer.
It's pretty effective and lets you do nice stuff like compare the code to the tickets, reply to comments and stuff like that.
This generally works for me. I think a solid spec is the key for the models to review against. I ask them to follow the spec, and if the spec doesn't say, follow first principles.
Also, if there is a user experience change, they need to notify me.
How about your pattern?
If an engineer paid $150,000 a year reviews some code manually for about 30 minutes, the cost of their time is about $36.
So code review has become newspeak for “testing” and “education”?
Well not really testing. We the users are the testers and it falls to lucky us to really discover and find the errors, edge cases and security issues. The consequence of the rush to continually release…
Is the AI advance in the field of software testing nothing more than “now an AI agent can do a code review?”.
> ...Architecture that sets a pattern others will copy...
I'm happy that even Anthropic agrees this is best left to humans.
Honestly I’m not worried about code anymore in my codebase I’ve not had bug problems for a while and incidents get fixed automatically.
I monitor new features with an agent who checks recordings.
Most actually good features need still 20-30 iterations to design the userflow to actually be interactive, but then there is also another challenge to translate usage patterns into features and how to present functionality with the least amount of friction as possible.
How to efficiently run monitors without wasting your token spends. A best 'monitor' best practice would be nice.
Still this article was very helpful and gave me a lot of new ideas.
I have some bad news about that "monorepo" designation.
Vibe-coding complex software can work, but it's going to take a really disciplined team to manage this properly, like in this article.
Personally, I find Claude's degenerate urge to just spew uncontrollable amounts of code quite counterproductive. Yes I get it, you get paid by the token. I'd like to see more reasoning, more understanding, and less generating.
Truth is, we are still very very far from actual AI software engineering. Some people are working on useful tooling, and (predictably, like anything good and useful) it is taking a long time. LLMs are great, and hopefully they will help us accelerate building the tools that bring this profession to the next level.
30k PRs in a month (or whatever) is not the brag they make it out to be.
How is this a proper product? Every time you open the desktop app the buttons may be on a completely random place
I think a lot of people like myself still want to feel in the loop but I don’t have the right tools to stay on top of the (generally good, productive, and still thoughtful) contributions from peers anymore.
Of course, just like "line of code" is a worthless metric, "change merge per day" is;
(automerged PR, JOB IS DONE)Have you seen the benefit of the churn of 1000 PRs a day?
Ah, so you've privatised the benefits and externalised the costs.
I don't want to use a product that shifts under me as I use it several times a day. I want a product that actually moves slowly and works very reliably.
As an end user: why would I care about how many hundreds of releases a month are being pushed if maybe 1% of them have features I care about?
Speed is not the product.
Human contributions to issues, comments or submissions, were drowning in a sea of AI vomit. Tons of the issues had those AI issue submissions with the classic confidence of having identified the problem, but that smell I know a lot of us recognize of it not actually knowing what the problem is at all because it hadn't failed to fix it 5 times yet.
Worse, everything after the /grill-me is injected a second time into the prompt as a <command-args> so your prompt is literally added to the context TWICE. (first time in the original user prompt.)
I swear Anthropic is intentionally bloating your context so you burn tokens on useless bullshit like this.
https://github.com/anthropics/claude-code/issues/92769
The <command-args> I know because I just audited my raw Claude transcripts via ccglass. You can verify by just asking Haiku "hey, /grill-me does this skill autoload for you?" and then "/grill-me hey, how many times does this text appear in your context?"
You can trivially doublecheck me. The above prompts won't even eat 1% of a 5 hr on 20$ plan.
their claude code changelog 'fixed' % over time.
https://i.imgur.com/KJcem6c.png
I believe their peak month was >13k issues opened, but worth bearing in mind: "This issue has been automatically locked since it was closed and has not had any activity for 7 days.".
Not only that, but AI is categorically terrible at ux review. I don’t mean aesthetics, I mean attention flow, workflow in general and what the user anticipates as a logical path to feature discovery and exploration. Increasingly, features are hidden behind workflows that meet a single use case but fail to be universally logical or in many cases, even accessible at all outside of the test case.
So, people… use AI in code review. It’s actually pretty great at it if you do it right, and you can ship better software with fewer bugs than ever before. But please, don’t pressure your ux guy to “10x” with AI. In fact, now that you are “10x”ing your development cycle, you are probably going to need to hire -more- ux developers if you don’t want your product to feel like slop even if it’s gold under the hood.
On the other hand, I would have expected Anthropic to have a drastically more elegant SDLC.
I used to do it but now I only review code organization and structure.
Mostly I define the product behavior I want and discuss the technical implementation beforehand.
Once that's in a spec I let it build in my durable implementation engine. Automated verification, review rounds from different agent types across different models, issues triaged into meaningful clusters and flagged for false positives before finally being fixed.
That loops until the code is up to spec.
There is no special secret, LLMs need repetition and multiple passes.
They'll one shot bigger and bigger things in their default harnesses but they'll be filled with issues no matter what.
* They can afford frequent CC incidents since their value is having the best models, and 95%+ uptime is still OK-ish.
* The core functionality of CC (the harness "agentic loop") is simple, an if that works then for most purposes CC works.
* CC just stores a bunch of config and history files, there isn't any financial or irreversible safety loss if they mess it up once in a while.
Once you work on something where a minor slipup can result in a huge real world mess, then I don't see how their approach can work.
I've written a harness myself and fair enough, it's more complicated than you might expect.. but surely at this point the claude CLI is pretty stable, and this must mainly be a ridiculous amount of test churn.
Or maybe their team of clankers is just creating busy-work to make the humans think they're essential. That's not unheard of in the human behaviours that they're trained on.
good model
Honestly not sure that there are that many effective changes a day given how little complexity there is to their public facing product.
A random twitter said that, 6 month ago, it was 100% !
DECLINE ! SELL THE STOCK OPTIONS ! (wait, there are none, confused!)
I vibe coded my harness that is superior to CC in functionality, and it still doesn't flicker.
I don't need to hear their bullshit.
Anyway, flagging as misinformation.
Over the last few years we have added mcp tools, memory graph, md files and skills, automated everything to the point that on desktop the system runs itself, besides proceed/approved/explain.
We have 100's of PRs/jobs/projects and Claude builds down the list automated and only asks a few questions point form if needed.
Very impressive now, compared to 6 months ago.
You can view it if interested: www.vekmem.com