Welcome to the Future

I had the opportunity to attend 3 AI-focused events in San Francisco this last week, one run by an investment group, one by my own employer, and one by another company as part of a conference. What I saw in those events was a clear shift, for better or worse, towards the probabilistic nature of AI workflows over discrete, more classical software. This leaves me very worried. Software quality has already gotten worse (citation needed, I agree), and as vibe coding and a "good enough" mentality takes over, it will only drop further.

What's in software?

To me, good software is about certainty. It either fails or succeeds, and I can mess with the set of instructions that run the program to alter its behavior in a predictable way. AI tools do not obey the same rules, as they are probabilistic in nature. Success is relative, and chaining AI answers leads to less and less correct answers.

Classical computing, which is a term I never thought to use before this week and am probably using incorrectly, is about discrete success or error states. "Do this thing, fail if it didn't work, then with the output do this other thing." It's how we're taught to program and it's how programming languages operate. This is the world that I understand: I do something and have the correct failover and fallback mechanisms in place to keep moving forward. Sometimes things just don't work, you need to give up.

There are a myriad of ways that things can go wrong. You could misunderstand the requirements, misinterpret the response from another service, fail to connect to a service, or have some solar rays come and disrupt your program. Except for the last one, basically every software engineer knows how to plan for these potential failure modes and (hopefully) strive to write software that is resilient to them.

But the idea is: there is only the concept of success or failure. It could be unpredictable and it could involve some randomness, but executing your program gives you a definitive answer about whether it believes it succeeded or failed.

What's in AI software?

Contemporary computing (again, making up terms as we go) is probabilistic in nature. AI answers are 70, 80, maybe 90% accurate. Maybe pushing 99% for some of the simpler tasks. This present software engineers with a fundamental mismatch of expectations: the program can run without errors and still produce an incorrect result.

I recently have been trying to learn a new programming language and found that basically every major LLM (ChatGPT, Claude, Gemini Pro) would hallucinate packages and syntax for the language for what I would consider basic things. When I asked for citations to see if the package had been deprecated, it would send me to a tangentially related GitHub repository. To me, this kind of behavior is not "good enough", and it's led to a whole new kind of security threat: slopsquatting. When AI can predictably hallucinate certain package names, bad actors will make actual packages with those names and use them to spread malware.

Even a 95% accurate model is only 60% accurate if you're asking it to do 10 things in a row. Do you know how many tasks ChatGPT is running under the hood when you ask it a question? What about AI agents who are supposed to help you navigate a support tree, or act as a software developer so you only have to come up with English requirements? Based on my experience using these products, these agents are less than 95% accurate. I'd guess 90% for the tasks that I send them. 90% accuracy per task for 10 tasks in only 35% accuracy for the final result. Would you pay someone to do a job that would need correcting more than 60% of the time?

Vibes

This, for me, transitions nicely into a conversation about vibe coding. I build software for a living, and have been doing so for 10 years now. I'm not super experienced but I've been around the block and seen some things.

Let's be clear: it's obvious when someone has vibe coded something, at least in languages I'm familiar with. Like I mentioned before, these agents are nowhere close to reliable, and one of their key characteristics is a lack of consistency. Patterns ebb and flow, from homework-style for loops to shitty documentation (on a getItem function you might see: "gets item") to bizarrely optimized bit manipulation. Even in my workplace we've had people check API keys to repositories; for us this isn't as big a deal since all our codebases are private, but for others...

More and more of these apps are appearing by the day. 2 people I talked to this week had vibe coded some applications, and both of them had their OpenAI API keys hardcoded on a public-facing website.

I also encountered 5 or 6 people who couldn't explain how their product worked. I'm worried that I only caught this because I have some level of experience in the things they were trying to work on. Is vibe coding opening up a new path for grifters and charlatans? Or is it democratizing access to software development?

Can you replace people with AI?

This came up quite a bit at the investor-focused event I was at. Can you make your company "more efficient" (aka: pay fewer people and pay them less) with the correct application of AI? One of the speakers said something to the tune of: AI will let you hire more junior people instead of midlevel people, and your more senior people will get to work on more interesting things. OpenAI's new o1 models are supposed to be "PhD-level", whatever that means.

So you might be able to hire fewer and cheaper employees. But this perspective misses something important, and I found myself rather shocked it didn't come up:

If you only hire junior level employees and supplement them with AI, they will never develop into senior level employees with deep, institutional knowledge.

To me, this kind of argument reeks of consultant-style headcount analysis. Optimize the immediate results and pray you figure out the kinks later. And if we're being frank (even though my name is Simon), it's my opinion that we won't: things will just get worse and worse.

Am I gatekeeping?

The last thing I want to do is gatekeeper software development. I think it's one of the beautiful careers where a formal education is not necessary to succeed (I am an example here), you just need a passion for logical operations and strong attention to detail.

But AI is removing the 2 things that I thought were necessary to succeed in this career. It certainly has me doubting myself at times, and then at other times I find myself infuriated that someone couldn't even both to double check the AI's work. As they say: if you're going to cheat, at least rewrite it in your own words.

But that phrasing itself is a problem. Is it really cheating? I've successfully used LLMs to help debug my code. But more often than not, it hasn't helped me and has usually been absolutely incorrect. Is that because I'm now a senior engineer? Is this kind of solution a net benefit to more junior engineers? Or are they stunting their growth by deferring to the AI overlords? I'll have more thoughts as everything develops, but right now I'm staunchly in the anti-AI camp.

The future

The post photo is from the afterparty for an AI conference hosted in SF this week. When the robot DJ started playing everyone grabbed their phones, filmed a 5 second video, and then frantically shared it out as an Instagram story. There were 5 people close enough to me where I could see what their caption was. One didn't write a caption, but the other 4 all wrote

Welcome to the future!