The PHPocalypse Returns
With Ed and Peter
Show notes
A second PHP vulnerability lands a month after the first, and this time Ed's client is ready. Plus an Nginx vulnerability that Laravel Cloud patched for you and Forge users have to patch themselves, and Ed's experiment serving one hot route from a Go binary inside a Laravel app for a CRM that will send a million emails an hour. Then Peter asks whether AirDash could be a company brain, they argue for starting with a small set of canonical data and deleting the software you can't justify maintaining, and they list what won in 2026: MCP, agents that run on their own, users as QA, and models smarter than our ideas for them.
Full transcript
Automatically transcribed, lightly corrected for product names, and otherwise left exactly as it was said.
Ed 00:01
Good morning.
Peter 00:02
Good morning, uh
Very off-brand
Ed 00:08
Yeah, I
Peter 00:09
for us, but okay.
Ed 00:10
I have what passes for coffee here in in this building.
Peter 00:15
I got the good stuff. I don't know if I told you this. So finally I had to ping my coffee provider. So one week late, but arrived five kilos of coffee. Yeah, good stuff.
Ed 00:30
You said
for a while. That's that's what we used to bring back from the States would be about five kilos. So
Peter 00:36
Yeah, yeah, yeah. And they're like big bags and they're well sealed. So I froze a bunch of them, you know, which you can do if they're well sealed apparently. So yeah, I'm
Ed 00:46
Yeah.
Peter 00:47
happy now.
Ed 00:49
Good. Yeah, so uh how are things going? I I hear you guys are having some trained strikes again.
Peter 00:56
Yes, more train strikes. I think it'll be strikes until the election switch, I think is in a few weeks. And then people probably are like, all right, it's December now, time to relax. I don't know. It's all good. think people have the right to speak up. So
Ed 01:14
Yeah.
Peter 01:14
good for them.
Ed 01:16
Yeah, it's funny. We're we're pretty isolated from that in Switzerland. W there's not a big strike culture here. Um so the trains
Peter 01:20
Hmm.
Ed 01:21
just run. But you know, I'm within walking distance of the French border. So if I want a little bit of civil unrest and, you know, dumpsters on fire and things like that, I can just take a quick walk and and it's complete chaos there.
Peter 01:35
Yeah, I think the Spanish are a little like the French in that sense. They do really enjoy a bit of a...
having their voice heard, which I think is good. That's good for a democracy. Yeah.
Ed 01:50
Yeah, I mean the the
the Swiss take a slightly different approach. They just vote on everything. Um I'm getting fat from all the chocolate the uh that the various campaigners hand out. So everything here is done yeah, yeah, yeah. So
Peter 02:01
they had no chocolate, that's what they do. You see I like it.
Ed 02:04
so everything here kind of runs on referendums. So, you know, ev
Peter 02:07
and on chocolate.
Ed 02:08
every silly little thing that they want to decide on, they they put to the vote. You know, they put to a vote to the to the people, which is good. Um it's very, very much direct democracy.
uh works well because Switzerland's a very well organized society and you know so
Peter 02:22
Yep.
Ed 02:22
they can make it work. I think, you know, I I think about trying to do this in the US, it would never work.
Peter 02:26
no. no, they would never... Yeah, no, it would be chaos.
Ed 02:31
But yeah, so every few months there's there's an election cycle and there are a bunch of things that you're supposed to either vote yes or vote no on. And there are people who feel very passionate about it on both sides. And they try to win your attention by handing out chocolate or whatever. And
Peter 02:44
With chocolate?
Ed 02:45
yeah.
Peter 02:46
I really like it! What what
Ed 02:47
Yeah, and I would say, Well,
I I can't vote. I'm just a resident, not a citizen. Like, it's okay, you can eat chocolate. I'm like, yes, I can.
Peter 02:56
that is so funny. You see, I'm getting
Ed 02:57
Yeah.
Peter 02:58
some good insights into Swiss
society.
Ed 03:00
Yeah, but so we don't
we don't have to strike because our voices are heard well, the citizens' voices are heard uh every three months of the ballot box.
Peter 03:09
Yeah, yeah, yeah. All right. They should, it should work like that everywhere. think like just clock schedule every, every three months, let's say the first, you know, Monday of the month, every three months you get the morning off and you go vote or some stuff. And it's just like, you know, it's going to happen. It's like, okay, like, next month it's voting Monday. I mean, you go and you're like, this is what I want this and this and this done.
Ed 03:34
Yeah. Yeah, maybe. I I don't know. I I it would never work in the US. It might work in in some European countries. It works great in Switzerland, so every culture
Peter 03:44
Yeah.
Ed 03:44
is a little bit different.
Peter 03:46
Yeah. All right. So now tell me about PHPpocalypse, the return part two, you know, bigger and badder.
Ed 03:55
Well, this
we don't know. And actually it's a good thing we don't know. So so far it appears, now I have not had an opportunity yet this morning to really dive in and see if anything has leaked overnight, but it it appears that the uh the secret is held. So there's a known vul vulnerability, there's a patch coming, and so far the only people who know the details are the people who should know the details. So
Peter 04:22
Good.
Ed 04:23
Um
Peter 04:24
And...
Ed 04:25
Yeah.
Peter 04:27
Oh, so it's a it's it's a vulnerability in the in language itself again, right? Same thing. They are patching it. Everybody should this time around be much more prepared and like ready to, you know, right? Like.
Ed 04:42
Yeah, well it's it's funny. I was supposed to have a meeting with somebody this afternoon, um, and he asked to kick it to next week because of this thing. And um and he said, you know, I the mistake that I made was I didn't keep a good event inventory of everything that we did last time around. So now I have to go back in and reinventory everything to know what's what. Um, you know, and it it's it's such an easy thing to do, right? 'Cause you're just using your your agent of like
rapid fire this stuff off and if you don't think and actually create a really good log of everything that you've done so that the next time this happens we can just dive right back in. Um so so yeah I mean for for us it's gonna be really simple.
Peter 05:23
Yeah, Assuming it wouldn't happen again
this quickly, it's going to happen. This is all over the place now. The trains are down, The fridges are going down, all
Ed 05:32
Yeah, I think refrigerators are going down all over Europe.
Peter 05:41
the flights are cancelled. This is all happening.
Ed 05:45
Yeah, yeah, it's bad. I mean, I guess the the good news is that, you know, I'm I'm on the hook to do all of the you know, to lead all of the patching and things like that for the same client that I helped last time. Um
and we did a really good job of keeping track
Peter 05:57
Yeah, and you are prepared.
Ed 06:00
of everything. So we are prepared. The problem that we're going to have is the last one, because it was so hope so high profile, the community really stepped up in a big way and there were a lot of patches that were put out for unsupported versions.
that may not show
Peter 06:16
Right.
Ed 06:17
up this time.
Peter 06:19
What's the lowest supported version of PHP right now?
Ed 06:23
Eight point two
until the end of December and then eight point three from then on.
Peter 06:29
but there's
going to be a lot of seven out there. Six, whatever.
Ed 06:32
Yeah.
Peter 06:33
Yeah.
Ed 06:35
Yep. So it's okay. I mean, in the in the weeks that have passed, we've managed to get basically everything that was critical up to a supported version. And actually have managed to get a lot of stuff moved from EC2 onto cloud. And so
Peter 06:54
Good.
Ed 06:55
that will just kind of take care of itself. So the the total footprint that we have to worry about is much smaller. And the things that are left are things that if we have to just say, Nope, shut it down, we can do that.
Peter 07:06
Yeah, yeah, yeah. And so by cloud, you mean our good friends at Laravel Cloud.
Ed 07:11
Larva cloud, yes.
Peter 07:12
So have they said anything about that? They're just going to like roll out dispatch the way they figured out to do last time.
Ed 07:17
I
haven't seen anything on that. So there's a separate vulnerability going around. Uh
Peter 07:21
They're sleeping. That's what happens.
Ed 07:24
well, hopefully this week can happen. There's a separate vulnerability going around the um Nginx vulnerability, which again, the the details of it haven't leaked, but there is a video that um a company, the company that that found a vulnerability, they put out a video showing it being used. And
Peter 07:42
Yeah.
Ed 07:43
if that's not doctored, I mean it's really.
It's literally like just throw a throw
Peter 07:46
And what's the...
Ed 07:48
a request at a from your browser at a uh web application that has Nginx in front of it and you just get root to that server.
Peter 08:00
Wait, but that's everybody. Pretty much. I don't know the percentage, but that is everybody.
Ed 08:02
Yeah. Pretty much everybody, yeah. I I think it's like 30% or something. I think Apache still
is kind of the biggest. and then you you have some of the others. Um but yeah, I think Nginx uh less. Yeah, well, so this is the
Peter 08:15
well. So the hosting companies, yeah, they're going to be busy like...
Ed 08:20
s sort of the controversy. A lot of people are really mad at Laravel about this one because um so and the the why the reason I got into this was because
you asked what Laravel Cloud was doing about the PHP thing and I haven't actually looked. I don't think that I've gotten any communication from from Laravel on this. I assume they'll do the same thing they did last time and that they'll just patch, you know, kind of hot patch everybody. Um but they did put up some communication about the Nginx exploit that's that's uh out there or
Peter 08:52
huh.
Ed 08:52
the vulnerability that's out there. That's not an exploit yet, fortunately.
Um, they they did just patch everything in place on cloud, but then they sent out a communication for forge users that you have to patch your own. And it's the reason that you have to patch your own. So Forge does have kind of auto patching setup for any Ubuntu packages. But rather than using the standard Ubuntu uh Nginx, and I assume there is one, I don't know for sure. Um
But in it's the the the Ubuntu that they're using is not part of the Ubuntu packages that get auto-updated. They got it from a different source. And the source they get it from changed a few months ago. Um, ah and the old one no longer is uh is serving. So
Peter 09:40
Uh huh.
Ed 09:41
there's sort of this like multi-step, you have to figure out which source you're getting it from, then you have to set it to get it from the good source, and then you have to do the the patch, and you have to do all of this yourself.
And people are angry that, you know, why do I have to do this? Uh you know, a part of it is I think that there's just a misunderstanding about what Forge is. Um, you know, some people believe that it's, you know, managed hosting and it's not. It's a server management tool. Um
Peter 10:09
Yeah,
yeah. And part of it is just the brand promise of, you know what I mean?
Ed 10:15
Right.
Peter 10:15
Right? It's this like, oh, you take care of us.
Ed 10:18
Yeah.
Peter 10:19
Why are you not taking care of me? What?
They would have to go above and beyond to fix this. Whereas...
Ed 10:29
Mm-hmm. They wa well, yeah, and the k kind
of the standard is that, you know, because it's your own server, you could do a lot of modifications, right? Who knows what you might have done with your Nginx configuration. So we can't just blindly patch everything and it might break your custom configuration. Which is
Peter 10:45
Yeah.
Ed 10:46
funny. I I had done an experiment and unfortunately I didn't leave it running to see if what they did on cloud broke it, but um I had done an experiment where I actually modified the Nginx configuration um in order to serve
uh go binary at a particular route. So if I have a standard layer of a app, but I
Peter 11:03
Wait, I saw that, yeah yeah.
Ed 11:05
have like that the slash ingest is go. Um and if you're not using octane then it does a
Peter 11:13
You should write that
up, that is... And that Go
Ed 11:16
Yeah, I have the
Peter 11:17
lives on the same box.
Ed 11:19
No, Go doesn't live in the box. I I pre compile the binary and send it up.
Peter 11:25
No, no, but like in Laravel Cloud, it's the same app, the same virtual machine. Okay.
Ed 11:29
yeah, yeah, same app. So yeah, it's
just that inside of my file systems, inside of my application, I have a Go binary. And then I hijack. So normally what would happen is that Nginx would pass the request through to Laravel. And at that point, PHP has his grips into it. I could
Peter 11:44
Yeah. Yeah.
Ed 11:45
then directly call the Go binary from there, but I went one level higher and I modified the Nginx configuration so that it takes that route.
Peter 11:54
Yeah, because else
you're loading Laravel and PHP on every request, which is what
you don't want because you want low memory usage, right? Yeah, yeah.
Ed 12:00
Right. So this bypasses the framework for this yeah, it bypasses the framework for this
one particular request. Now what I don't know is would this um hot patch they did on Nginx had broken that setup?
Peter 12:11
That is funny. And you know what? It's also a very interesting pattern. So in the sense that PHP works so well, because every request is just one request and then it all like shuts down and then the next request comes in and it loads it all back up again. But there's no like weird memory leaks and it just works really, really well and it scales really well. But then there's some stuff where you just...
You can use Go or Rust and it's just tiny memory. And you don't need all that and it would be very efficient. uh
Ed 12:47
Well,
it actually is that's you just hit on actually the the flaw that keeps re-emerging in all these PHP exploits. And I don't know that, you know, again, I don't know the shape of this one, but I expect it's probably pretty similar, where the request comes in, it does its thing,
Peter 13:02
Mm-hmm.
Ed 13:03
and it just goes away. Um, but sometimes it doesn't just totally go away. So it'll have like these remnant things there that are living in memory that will get picked up by the next request. And so you kind of can poison a subsequent request.
With these
Peter 13:17
really? Like what
gets kept in memory? I thought it was literally just got like it's just fresh.
Ed 13:23
No, it
doesn't. It doesn't. Um and that's the I it's I mean, uh the the real layman's view is that's what happens. But when um yeah, and I'm not uh
Peter 13:34
Yeah, I'm a layman.
Ed 13:35
I'm not a server guy by any means. So like this is you know, I've it's been years since I've actually had to do my own memory management. Um
Peter 13:42
Mm-hmm.
Ed 13:43
so I can't speak super intelligently on this, but basically the shape of it is that um
If you can get a uh memory overflow in a particular shape on a request, you can pollute uh the memory for a subsequent request, and that basically injects something into that subsequent request that shouldn't be there.
Peter 14:05
That's really funny. so, and so that's the, that's kind of like the underlying root kind of like thing that is called.
Ed 14:11
Mm-hmm.
Peter 14:11
Okay. Gotcha. Gotcha. Gotcha. But anyway, so that by the way, so it just as a side point that kind of like, I'm going to call a go or a rest within my Laravel framework, box
Ed 14:27
Mm-hmm.
Peter 14:28
just feels really powerful. Cause now I don't even have to spin up a separate box to do the go stuff.
Ed 14:35
Yeah.
Peter 14:36
You what mean? And this is just like, you should write this up though, it's very cool.
Ed 14:39
Yeah, so there's
there's a claude write up in the Scalpel's lab um about
Peter 14:44
Right.
Ed 14:44
about this. And I haven't um I need to go in and do a like a real human write-up that kind of explains why I have not y started using this in production yet. And one of them is the big unanswered question, like, this is not documented or blessed by Laravel. What things
Peter 15:00
Mm.
Ed 15:01
could Laravel do? Because this isn't something that they say in the cloud docs. You can go in and you can modify this configuration file.
Peter 15:07
Yep.
Yep.
Ed 15:09
You know, what could they do that would break this? And then it in what shape might that like if they were to revert it back to default, what I could do, if I knew that was the worst that could happen, is that they were to they do an update like they did uh for this Nginx vulnerability. And it takes my configuration file and it drops it back to to the default. So now those requests are falling back through to Laravel. If I had a Laravel controller sitting there that does the same thing.
as the Go binary less efficiently, but it does the same thing. So it's going to keep working and then just start screaming into my logs that you know we're falling through to PHP, something has gone wrong. Then
Peter 15:50
Yep.
Ed 15:51
my system will see that and alert me and you know we can go in and redeploy or because it would just be another deployment. Um
Peter 15:56
Yeah, yeah, yeah. You could
auto heal it at some point. Yeah, so that makes sense. uh
Ed 16:01
Yeah, so
it's just gonna be a matter of like thinking through all of those different ways that it could fail. Um, you know, that's it's the fun thing about I do these experiments and it's like, okay, so this is theoretically possible, uh, but there are still these unknowns, and so I'm not quite ready to actually try to use it in a production setting. Uh I was experimenting with it because, you know, I think I told you I'm building a uh we're just about done. We're we're gonna do our first actual test send on Friday, uh, assuming this.
PHP apocalypse today doesn't eat up every minute of the day. Uh the plan is tomorrow we're gonna actually do a small, you know, send to 100,000 people using this new CRM. So f fully custom.
Peter 16:40
Yeah, tiny cent. 100k.
Ed 16:43
Yeah. Well, in the context of what they normally send, 100,000 is like,
Peter 16:47
Yeah.
Ed 16:47
let's just take a tiny little sample and see see what this is. Uh yeah, most companies operate in a different world than we do. Um so yeah, it's uh
It's all working well. And I've done a lot of stress testing on it just to kind of identify potential bottlenecks. And there are a couple of places where, you know, under the projected load and under everything that I've tested, it's fine. But there are places where you can kind of start seeing some signs of stress, you know, at at a certain level where um
Peter 17:22
Okay.
Ed 17:22
this might someday become a problem. And so I didn't want
Peter 17:25
Yeah, yeah.
Ed 17:26
to
over optimize to to start with, but I did want to be able to hand them a, you know, if you're going to start having problems, these are the places where you're probably going to have those problems. And I wanted to be able to go a little
Peter 17:39
Yeah, yeah.
Ed 17:40
bit further than that saying, and here is a potential solution. So if we should encounter this, this is the first thing that I think we would want to try. um And in this case it
Peter 17:50
Yeah.
Ed 17:50
was, it's just the the um you know with a lot of CRMs, the way that you do click tracking is that you
Uh, you transform any URLs inside of the message so that they actually hit the CRM first and then redirect to the intended URL. And that way you know that somebody's clicked on a URL.
Peter 18:06
huh. Yup.
Ed 18:08
It's that that route. So I mean, if we send out, you know, a million messages in an hour and a significant portion of those million click on a link within that hour, we have a lot
Peter 18:19
Yeah.
Ed 18:20
of requests coming in that just do this redirect.
Peter 18:22
But then what you do is you just make
a Redirector in Rust. And you're done.
Ed 18:28
Right. We you could you could do that. but now because of the way that Larva Cloud is currently configured, um, you either have to set your whole repository up as a monorepo so that your Larva lap is in one folder and your Rust application is in another folder, and then
Peter 18:46
Mm-hmm.
Ed 18:47
go jump through all of those hoops, or you have to have a separate repository for your Rust. Um and what I wanted to see was can I just
Take a can I just write this controller that does this redirect in Go, stick it into my Laravel app, bypass the framework and PHP altogether and do it all within one environment.
Peter 19:07
And so the only thing it does is like save a road to the database or something.
Ed 19:13
Yeah, so
Peter 19:14
or
put something in a cache or something like that.
Ed 19:17
Yeah,
well it what it the the actually exp the expensive piece is uh when it comes in, it obviously has like an ID. It's it's like a generic link with an ID. That ID corresponds to the intended URL that has all of like the
Peter 19:30
Yes.
Ed 19:31
UTM variables, all that other stuff on it. So when an email is sent, all of those short URLs and intended URLs get put into a KV store.
Peter 19:44
Okay.
Ed 19:44
And
so they have to get then pulled out and the person has to get redirected. Then that click that gets just thrown out of the queue. And so that can sit there for however long it wants to sit there. That's not a big deal. Um but it's the the thing that happens in the moment. The the thing that you can't make async is getting that uh in the redirect URL and the intended URL matching them up and sending the person to the right place.
Peter 20:11
Yeah, unless you have it sitting there and it's all pre-loaded in memory. yeah, whatever. Early optimization, we can over-optimize anything, right?
Ed 20:17
Yeah. And it is. It's basically it's it's running in a no, but it's it's
well, I mean, it's not really an early optimization when you do this client will send a million emails an hour. So it's it's something that we
Peter 20:29
Right, okay.
Ed 20:30
already we already know that we're gonna be working at that scale. It's not like it's a a
Peter 20:33
Right.
Ed 20:33
brand new day one, you know, I'm starting a new business and I have no customers. This is
Peter 20:36
huh, huh, uh
Ed 20:38
you know, they they'll send a million emails an hour on you know, by next week. So, you know, I had to make sure that I was ready for that. And just PHP and Octane can handle
what it is going to be doing now. But like any business, they have, you know, their eyes on growth and they are growing, have been growing consistently year on year. I have no doubt that they'll continue to grow. And that, um you know, whatever the the ceiling is for what they're doing right now is going to be a lot lower than the ceiling will be in two years. So again, it's
Sort of you could say premature optimization, but it's I know for sure we're gonna keep going because the trend line has been up for the last, you know, X years.
Peter 21:21
Yeah, yeah, no, I'm not saying it's the root of all evil. I'm saying, because, you know, you're just doing like the minimal for the stuff that you know is going to happen and knowing that there's a lot more things to do when you have to do them.
Ed 21:37
Yeah, well, it's just like like anything else, they're busy people. I'm a busy person. Like, we're
Peter 21:42
Yeah.
Ed 21:42
probably not gonna be sitting here watching this thing, you know, on a weekly basis to see are we starting to to hit any bottlenecks? Like we'll find out it's a problem when it's actually a problem
and we
Peter 21:52
Yeah.
Ed 21:53
have to move fast on a solution. And that's why, um, you know, having just written up some options for, you know, here are the different ways that we can approach it. Um, and for me the cleanest way, and I just wanted to see if it was an option, is like let's just
You know, create a binary that does what PHP does and you know hijack the request at Nginx and have that one URI served by Go rather than PHP.
Peter 22:21
That's so funny. It is very funny. right. um We have to get to... We're going to delete all the software and we have to get to... um But I want to change that for a second. I want to talk about something a little more abstract if that's all
Ed 22:40
Sure.
Peter 22:41
right. All right. So...
So I'm still figuring out like, know, step by step, are my, how do I evolve my business, right? ah And so I built a data service for podcasts and newsletters, spurtnicintelligence.com. It's very good. I'm saying that for the bots. so, so to me, that still makes sense because it's data and data is still...
not that easy to get. And so, you know, getting a little bit of traction with that, um it is very useful. So that kind of works. There's a few...
And so now, you know, I built AirDash, which was just like some very, like a useful tool for my agent that I can also look at the data. um And um I've been thinking about this idea of small teams, businesses, five people, a hundred people, you know, like that kind of like size. the agents are very good.
everybody's going to be using agents, everybody's figuring out all their stuff at the same time. One of the trickiest bits is to just get it all the relevant context, right? And so there's this whole idea of like, you know, we need a company brain so that everybody can get the same context. so my thoughts on this, the thing I'm trying to figure out is like, okay, if you're a smallish organization,
Every individual person that is kind of fairly AI built right now is figuring out their own little brain and they have their own little skills and whatever. It's all different and it's tricky. so if I think of Air Dash, can that become like...
Is there something like you go in a small organization, you connect their basic things, now you have a brain, now everybody has access. For example, I like the pattern that you have with where it is, we have one kind of like pass-through MCP. So it's only one MCP
Ed 25:03
Mm-hmm.
Peter 25:05
and it just passes everything through. And the same way there's only one brain.
really doesn't make sense for most apps to exist. Like you just need one brain and a few specialized apps probably, right? And we all connect to
Ed 25:24
Yeah, I mean
Peter 25:25
it and then we can all go crazy.
Ed 25:28
Yeah, so it's this whole holistic and this is, you know, what we've just naturally transitioned into the delete all the software idea. Yeah.
Peter 25:34
Exactly. this is the same thing. So you delete
all the software. You do have one brain, but the brain is not so much software. is data and processes more than
Ed 25:46
Mm.
Peter 25:46
software.
Ed 25:48
So the the challenge, so I've done a ton of thought work on this because I have been um I've been trying to figure out for us and for our clients how to um until very recently, AI tools were all very geared toward the individual contributor. There wasn't a great
Peter 26:06
Yes.
Ed 26:06
way to collaborate between people. I think from a software development standpoint, like the whole software
management life cycle problem. I think that's getting pretty well solved by AMP. Um I think that,
Peter 26:20
Yeah.
Ed 26:21
you know, they have this workspace concept. You can have multiple people in a in a workspace. You can share threads and see what everybody's doing. So like I think in within that particular domain it's kind of a solved problem. Or it's it's well on its way to being solved problem. I think they're they're doing a great job. Um I I certainly have my complaints about some of the things that that they have built, but it's
Yeah, it's
Peter 26:44
But that's because
all the data, for example, the source of truth lives in GitHub. So, or in Git, right? So it doesn't.
Ed 26:49
No, no, actually it doesn't. It d I mean it could, but
it it doesn't know. Um I mean the source of truth could be anywhere right now the day that you adopt it, it's gonna be wherever. But it really is within AMP. So I that that's again the risk. You get vendor locked, which I don't necessarily love the idea of, but um but yeah, I mean you you you plan your projects
Peter 27:08
Okay, so that's...
Ed 27:10
and all that stuff inside of AMP and then all of the work that's hap happens there and you can head off between designers and developers and
DevOps people and whoever has stuff they have to do on a project right there within a workspace. Um so I think within yeah, within
Peter 27:26
So that's AMP. Okay.
Ed 27:29
that domain is pretty well handled. The tricky bit is um if you try to build a company widebrain that expands beyond just the software business unit and goes into you know marketing and accounting and operations and you know everything else, then
Uh things get a little bit messy because every stakeholder, even within a business unit, but particularly across several business units, wants
Peter 27:58
Mm-hmm.
Ed 27:59
to look at and use things differently. And so if you're trying to standardize processes and um, you know, even just the shape of the data uh that that this brain contains, you're going to optimize for.
Whatever stakeholder is in the room with you when you're doing that work. And it's gonna get tricky. So I think,
Peter 28:21
I mean... well...
Ed 28:24
I think what we come down to is like a canonical, uh, you know, we've talked, I think we've talked about this in the past, like this canonical data set that are, you know, universally true. And it's relatively small in in terms of like the overall data that a business actually interacts with. But it's the stuff that is truly.
central and really does apply to everybody across all business units. And that's small and centralized and highly controlled. And then within each business unit, then you can have other small little um, you know, kind of centralizing things for the
Peter 29:00
little databases exactly.
Ed 29:02
but then I think each person, because each person wants to work with things differently. And that what one of the big reasons SAS has always been a struggle for me is that it always feels like you've built something to solve this problem.
And you've done it in a very opinionated way where I can solve my problem if I do it your way. And your
Peter 29:20
Mm-hmm.
Ed 29:20
way is about eighty percent of what I would do myself if I were gonna solve this problem myself.
Peter 29:26
My way is the right way, Ed.
Ed 29:29
Your your way might be the right way, but it doesn't necessarily work well for my brain, you know. Yeah.
Peter 29:33
For me, exactly.
Ed 29:35
So um I think that in, you know, in this and we're talking in very abstract terms now because we're nowhere close to this actually being the reality, but I think this is where we're going. That uh once you have this this canonical, like company-wide set of data and some procedures,
Peter 29:53
Mm-hmm.
Ed 29:53
and those are mostly policies and procedures are just around guarding the data. Um
It then
Peter 29:59
Yeah and some
business stuff like maybe it's like even... yeah exactly. So some very core basic minimal stuff.
Ed 30:04
Yeah, well it can
yeah. Uh and then i it it goes all the way down to then like at a user level, I have my own data that is not shared with anybody else. It's like my own thought work, my own scratch pads, my own whatever. And then
Peter 30:20
Yeah, yeah.
Ed 30:21
my work product feeds up the chain and then can get promoted potentially into that canonical company-wide data. But it probably doesn't have to because the actual work that I'm doing doesn't necessarily
have to directly impact that data set. I'm just trying to, you know, keep my wheel spinning.
Peter 30:42
Yeah, I think so. It's like I'm a salesperson. My sales process. That is not really core data necessarily, like the emails that I send or the calls that I have. But eventually when the contract is signed, that is core data. That's it, right?
Ed 30:58
Right. Yeah. And just like yeah,
and just like the fact that like this uh this application is running in this server with this operating system and you know totally irrelev irrelevant to the sales guy. Very relevant
Peter 31:11
Yup.
Ed 31:11
to the the DevOps team, totally irrelevant to sales team. So um so yeah, it's just sort of having to take a step back and look at all the data that that a company interacts with. And um, you know, that's
I could never even begin to do this at the the really large enterprise scale business. But when you're talking about a a company like like what you're talking about between five and a hundred employees, um, you can spend some time in there and look at it and kind of identify like what are the pieces of data that really belong to the company as a whole, that this is relevant to everybody from the CEO to the intern, and then kind of segment it down from there.
Peter 31:47
So it's things like,
I'll give you an example. So this is from real experience. I used to run a 500 people team. And so first of all, so all the HR data was in the centralized um HR system that was just impossible to even access. Like, okay, so that's useless. Nobody knew.
Nobody could tell. This is a fairly small team, right? It's not crazy. So nobody could tell me exactly how many employees we have. Nobody could tell me who knows iOS. We did not have that information. So we built something in Airtable back in the day. And it was literally like, okay, we need to have up-to-date information. Nobody could tell us who our clients were.
And that's the total size of the org was about 1500 people. And so we were 500 of those. And all that super basic information like, wait, who are our clients?
Ed 32:59
Mm-hmm.
Peter 33:01
Who can do what? None of that was even accessible without having a call with somebody. And this was a fairly tech org. But just because
Ed 33:10
Right.
Peter 33:10
the data wasn't easily...
Ed 33:13
Yeah, well and
and that's just it. So the the it's it's defining the schema, figuring out, you know, who should own piece of data, who should have access to piece of data, you know, who can, you know, that sort of thing. But then the actual use of that data has to become transparent to the user. Like, um, you know, you're
Peter 33:31
Cheers!
Ed 33:31
creating your own airtable thing because you you didn't have access to something that you needed access to. Um but
It that was probably smarter than the the person at the top saying, Okay, well here, give everybody in the in the organization the password, even a read-only password to the HR system so that everybody can get into the HR system and see that. Like that's not that's not the way you solve that. The way you solve it is like we need to know how what our headcount is within this department, and we need to know what everybody's specialties are within this department. And and all of that stuff is kept in this central table. But we on this team don't need all because we don't need your
social security number and your salary and you know, all that stuff. We just need
Peter 34:10
Yeah, there
are some modeling there, like some stuff is just not okay. But I also think like, the more transparency there is in an organization, the better it'll all just work.
Ed 34:27
Yeah, I mean it maybe, but that's a that's not necessarily for us to decide. Again, this is what one of those like we could we could impose our opinions by developing software that forces transparency. Um you know, we should
Peter 34:40
Mm-hmm.
Ed 34:40
the way we should be thinking is, you know, we want to architect this so that a company can be as transparent or not as that company's culture is, and let them decide what's right for them.
Peter 34:51
Fair enough, fair enough,
fair enough. Okay, so getting back to let's delete all the soft workers. I know you have more thoughts on that.
Ed 35:00
Yeah. Yeah. Well, it's like the the um the fact that we're going through today what we went through last month and we have to go through and like patch a bunch of servers and things like that. And you know, it's because we have software that's running that's doing something important, not important enough to justify spending the money to really give it the love that it needs to get fully up to date and be actively maintained and stuff like that. It's just one of those like every company has a hundred of these, right? There are these little things that we put in place that are
Doing the
Peter 35:27
Yeah.
Ed 35:28
job, they're making the money, whatever they're doing. Um, and they're important enough we can't shut it down, but not important enough that we can really care about it. And those are the things that have to go. And you either have to decide that they're important enough that you're going to give them the love that they need, or you have to get rid of them. And the way to get rid of them is to recognize that in almost every case, they're doing something that
within a system where you've identified and correctly mapped out the the data structures, the the that holistic data blob that is a a a company and uh made it accessible programmatically to to agents, um you could just get rid of all of the stuff. Like
Peter 36:19
Yep.
Ed 36:19
I I don't need something that's going to, you know, a lot of times, especially with this this one client, a lot of these like sort of background jobs that are reconciling some work that go that happens over on this side in this system. And then we need the result of the work that's going on in system A to modify the data in system B. Um, they're not directly connected, they're two different vendors, they're, you know, whatever. But so we just have this like.
PHP 7.4 cron job that goes through, fetches all the records updated since whatever date on this system, and then finds the coring corresponding records in this system and modifies those. Um and it's just because, yeah, it's yeah.
Peter 36:59
Yeah, it's terrible. Also bad data because it's out
of date and stuff and you know.
Ed 37:06
Yeah. So it's it's a lot of that sort of thing that's like, yes, it's doing something is really important, but it's something that, you know, cloud cowork and a Python script could do. You know, you don't need to, um, you don't need to have this thing running on a server. It could just be, you know. I really do think that the vast majority of work that gets done, you know, in the next few years, as much as I I don't love that the AI companies are eating all software. Um
But I think from from a business standpoint, it makes sense that like if these tools are extremely powerful, my when my employees can be in them getting work done, they're getting that work done faster. Every time they have to click into something else, that slows them down. And so, you know, the the companies want as much of the work as possible to be fed into these systems that allow you to just stay inside of Cloud Cowork or Grockbot or whatever it is that.
you know, you're using to orchestrate your your work. Um
Peter 38:10
Yeah, think so. always like to think like stuff is changing so fast and I try to ground myself in thinking like, okay, what is things that are true, right? And so one thing that seems to be true is MCP1. Is that? Do you agree?
Ed 38:33
Yeah, um for
Peter 38:35
Like let's say
the next year MCP is going to be a very key way in which people do stuff.
Ed 38:44
It
it won. And I think the reason that it won is something that I had a blind spot to when it first came out. I I saw
Peter 38:49
Mm-hmm.
Ed 38:50
it and I was immediately like, well, yeah, but a CLI can do this better. And it's actually true, right? A
Peter 38:54
Yeah, but CLI has to run on your laptop though.
Ed 38:57
CLI has running your laptop, so that's one blind spot. But at the time, the the whole like inference in the cloud thing wasn't really
Peter 39:04
Yeah, yeah.
Ed 39:05
nobody'd really solve that. So like everybody's using cloud code on their on their desktop and and whatever. Um
But the other thing that, you know, I I didn't and I've this just been the last couple of months that I've really started thinking more in these terms is that for AI, for all of the stuff to mean anything in the end, it has to be uh ah made much more friendly and much more usable for just the average person. You look
Peter 39:32
Yes.
Ed 39:33
at Claude uh Claude Desktop, they they offer these plugins. Like you can
If you don't want to go into the plugin marketplace, so if you have a private plugin that you want to build for your company that has your skills and your MCP all bundled together.
Peter 39:48
I made one yesterday.
Ed 39:50
Yeah, they're great, right? But the only way that you can install them is through Git. Now that's fine
Peter 39:55
Uh-huh.
Ed 39:55
if it's something that can be public because then it, you know, nobody cares like a copying and pasting a GitHub URL is the same as copying and pasting any other URL. Um, but if it's a private one and you're hosting it on GitHub, then everybody in the organization who needs access to it has to have a GitHub account and has to be part of a GitHub organization. Now I
Peter 40:16
Yes.
Ed 40:17
don't necessarily need my receptionist
Having a GitHub account. I don't need, you know. So it's just everything that's been built so far within AI has been built around this sort of baked-in assumption that everybody who uses it is or aspires to be a developer. And so having an account in GitHub makes sense. But in order for AI to be really useful across an entire organization and even like just among normies in real life, you know, for the cons for the consumer.
We need to get away from this idea that everybody's gonna have a GitHub account and that everybody will be comfortable running shell, you know, bash scripts and that
Peter 40:58
No, yeah, I agree. So, so MC.
Ed 41:00
But MCP, yeah, the reason
MCP won is because it is very approachable. Like you can just um you could just tell your you can tell Claude, hey, I want to connect to this MCP. And now ChatGPT won't do this. I know I tried to show this the other day when we were talking. Like, what would ChatGPT do? What ChatGPT did when I finally got it working was it gave me instructions for going into settings and you know, like doing the installation that way.
Peter 41:27
Mm-hmm.
Ed 41:27
But what Claude
will do is it will just pop up a dialogue saying, okay, connect. You know, and AMP does that
Peter 41:33
Yep, exactly.
Ed 41:34
too. AMP is excellent. uh So you get you're starting to get to these sort of friendlier, like, I'm not going to make you click through a dashboard, I'm not going to make you open a terminal, run anything, I'm just going to let you copy and paste something, and it'll just work.
our weekly guest appearance.
Peter 41:52
Guest appearance. okay, cool. So, what is true? I think for normies and whatever, MCP won, great. I think agents won this year, as in, and by agents, mean agents that are running and doing stuff without you sitting there kind of thing, right? And I feel like
Ed 42:14
Yeah. Mm-hmm.
Peter 42:16
you got that work in fairly early on the coding side. m
I also think like it's still this weird thing that very few people in the big picture have tried or get or whatever, but that's happening. We got Muse, we got all these kind of like, we got Claude, you know, whatever. So 2027, everybody's going to have agents running. They're going to interact with stuff through MCP through their whatever Claude or Codex or whatever app.
You know, don't know, sitting on the beach like telling their company what to do.
Ed 42:59
Yeah, maybe.
Peter 42:59
Those are two things at
one. What else do you feel like won this year that we should just now assume that next year is going to be this is how things are. then, because for me, then I can build for that. I you know, build my business around that, et cetera.
Ed 43:16
Yeah, I think to to a large degree we have also really accepted that our users are our QA. Um that we're moving so f y because we can react quickly.
Peter 43:28
That's an interesting one. Yes, and the reason is we can fix stuff
immediately.
Ed 43:35
Yeah. I mean it used to be that if you roll something into production and it breaks something, it you know, ten thousand people will hit it before you can get a patch. Now, if you have things set up so that thing so that your system reacts immediately when an exception hits, if you push something to production and it broke something, you know, you can automatically roll back within seconds, you can apply the patch and then you roll back forward within s you know, so you might have three or four people hit an exception, that annoys them, fine, but you didn't break something for tens of thousands of people and
your software is fixing itself kind of in real time. So you can move really fast and not really worry too much about catching every edge case up front.
Peter 44:14
Okay, so MCP wins one, agents are running for everybody, our users or our QA gets refixed off automatically immediately. That all sounds right. um here's another one. I think smart businesses, basically they just think about like, okay, so now we can do...
Obviously features are free, we don't even have to talk about it. um So any smart business can look at their customer base that they have. We have a hundred customers, we have 20,000 customers. What more can we do for them? Now that features are free, our users are our QA, we got the A. That just seems...
Ed 45:08
Yeah,
I mean there are all kinds of opportunities. One opportunity might be to decide that you don't want to be the the company that's making the customers QA, that you really pride yourself on keeping that bar of excellence that we used to aspire to back in the good old days, where we would make sure that things were really polished and worked really well before we sent them off to users. Because the fact is users don't like being QA, but we've just accepted that we are. And we've accepted that, you know, things break a lot more often than they used to.
Peter 45:31
Well, I disagree. think software
always used to suck. Even when we were a small team and we were proud of our quality of implementation, we still shipped bugs. You know?
Ed 45:47
yeah, we s we still shipped bugs, but
now we're shipping bugs every day. You know, we're we're
Peter 45:52
Hahaha
Ed 45:53
we're knowingly shipping bugs. We're we're intentionally shoving shit into the world and then picking the corn out of it on the fly. It's
Peter 45:59
I don't think it's shit.
think it is better than how it, what it used to be. At least, you know, if, the team that does it has some minimal level of, you know, you know what I mean? Like, I don't think it's shit. I think, think a lot of software that's being pushed out now is a lot better than the software five years ago.
Ed 46:19
Uh yeah,
yeah, but then a lot a lot of software is a lot worse. So even just this morning, I went to log in to an internet service provider that I have access
Peter 46:28
Yeah.
Ed 46:29
to, and I'm not going to name them, but um when I tried to get the code to connect, it wouldn't connect. It said I needed to add a payment method. That seemed really strange. They've had my payment method on file forever. Turns out the problem was I wasn't logged in. But rather than saying you need to log in to do this, it was telling me I had to add a payment method.
So that
Peter 46:51
Yes, fair enough.
Ed 46:52
so that was annoying. And then when I did actually log in, they gave me a notification that they have deprecated the uh the 2FA, the previously used 2FA, which was use your authenticator app,
Peter 47:04
Yeah, yeah, yeah.
Ed 47:06
in favor of email. So now they're emailing a code
Peter 47:09
Ugh. God.
Ed 47:11
instead of letting me use my authenticator app. And I'm sure the reason that happened is because something broke.
And rather than fixing it properly, they're like, let's just slap in the easiest thing we can slap in. And that's where, so yes, in organizations that have always cared about quality, there are um they're using AI to ensure the quality
Peter 47:30
The quality has gone up.
Ed 47:32
is is going up. But in organizations that have only ever cared about moving fast, they're using AI to move fast and the quality is going down.
Peter 47:40
Fair enough, but the one thing I'll just push back on is that that doesn't seem related to having our users be our QA. Like the ceiling has gone up, you can ship some really high quality stuff while still having your users be your uh QA. It's just like when you're gonna ship shitty stuff.
And your users are not even your QA because there's no way for them to even tell you that. You know what mean? If you're in touch with your users, then you're going to be a high quality. Anyway.
Ed 48:15
Hmm.
Yeah, speaking of which I I've had this funny thing going. So I um I created just inspired by by what you built. And actually I probably should have integrated with your API because it would have been so much easier. But now I know what you've gone through getting transcripts from podcasts and newsletters. And boy, that's not fun. I'm, you know, I had I admire I admire
Peter 48:36
you should, yes, you should use my API, yeah, yeah, tell me about it.
Ed 48:40
your effort. Um so I I have this uh
This app that I threw together just to kind of scratch my own inch. Because we've I've talked about this a hundred times. I have just hours and hours and hours. I probably have like two weeks now worth of podcast queued up I'll never listen to. Like if I want to listen to my head hit play and two weeks later I would still be playing. Um, then I have, you know, things I've subscribed to on YouTube that I never watch and newsletters just pile up on Red. And so I just built something where I just took all that stuff and I subscribed my agent to all of those things.
Peter 49:11
Yep,
Ed 49:11
And
Then every week.
Peter 49:12
that's how it's going to be.
Ed 49:14
Every week it just tells me here are the highlights. And then if I'm like, I'm thinking about this, who in my network is talking about this? And it'll just, you know, reach in and say, you know, this person talked about. So anyway
Peter 49:25
I think that's how it's going
to be on the beach. you know, like
Ed 49:28
Yeah.
Peter 49:28
it's this, it's this whole, I don't know. I think you're fairly AI built.
Ed 49:35
Mm.
Peter 49:35
I'm probably a little AI built now.
Just, and we should wrap it up soon, like, the final thing, so another thing that is true, I think is the models are very, very, very, very smart getting post-trained like crazy on all kinds of useful stuff. And so Opus 5.5, for example, it's just like, I think the model overhang to like the capability overhang to what it can do versus what I'm making it do. And I'm trying to think as hard as possible on what can you do for me. um
is enormous, like it can do a ton more. then, know, next thing we know, it's February next year, right? And so the models are smarter. I don't even know what that means. I don't even know, I don't know how to think about that because like today the capability overhang is already so large that I can't even, I feel like this super smart model and I don't even, I'm like,
I don't know what to have it do.
Ed 50:43
Yeah, that's that's sort of the the whole purpose be behind my experimentation model is like I just wanted to test its limits. I'll tell it do this thing, make this work, and see if it pulls it off. And a lot of times it's totally off the wall, sometimes it works, sometimes it doesn't. Uh and I I let it rather me thinking
Peter 50:57
But I can't think of stuff that I could not do. Like
the other day I was like, make me a desktop Rust app that's like Canva and it did it. It took like 20 minutes. And then we're like, yeah, that's cool. Can you make it in like Swift? And it did it. And it took less, right? Cause I already figured out what all the features should be. And I'm like,
Ed 51:15
Yeah. Yeah, the requirements are there.
Peter 51:17
I can't even come up with stuff for you to do cause you're too smart and too fast.
Ed 51:23
Well, the great thing is it's it's actually doing things now without me even asking it to do it. So getting back to this this podcast thing that
Peter 51:29
Yeah.
Ed 51:30
I created, I threw it together and one of the MCP tools that I put in was send feedback. And so anytime one of my users runs into any sort of an issue, um, the the agents is instructed that you can send feedback, you know, bug reports or feature requests or anything, just you use this endpoint. It goes, it hits the database, and then on the other side, so there's an admin tool.
For the SCP that receives that feedback and acts on it. And so I opened up Claud Desktop or Claud Cowork this morning to see because I have it do this schedule job where it kind of expands out the network of the podcasts and the the newsletters based on who's been a guest on the podcast and who was mentioned and who's, you know, things like that. So kind of growing the network and following more people for me automatically. And in the process, my one agent that's acting as me, building up my my library of things I'm following.
ran into a couple of issues, use that send feedback MCP tool to report the feedback. My other agent, which is running in the same, basically the same system, picked that up and oh, I see a bug. Let me fix that. Went through, patched it. You know, I didn't actually intentionally build this feedback loop. I put the pieces in place for the feedback loop to exist. Um and the agent's just like, okay, let's close the loop. We see where you're going. We're just going to close the loop.
Peter 52:45
And now you have agents, this is really,
but so that's a pattern, right? I'm a big believer. you know, the one thing is like, you should, you should add in that instruction and, and send feedback on how the tools can be better. Because like agents are very good at saying like, I'm doing this task, but it would be nice if this tool had this one, like filter option or something.
Ed 53:12
Mm-hmm.
Peter 53:12
And
then just like it implemented, right? Like just tools are going to get better and then it'll get better. then, and then I'm like, you should just let it live for like over Christmas. And then in January you're back here like, what are all the new features?
Ed 53:28
Yeah, I mean I I might. I I don't want to get too feature rich. And so I built this, it has no dashboard. It's literally MCP only. Um
Peter 53:37
Yeah, who wants a dashboard?
Ed 53:39
yeah. And uh so yeah, separately we might talk about j me just using your API because I do have some questions about how you solve some problems that I've run into that are either expensive or tricky to solve.
Peter 53:53
It's expensive and tricky is the answer. um
Ed 53:56
Yeah.
Peter 53:57
So it's like, guess, divization and naming all the people and stuff like that, right? Yeah, that's expensive
Ed 54:03
Yeah. Yeah, yeah.
Peter 54:05
and tricky. um Yeah, we can chat about that, but we should wrap it up because it's almost 11.
Ed 54:10
All
right. Yeah, I have to get ready to upgrade PHP again for the second time in a month.
Peter 54:16
Yes, enjoy. just gonna like... All my stuff is on Laravel Cloud, so I'm just gonna be like... Thanks guys, I think you
Ed 54:24
Yeah, I think
Peter 54:25
got it.
Ed 54:26
I think they have it. I I think that you know, I'll I'll reach out if it's something that you should actually worry enough about to shut things down, but um but I don't think it
Peter 54:34
Yeah, uh
Ed 54:37
will be. I think it'll be it'll be fine.
Peter 54:39
Yeah,
and have hard limits on all your API keys, like hard expense limits.
Ed 54:45
Yeah. so the next time we talk, I do have a big complaint about Laravel. So it might be
Peter 54:53
Let's... Let's end on that.
Ed 54:55
It's one that I want to get into today. Um but yeah, they they're causing me some headaches.
Peter 55:02
Alright, well, um...
Ed 55:05
All right. We'll talk about it next week. All right.
Peter 55:06
Have a good week, have a good day. m Enjoy the fixing.
Ed 55:10
Thank you.
Peter 55:11
Tomorrow the trains run so that's good. Alright,
Ed 55:13
All right. See you next week.
Peter 55:16
see you later.