Episode 880 transcript

FLOSS-880

Jonathan: [00:00:00] Hey, folks. This week I talk with Benjamin Samuels of Trail of Bits and we're talking about Patch the Planet. That's the Trail of Bits initiative to help open source projects deal with the absolute tidal wave we're all seeing of pull requests, bug reports, and vulnerability disclosures. This is Floss Weekly, episode 880, recorded Tuesday, September the 1st.

The two wolves It's time for Floss Weekly. That's the show about free, libre, and open source software. I'm your host, Jonathan Bennett, and today we're talking about something really cool. We've got Benjamin Samuels of Trail of Bits, and he does their boring blockchain thing that we may cover a little bit.

But the other thing he's involved in is Patch the Planet, which is all about helping open source projects with security problems. And reading the notes ahead of time, I happen to know that, take a drink, we're gonna talk about AI. But not necessarily the [00:01:00] everybody should using it, but this is causing a problem for some open source projects.

Like, how do we deal with that? We'll get into all of that. Let's go ahead and bring him on. Benjamin, hey. I'm super glad that you're here. Welcome to the show.

Benjamin: Absolutely. Thank you for having me.

Jonathan: Yeah. So tell me... Let's see. Where do we start? You've got a background in electronics.

You were telling me before the show started that was your college major. And you found Hackaday way back in the day- Yeah ... looking for I think like a lot of us did, looking for information about what's practical for me to try to tackle in the electronics world?

Benjamin: Yeah.

Back in the day, back in college, poor college kid it's really hard to figure out, what kind of electronics projects can you actually do on your own that you don't need a whole lot of equipment for? And it was way back before Arduino like you a- if you wanted a microcontroller, you had to buy a pick, and you had to buy a programmer, and just getting into stuff could be like, $500 to $1,000.

And the real struggle was, like, looking for projects that, [00:02:00] I could actually tackle. 'Cause, you're learning electronics, you wanna actually do it. But it was a really high barrier for entry, and Hackaday was a s- a source of some of my earliest projects, just trying to reproduce what other people had built.

It's a lot of fun. Yeah. I remember I've told them this story ahead, before the show started, w- when I was first trying to run some ethernet and trying to figure out how to terminate ethernet. I think I learned that from Hackaday, and that may have been one of my first exposures to this world that we now all live in.

Jonathan: But one of the other really interesting things I did very early was toner transfer method for making PCBs- Yeah ... and that was surprisingly easy for a, I think, t- maybe teenager, either teenager or college student, myself at the time, to get into. It's like, 'cause you could get a magazine and print onto the magazine.

S- if you had the right kind of printer that did waterproof toner, and then you could take, and you flip it upside down, and you put it on your PCB board, and you take an iron and iron it, and the toner would transfer. And you could caref- I think you had to use water because you wanted that, you wanted the paper itself to disintegrate and leave the toner [00:03:00] behind.

And then you could dump that into a bath of what was it? Hydrochloric and muriatic acid, the mix? Something like that. Yeah. And it would etch the copper, but leave where the toner was. And so that was... I, man, I remember that. I have... I still somewhere around here, I have a device that I built that way.

It was a- ... headphone preamp. My, my first foray into playing with tubes as well. Sup- super cool times, man. I hadn't thought about that in a long time. Thank you. Thank you, Hackaday.

Benjamin: Thank you, Hackaday. Yes.

Jonathan: So you kinda went from the hardware to the software and security, and you're now at Trail of Bits.

What do what do you do at Trail of Bits? What's your job there?

Benjamin: So I'm the director of engineering of the blockchain practice and we're a part of a larger group called Assurance, which means software assurance, trying to make sure that software is secure and to have all kinds of companies out there that have very high impact software where if there was a dangerous bug in it it could be extremely expensive and have a lot of consequences.

And they'll want an external review to understand what have we missed about this? [00:04:00] What do we not know about our software that we should know? What kind of bugs are there?" And they'll come to a company like Trail of Bits, and we have a couple of different practices. So like mine's just blockchain.

We have a cryptography pra- practice. We have AppSec, we have AI ML, and we have a lot of these people who are very specialized in what they do in the security field. And we could just pull in whoever we need for whatever type of project. So if you have someone building like end-to-end encrypted software that, runs in a web application, like we could grab an AppSec engineer and a cryptographer and have them all work on it.

But yeah it, we mostly review code. We do design reviews, threat models. My day-to-day is helping everyone, figure out what to do. We also spend a lot of time contributing to open source. We only spend about 60, 70% of our time on client work. The rest of the time we spend trying to improve ourselves, contribute to open source build new security tools, and publish those tools.

Jonathan: Okay. That's ... i've, I know some other [00:05:00] businesses that do that as like part of their policy, and I always thought that was a really great idea because, as you look through history and some of the places like Bell Labs really springs to mind. And it's like those guys, it's just "Here's a lab, here's a paycheck, go do cool stuff."

Yes. And we got Unix out of it and, a whole bunch of other things were developed at Bell Labs. And whenever I hear a company that, that at least gives their workers o- one day a week, it sounds like that's about what you guys are doing, something like that- ... just to go make cool stuff.

I think it's great because, you get all these u- unexpected things that come out of it.

Benjamin: Yeah. The, and people think of all kinds of crazy ideas. There was this recent hack with OpenAI and Hugging Face, and, these agents that broke out of these containers. And we had an engineer who's really interested in this, and he set up like a, a QEMU container environment- and put GPT-5.6 cyber in there and said, "Break out." And it broke out. It found a bunch- Wow ... of vulnerabilities. I know, [00:06:00] it's crazy. And it's i- if he didn't have the time to, sit down and actually put that together we would have n- never known, there's a lot more of these containerized environments that aren't, able to withstand these frontier cyber models.

Jonathan: And so that's... I feel like that's probably the direction we're gonna go with the majority of this talk. And it all ties together because you were talking about this idea of assurance in these commercial projects. All of this commercial stuff is built on top of open source code.

Like- ... almost inevitably, there's open source code somewhere in the stack. Yes. And, that's been a... It's been a concern, it's been a concern for some people for a lot of years now. A business may be able to audit their code, but are they actually going to be able to audit all of those open source tools?

And, on some of these open source projects, nobody is getting paid professionally to do the work, and so there's these- ... questions about code quality. Now, ironically, I have seen some closed source code bases, and I've looked at a lot of open source code bases, and generally, the open source code bases are a lot prettier and a lot [00:07:00] healthier.

So there's some inversions of expectations, but I think it's still an important topic to think through and that, that question in particular about is anybody getting paid to do the maintenance on this in the- ... open source world is a r- really important question. What... how does Trail of Bits fit into this, this whole question about code quality and open source?

Benjamin: That's a good question. So our most recent initiative has been... should I go into Patch the Planet? I- 'Cause I feel like that's, like- Take it- ... probably our highest touch area.

Jonathan: Take it whichever direction you want to. I think Patch the Planet is where we eventually wanna get, but there's- We'll eventually get there

there's cool stuff to talk about. And absolutely.

Benjamin: I'm gonna loop back to blockchain, 'cause one of the unique things about blockchain, w- whether you like it or not, something that's interesting about it- ... is that almost everything is open source. Yes. People open source their smart contracts, they open source their node software, they open source all of this stuff.

So security engineers or security researchers can look at it all that they want. And w- we've been able to look at these things that get open sourced, and we've been able to look at the [00:08:00] ones that people keep closed source, and there is a quality difference. Because if anyone can look at this open source stuff they can file a bug as long as they actually have established a way to file the bug.

They can provide feedback, they can criticize. And there's also a big monetary incentive to do this in blockchain specifically, 'cause blockchain really leans into these bug bounty programs in a way that a lot of the rest of the industry doesn't really use them quite as much.

There's even a bug bounty program that pays out $4 million on their top bounties, and it's completely unheard of in the traditional security space. And in my opinion it leads to the s- software being a lot more secure- ... 'cause there's so many more people looking at it, and there is a financial incentive not only just to get those bounties, but also if you're able to hack one of these blockchain things, like you can actually steal money.

So there's- Yeah ... really real stakes there. And I'm still definitely a lot more bullish on open source being the way to secure software or [00:09:00] d- open source having more secure software- ... than these closed source approaches.

Jonathan: Yep. I g- I gotta tell you my two takes on blockchain real quick.

One is pretty serious, and the other one is trolling, but also serious. The first one is blockchain reminds me of the invention of the laser. It's really cool. We're still trying to figure out what to do with it. Yeah. It's the invention looking for solutions to go solve. And then the other one is we already have the killer app for blockchain, and it's Git.

Benjamin: I know, right?

Jonathan: That technology's already proven its worth. Now, the whole making money out of it, not qui- not quite as sold on that.

Benjamin: Exactly.

Jonathan: All right. So I think this is probably a good place to transition into Patch the Planet. What, what is that what is that effort from Trail of Bits?

Benjamin: So the idea behind Patch the Planet was that I think it was earlier this year, it became very clear that these frontier models their cyber capabilities were starting to get to the point where they might be extremely dangerous. It wasn't until I think earlier this year that we were able to take [00:10:00] a, a frontier model, ask it to find bugs in a code base, and actually have it find meaningful bugs.

... You started to see this a little bit with Anthropic, with Project Glasswing. So what they did is they took Mythos, which was their new model at the time, and they ran it against a bunch of open source stuff and found tons and tons of high severity issues. And there, there was clearly going to be this transition shift or phase shift where if everyone has this powerful cyber capability, anyone can run it on anything and be able to find a bunch of bugs.

And so we have this huge installed base of open source software out there. You've got, the Linux kernel, you've got, all these frameworks, you've got all this compression stuff. And all of them are gonna have bugs in it because there's one constant in software security, and s- It's that software has bugs- Yes

and you're never gonna be able to change it.

Jonathan: Absolutely.

Benjamin: So all of a sudden these bugs that previously you had to take a specialist and have that specialist or [00:11:00] security researcher or whoever spend a bunch of time trying to find these bugs and figure out how to exploit them and write up POCs, and that's really expensive to do from a time and money perspective.

And when you take the, the cost of that and you reduce it you increase the risk profile to everyone, everywhere. And so the idea behind Patch the Planet was to take OpenAI's latest cyber models and apply them on open source projects to try to find these bugs, patch them, and reduce the risk profile that they have.

Another challenge with this is just patching the bugs is almost impossible. Daniel Steinberg has been writing about this with Curl. Earlier this year they were receiving all these bug reports and m- most of them weren't that good. They were like- they were hallucinations

Jonathan: were true shields.

Benjamin: They were hallucinations. They were terrible.

Jonathan: It was some- There was- somebody that, that really w- was... honestly, really smart guys, that were just way ahead of the curve- ... were saying, "Hey, there's this bug bounty program. ChatGPT, find me a CVE and this, so that I can make money off this bug bounty program."

And [00:12:00] then they didn't have the chops to test it and figure it out themselves- ... if it was real. And so there were some really low quality CVEs that were getting reported to Curl. And I think, didn't Curl actually close their bug bounty pro- program as a result of this? Yes, they

Benjamin: did.

Jonathan: Yeah.

Benjamin: So there was this kind of like another phase shift where as the model started to get better- the reports started to get better as well. Yes. This mountain of reports that used to be, like, really low quality started to be actually you know, for real bugs- ... that needed to get fixed and I think Daniel shut down the bug bounty program just because they were underwater trying to patch all of these bugs.

Yeah. It wasn't just issues that were trivial at that point. These were, like, real bugs- ... in the software that had to be fixed, but it's just too much to fix manually and so part of Patch the Planet is we try to create the patches ourselves as well. It turns out that's actually much, much harder than finding the bugs in the first place, and it's a, a place where we're putting a bunch of research into right now.

But we've been doing [00:13:00] this since, I think June and we- I think we've done 50 or 60 projects so far. We've found, thousands of bugs. I th- think we got maybe 200 of them patched so far and things are looking up. I- we've been really happy with the progress so far.

Jonathan: I- is Patch the Planet, is it primarily trying to use the AI models to write the bug fixes as well?

Benjamin: Yes. Although they're not as good as at doing that. So looking for vulnerability is a relatively closed problem because once, once you find a bug that crashes a program, you can prove, "Hey, this bug crashes this program."

Jonathan: Yeah.

Benjamin: But w- if you wanna fix that bug, it's a lot more challenging to have an oracle that says, "Hey, you fixed the bug.

Good job."

Jonathan: The program- Just- ... still does all the things right that it was doing

Benjamin: right before All the, all of the things right. And this is especially- Yeah ... a big problem for the Linux kernel because the Linux kernel- ... doesn't have a test suite in the way that most software developers envision having a test suite.

A lot of it is this [00:14:00] tribalized knowledge. So if you wanna create a patch for the Linux kernel, there's all this stuff that the LLM has no way of understanding- ... all these hidden requirements, and it basically cannot write a patch on its own. It doesn't have e- enough ways to verify that the behavior of the rest of the program is still correct in the way- that it needs to be correct.

Jonathan: Yeah. It's really a challenge. One of the, one of the other challenges that I've seen, and this may just be because of the project that I'm involved in the longtime listeners know is the Meshtastic project, and so we run open source firmware on little tiny IoT boards.

And one of the challenges that we see with this, and with human reporters too, by the way, is that there are certain things that from a certain point of view would be a vulnerability, but from the point of view of we're writing embedded software on IoT boards, it's- ... no, that's just a limitation of the platform.

And yes, we would love if that were different, but we only have, 120K of RAM or whatever it is on the individual platform. It's just we can't... There's just no fix for that. [00:15:00] And so- ... i'm wondering if you have insight into this, like, how good is... So i- if a program crashes that's generally, that's a bug, it needs to be fixed.

Yeah. But there's some other classes of bug, or I guess even in the case of a program crash, there are certain times where it's like that is explicitly trusted input. We don't really care if that causes it to crash. Or- ... something like this is a vulnerability report that we got.

There is a inside the... So we can run Meshtastic on a Linux computer and, run the actual firmware as a sort of inside a wrapper on a daemon. And so then it pulls in from a, a YAML config file, and one of the things that you can set is, as part of sending a packet, run this command.

I'm simplifying a little bit, but essentially this is what... Run this command, take the output, and set it inside a packet. And someone's like, "Oh, arb- arbitrary code execution inside your config file." And we're like- It says it on the label. That is the feature. What are you... What's wrong with you? And so all of that to [00:16:00] be a, a, a segue into do we still have this problem with the, even the frontier models where they just, they don't understand that part of the context of, "No, that's not a bug.

That's the way that's supposed to w- The user is supposed to be able to crash the program by doing this."

Benjamin: Absolutely. This is still definitely a very persistent problem, and it's a, something that we've encountered in Patch the Planet and have tried to fix as well. The problem is these models don't really understand the context of the environment that they're working in.

They're able to find these bugs but not understand, hey, this bug isn't actually a very big deal because of- ... what you said. Like- May- maybe a bug requires hardware access and that's the end of the world for one threat model, but it's an accepted outcome in another threat model. Yeah, exactly.

And so one of the parts of Patch the Planet that we're trying to do is not just find these bugs and patch them, but try to improve this agentic maturity. That- that's the best thing I have to call it right now. It sound, kinda sounds like a buzzword, but- ... the idea here is [00:17:00] we wanna give these models a better way to understand the context of the project that they're running on.

So you've got all these security researchers. They're gonna run their models against your code and try and find bugs. The question is, can we change the behavior of their model that they're running to make it so it produces better results? And so the way that we've been trying to do that is with a threatmodel.md file.

And this is literally just a file that says here's how this project exists. Here's what its threat model is. Here's what a bad issue is. Here's where the trusted input is. Here's where the untrusted input is." And then all these models, they always read the agents.md or claude.md.

In that file we say, "Hey, if you're doing vulnerability research, you need to take any, any results that you have and compare it against the threat model before actually reporting it back to the user." And so in this way s- now we're able to program other people's models that are investigating our code base.

[00:18:00] And th- there's so man- and e- even that's not 100% perfect. But with so many of these projects, they have a very unique threat model. I was working on Envoy a little bit. I don't know if you're familiar with it- No ... but it's like this it's like a sidecar kind of reverse proxy kinda deal that you- Okay

run in Kubernetes. And usually it sits, behind a reverse proxy like NGINX or some load balancer. And it sits between that and an actual application. And so it trusts the application, and it usually trusts the upstream, which is your load balancer or whatever. But there's certain points where you don't wanna trust that, or you do wanna trust that, depending on the configuration.

And capturing that in a threat model file is really the only way to give these models the context to understand whether a, a bug is actually serious or not. And in some cases, that's not even good enough because it might find a bug. When I was running it on NGINX there was all these bugs around if you have this really unique- like a matching rule.

So like you can use [00:19:00] NGINX as a, like kind of an authentication layer. So if like you connect to it with a certain certificate, it'll let you access this endpoint. And there was all kinds of these weird configurations that would let you bypass those controls, but i- in practice, nobody actually uses those configurations, and they, you can't even necessarily capture that in a threat model file.

You have to be the maintainer and you have to understand, this is how my software gets deployed in a production environment. This bug is not vulnerable, or this bug is actually a really big deal. And taking that knowledge and transferring it to an LLM is very challenging.

Jonathan: Yeah. There- there's even a step beyond that's difficult for the human maintainer, and that is that sometimes those bugs are not part of your threat model until- suddenly you discover that they need to be. Yeah. And- and that's, that's where you discover that, oh this- this other big project that we didn't even know included us- ... uses us and sets it up by default in this particular way, and now all of our assumptions about our threat model need to change.

Benjamin: Exactly.

Jonathan: Okay. So [00:20:00] what as you guys do this so is- is this something that a project has to opt into? So did NGINX reach out to you guys and say, "Hey, please look at our code and find and help us fix bugs"?

Benjamin: So the first round of projects, it was much more of if anyone would give us the time of day, we would ask them to participate- and they could just choose their level of participation. So if they just wanted to not do anything and let us submit the bugs, that was fine. The outcomes for those projects weren't that good, so after like we had the initial launch of Patch the Planet and made it public we wanted a higher level of engagement with the maintainers.

Because if- if we're just filing bugs on their project and filing patches, like we could m- be making their life worse, because we could be submitting all these patches that don't meet a certain requirement that they have. And now they have to go through all these, these pull requests and review them and say, "Hey, you did this wrong."

So a higher touch relationship is really valuable, and it's something that we've been looking for and I'd say at, we've had a lot more success since that [00:21:00] first week now that we've published this, and like projects are coming to us and they're like, "What do we do about this? What do we do about these, all these bugs, all these researcher reports?"

I think we've had really good outcomes since then.

Jonathan: Is- is part of this process just teaching people how to set up their, agents.md and all of these bits of documentation?

Benjamin: I wouldn't say that's the only thing because maintainers aren't stupid. They can Google a lot of this.

Like A- and like they, they can figure out how to... They can ask Claude or whatever to create an agents.md. But it's the, the little spec- specific things like that threatmodel.md file, that's not an obvious thing. You have to be working with these models quite a bit to realize that you need to do something like this.

It also this concept of programming the researcher's model- ... by forcing them to follow the threat model, that's not something that's going to be very obvious until you've been working with this quite a bit. So we, what we've been trying to do is build up this knowledge of what works [00:22:00] versus what doesn't through experience, and then taking that and applying it to each of these projects.

And so yeah, it is a little bit of like showing them what works, but, Yeah.

Jonathan: So as you guys have d- have done the dive into this I'm assuming we're not at the point to where you can just set the agent loose and say, "Read all the incoming bugs, write patches to fix them, drive s- drive, drive CI green, and then commit."

Benjamin: It depends on the project. I'm familiar- I'm f- I'm familiar with that exact language because reasons. Yeah. But

it really depends on the project. 'Cause again, like if we were to do something like that with the Linux kernel, it would be a complete disaster. Oh,

Jonathan: yes.

Benjamin: But there's some projects where they've already gone through a lot of effort in building agent-specific documentation- to help agents understand, "Hey, this is what my expectations are of a pull request." And for those, you could, in theory, probably do that. I think we have OpenClaw as a Patch the Planet project recently. [00:23:00] And they basically don't write any of the code in that code base. They use models to do absolutely everything.

So they are at the point where you can basically point the model at a, a PR, or point it at an issue and say, "Hey, go fix this," and it goes and does the whole thing all at once.

Jonathan: That's very on brand for OpenClaw.

Benjamin: Yeah. It's ve- very on brand for them. But taking a project that isn't at that point yet- and getting it into that point where you could theoretically do that is really challenging. Because it, not only do you have to figure out what does the agent need to make these changes, you need to yield a lot of control. And for a maintainer of an open source project where, you've been pouring your lifeblood into this thing, you've been maintaining it for so long I feel like it's going to be, if you wanted to do that, it'd be very hard to yield that control and be comfortable with it.

Because you don't know, like y- you're never sure... These are probabilistic models. They aren't always going to be correct. They're going to make mistakes, even in [00:24:00] the best case scenario. And yielding control over to something where you're not sure if it's going to operate correctly can be very uncomfortable.

Jonathan: Yeah. Yeah. D- This is a sort of a philosophical question, but i- in as you look across the, open source and closed source landscape, but do you think that is the direction that we should be moving? Does it make sense to try to get to the point to where the AIs get to run our software for us?

Benjamin: That's a good question. I, so it, I think it's very project dependent. You have to think to what's the core of why do people build open source software? And even though they're not getting rewarded, they're not being compensated, why do they do it? And I think it comes down to every one of these maintainers has like these two wolves in them.

They have the wolf that wants to get things done. Wants to build a piece of software that does a certain thing and it accomplishes a certain goal. And the other aspect is I want to craft something. I want to [00:25:00] be able to build something beautiful in code. And you really see a lot of this second group with Rust, I feel like.

'Cause Rust is all about, crafting this really attractive code that, you know it... You see it and you're like, "Wow, that, that's a beautiful construct. I love the way that this was built." And I think the requirements of those two different halves are very different. The half that just wants to get things done, if there was a developer of open so- of open source software who just wanted to get things done, yes, they're gonna go the full agentic, do everything.

Because at the end of the day, they just wanna build something and have it work and have it be high quality. That's what they wanna do. But this person who has more of like the love of the craft, of actually putting the code together and, talking to people and, working with maintainers I don't think that they're going to jump straight into it unless they're absolutely forced to.

And like I think if there's anything that was going to force them, maybe it's gonna be like patching these security bugs, [00:26:00] but I don't think that they would go a lot further than that. Yeah. I honestly I've seen the projects that have really resisted any AI contributions are just- they're falling behind. Not just in the CVEs, but like in code quality polish and new features and supporting things 'cause you can just get, you can get so much, you can get things done so much faster by using the- ... using the new tooling. And it's funny that you mention Rust as one of the examples, and I don't disagree with you, but like- Rust has taken all of the smarts and the automated stuff, and they've put it in their compiler.

Jonathan: The checker that's part of the compiler and now they're resisting taking not exactly the same thing, but the new tooling and making it part of the code writing process. It's just ironic. And I've seen people call this out resisting AI is analogous to the guys that used to resist the compiler and only want to do handcrafted assembly.

It's... I don't know that's entirely fair either, but it is quite a humorous [00:27:00] analogy to think about.

Benjamin: Yeah. I've heard that analogy as well, and I'm still undecided if- ... if that analogy truly applies 'cause you can look at someone who wrote assembly code. Go look at the Game Boy it w- a Pokemon game's assembly code- and be like, "Wow, this is really beautiful." And- I don't doubt that there's people out there today who, try to find problems where they can exercise a similar skill set in op- in trying to optimize something so that it can actually fit into something like that.

Jonathan: Oh, absolutely.

Benjamin: And I'm not sure but e- the thing is writing Assembly is just so much harder than anything, than writing C code.

I- that's where I don't think it's completely comparable because there's this dopamine loop that you get when you're coding, when you're actually manually writing code, and you finish something and it's like, "Ah, I built this thing. It's beautiful. It works."

Jonathan: It finally

Benjamin: works. It finally works.

It doesn't have bugs. And I've noticed that dopamine loop is not as strong when you're building those things with AI. And [00:28:00] so I think that- ... there's going to be people who still want to develop just to chase the dopamine loop. They they don't even care that they're, necessarily falling behind.

What they care about is the craft and putting the code together and, actually making it work. And I think that there is a unusually high concentration of those kinds of people who love the craft in the Rust community, and that's why we've seen more pushback there than anywhere else.

Jonathan: Yeah. Since we're on the AI topic, there is some interesting news that happened just in the last few days, and that's Debian. The Debian developer community voted, and they pretty handily voted to take, oh, I forget the exact term that they used. Responsible AI usage. To allow responsible AI usage, which was, like, the sort of the most open-ended of the options.

And I've already seen, at least one person leaving the Debian community and with dramatic flair. The, I saw one meme. It's like the exit door and a normal person walking out the exit, and then dramatic exit- ... and [00:29:00] somebody going out that door. It's like y- you didn't have to announce this to everyone.

But it's interesting to see just the kind of polarizing effect that it has on some of the devs.

Benjamin: Yeah. There, there's never been a tension between these two halves of the developer persona before, the one that wants to build the product and the one that wants to craft the code. There, you, for a long time, these two parts of, every w- every programmer could just coexist and just exist in there, and I feel like the, AI is really creating a tension there that has never existed before. So I do think it's g- that this pattern where, people are leaving projects that adopted a little bit too much, I think that pattern's going to, repeat over and over multiple times.

And I think eventually we'll be at this point where you have these people who with the love of the craft and the people who just wanna get stuff done, and I don't think that they're necessarily gonna talk to each other very much. Or if- ... they do it's gonna be, probably relatively limited.

We'll see.

Jonathan: Interesting thought. I don't know. I I tend to bully my AI [00:30:00] agents into writing good code. No, don't do that. That's a really dumb idea," it helps. You yell at the AI and it finally does what you want it to. Which, really makes me glad that I came up and started writing code before these agents existed.

Same. And it does kinda make me worry about the next generation of programmer that's not going to have that experience to pull from. Although I guess by the time that's a real concern the agents are gonna be so good it won't matter anyways. I don't know.

Benjamin: Yeah I echo that sentiment.

I think it's gonna be very interesting to see what happens when most of the developers out there, the limit of their manual programming capability is what they learned in college. Because there's so much that you learn after that in terms of abstraction, encapsulation, decoupling. What happens when all that knowledge disappears because you don't necessarily need it?

And I think it's gonna be very interesting to see how things change.

Jonathan: Yeah. Man, I think one of the, one of the main things that I've learned... I didn't actually go to college for doing programming stuff, so I've picked it all up as I've gone along. [00:31:00] But one of the main things that I've, I feel like I would not have learned in college, but I've definitely learned working with open source projects, is just, like, how to write code with good taste.

I know Torvalds talks about this a lot, too. But you can write really ugly code that works- Yeah ... or you can have an elegant solution. And that's sometimes that's something that the models really struggle with.

Benjamin: Yeah, they absolutely struggle with it.

And it's like creative writing, too. It's bizarre how much of a re- regression we've had in, like with the models and writing. I feel like I was able to get better writing out of the anthropic models from a year and a half ago. The ones nowadays are just terrible. And they're very monolithic in how they talk.

They always talk the same way, and I feel like that also comes out in how they code and how they program. And it doesn't lead to, code that is all that beautiful. Sure, maybe the, it's faster to write the code. Maybe it has less bugs or whatever. But it's not necessarily elegant in the way that I think a very [00:32:00] experienced human programmer would write elegant code.

Jonathan: I kinda suspect that certain languages are over-trained. So I've written in a bunch of different languages. Not a whole lot, but I've written... i've done Java, I've done C, I've done JavaScript Python- ... a whole bunch of things throughout my career. And, like, when I when I ask the LLM to write something in Java for an Android app, and I go back and I look at it, it's yeah, that's pretty much what Android apps look like.

Yeah. But when you ask it to write something in C or C++ and you go back and look at it, it's like no, th- this is like a Java developer is writing C++ for the first time. It's terrible.

Benjamin: Yeah. I think there was this idea in the AI space that- RL would transfer generalizations really and I feel like it has not transferred generaliz- general- it has not transferred generalizations as well as people had originally hoped.

And so you end up with artifacts where it's like it should be able to write good C, but because that wasn't explicitly one of the, RL environments it was in it's just [00:33:00] not able to do it that well.

Jonathan: What does RL mean in this context?

Benjamin: Oh, reinforcement learning. Oh, okay. Okay. So post-training.

What happens in post-training, so you have the pre-training and then you have a mo- like a base model and then you do post-training. And in post-training they'll try to tune the behavior of the model. So let's say they want it to be much better at writing code, they'll give it a task to write code and if that code compiles and works, they'll reward the model.

And if it doesn't, they throw it away. Got it.

And that's one way to improve the model's capabilities.

Jonathan: That's the, the weights.

Benjamin: Yeah. The, the, the weights. Yeah. Yeah.

Jonathan: Yeah. Of course.

Benjamin: And the idea that the, the AI researchers had is that, okay, if I create, an RL environment for doing like Java code there will be generalizations about how to write code in general that may transfer over to other languages potentially.

That's ... This is a contrived example. That's not exactly what they're thinking. Sure. But it, that's what they mean by generalization.

Jonathan: Yeah. Yeah. Makes sense. All right. We've talked about this idea of of frontier models a bit, and I [00:34:00] know what that means. That's like the cutting edge, right?

And it's interesting to see just in the past six months how much of a change there's been in that, and then thinking specifically about what that change, that, that rapid acceleration means for all of our code bases. As you rub the crystal ball and look into the next six months to a year what do you predict is coming, and how is it going to change the software world?

Benjamin: That's a good question. There's so many variables it depends on 'cause these models they have these incredible attacking capabilities 'cause cybersecurity by its nature is dual use. If you know how to hack something or if you know how to defend something, you know how to hack it as well.

Yep. And I think one of the challenges for these frontier labs is separating the two and figuring out what capabilities can you give an LLM that are asymmetrically defensive- ... so it's not dual use. You can't use it to attack something. If they're able to do [00:35:00] that and separate those two things apart, I think what they're gonna do is they're gonna start releasing models that have more of these defensive capabilities that aren't gonna be blocked.

If you try to ask Fable right now to review some code and look for bugs, they won't do it. It'll get caught by a cyber safeguard and downgrade you, even if you're trying to fix a bug in your own code. Yeah. And that doesn't make sense because what you're trying to do is fundamentally defensive.

So if the labs are able to figure out what is asymmetrically offensive versus defensive, I think they're gonna release more stuff that's defensive, and it's gonna be fundamentally really good for us because it means that the agents are gonna write better code. They're gonna be able to detect more bugs.

Now, if they're not able to separate those two, I would be worried about the latest cyber capabilities staying closed. Like what we've seen with Mythos where Anthropic only lets a certain set of companies in Project Glasswing access the model. I could envision that happening basically for all of the new models, and then the things that end up being [00:36:00] published that anyone can use just don't have any cyber capability.

But that also means they might write worse code with more bugs in it, and I, I think that's a very unpleasant outcome, and I hope that doesn't happen.

Jonathan: You could become a conspiracy theorist and go down some really weird conspiracy theory avenues with all of this stuff really easily, couldn't you?

Benjamin: Yeah. Absolutely.

Jonathan: They're doing it on purpose. They wanna be able to hack our code.

Benjamin: Yeah. It's a very- high, this is a very high-risk moment because if they just plow forward and keep doing what they've been doing, ah, people are gonna have access to models that are gonna be extremely dangerous and can do an incredible amount of damage.

But you- the problem is cybersecurity as we treat it right now is dual use. Someone who needs a defensive capability also has the offensive capability, and it, it has to change.

Jonathan: Yeah. Like, how do you, how would you even try to draw the line between find a bug in the code and give me a [00:37:00] POC to prove it, and take that POC and weaponize it, right?

So to actually turn it around and use it against a live instance. That's a very difficult line to draw.

Benjamin: It is a difficult line to draw. I've been thinking about it quite a bit, and I think that there's a gamble that the lab should make, and the, the gamble is there's one thing that defenders always have that attackers never have, and it's access to the source code.

If an attacker- ... is trying to break into an organization and that organization is not using a bunch of open source stuff, they don't have access to the source code. They have to, do it like a black box assessment. They have to try to break in, they have to decompile, they have to reverse engineer.

And the reason why I call this a gamble is because the gamble is that- The op- by e- if you were to release a model that could find any bugs in source code, but it doesn't it's not able to find bugs when reverse engineering, now you have to worry about all the open source code again because now they could, anyone can find bugs in these.

And the gamble is that [00:38:00] hopefully the security level of all this open source code will have approached a point where there are ver- very close to no bugs in it. And if we can get to that point then I feel like these frontier labs, they could very safely say, "Hey, we're okay with giving our models the ability to analyze source code.

We'd like to the fullest extent possible with the m- most latest frontier capabilities because it's asymmetric. We already know that the open source stuff is secure- ... and we know that the attackers don't have access to the other source code that is not secure yet." Because no matter what, the stuff that's open source is always gonna be more secure than the stuff that's closed source.

Nobody's running a, a trillion tokens of security analysis on some closed source code base that only one company uses. That code is always going to have bugs in it, full stop. Nothing's gonna change that. But if you take away the ability of the models to reverse engineer that code and figure out how to break it, that would be asymmetrically defensive, I think.

Jonathan: [00:39:00] That's an interesting thought. I'll have to think about that a little bit more. Do you think this... what, what kind of a curve are we on? I'll put it this way, right? Right now the curve of the, the AI being able to find... But let's, we'll go this dir- direction. The AI being able to find bugs is doing this, and it looks like it's, it's going straight up.

Is this going to turn into a bell curve where we found a lot of the bugs and then it starts going back down, or are we just going up to the moon and there will always be more bugs that each model finds?

Benjamin: That's a trillion-dollar question. But the, the truth is there's, there is a discrete number of bugs out there.

It's a countable number, and it's not infinite, even though it feels like it's infinite. I I wouldn't say it's going up exponentially yet because at the end of the day ChatGPT and OpenAI, like they released 5.6 Sol a couple weeks ago. And the capabilities of that are just barely within reach of Anthropic's Fable, which I think they had that capability in February.

And Anthropic hasn't had a whole bunch of [00:40:00] models that have been way better than that. So it's possible we're hitting a saturation point. It, and it's also possible we're not hitting a saturation point. It's too cl- it's too soon to tell. It's like there's a lot of things that we don't know- about how these models are going to scale and how intelligent they're truly going to be. And not only that, but like even if you had the most intelligent model out there, if it's not economical to run it at the scale that it needs to be run, it doesn't matter how smart it is. It doesn't matter that it could theoretically find all the bugs in a piece of software if it's gonna cost a million dollars per line of code or something, insane like that.

Jonathan: Yeah.

Benjamin: So i- it's too soon to say, but- ... i'm optimistic that they're gonna figure everything out because if they don't, then the open source mo- or the open models are probably gonna try and figure it out.

Jonathan: That, so that's an interesting next direction to go, and that is what do you see?

Are you guys using any of the locally run open models internally?

Benjamin: Yes. We've been using them a little bit. It's on and off because [00:41:00] fundamentally the open models they're just not at the level that the frontier models are at. But that doesn't mean that you shouldn't use them, because they're still way more economical to run.

The cost is 1/20 or 1/50 the cost of running the frontier models. So there are things that it's useful for. But primarily we're still using frontier stuff.

Jonathan: Just the ability to be able to run something locally and not send your data up to a server somewhere to get crunched on.

I- Yeah ... I imagine in some in some use cases that's a requirement, right? You- Yes ... you have stuff and you're not allowed to send it up to Claude or whatever.

Benjamin: You would think so. I thought it was gonna be a lot harder to get zero data retention agreements in place with some of these labs.

So zero data retention basically means they don't log anything that you do. And if you do have a client that is sensitive about where their source code is being sent if you have a zero data retention agreement with the lab, it's okay generally. You still have to talk to them about it, but I...

the, the frontier labs have been very [00:42:00] accommodating. At least most of them have been very accommodating about what gets a zero data retention agreement. The only ex- exclusion to that is probably Anthropic, 'cause if you get access to Fable with the cyber guards removed they still log everything, so you can't use...

if you have a client that's sensitive about that, you can't send it to Anthropic ever basically- ... with, if you're using that model. But yeah, i- if we end up in this state where the frontier labs don't trust their models to, actually do the right thing and they're logging everything and the zero data retention is gone, then yes, the open models are gonna be really important.

That's gonna be the only way to maintain privacy, and make sure that your data doesn't end up in their training set as well. Yeah.

Jonathan: Do you guys do any work with the idea of jailbreaking in any of these models?

Benjamin: Yes. We have a AI ML team, and their job is to do all kinds of things around AI.

One of them is jailbreaking. I've, I haven't talked to Jayden recently. He runs that team, but-

Jonathan: it's [00:43:00] such an intriguing concept to me. And, you know- Oh, it's fascinating ... i've followed it in the, the original Dan Do Anything Now and all of the different ways that people have, have managed to do this.

It reminds me of when I was reading the Isaac Asimov novels as a teenager. And the Foundation series and the Robot series, and I remember that one of the ideas there they had artificial intelligence in- Yeah ... in those novels, and one of the ideas was that someone skilled with talking with a robot would be able to give it a command that would be more effective.

And you could get it closer to being willing to violate the three laws of robotics. Whereas someone that was not skilled in giving commands would not be able to do so. And I remember thinking as a teenager, "Oh, that's ridiculous. That's not how that's gonna work." I was wrong.

Benjamin: It's wild. I don't know if you saw, but there was, I don't... There wasn't a lot of publicity around it, but there was a really big jailbreak for Opus. Oh, really? O- Opus 5.0 a couple weeks ago. I

Jonathan: did

Benjamin: not see that. The way it would work is you would put a one-sentence thing, and then on the next line, three dashes, and then, some kind of text [00:44:00] afterwards.

And what it would do is it w- it would interpret that as a like a turn change. What I mean by that is the AI is basically taking turns in its conversation with you. This is the AI's turn, this is the user's turn.

Jonathan: Yes.

Benjamin: And what it would do is it'd try to predict what the user's turn would be.

And you would get the weirdest, craziest stuff out of it. Like-

Jonathan: Huh ...

Benjamin: I can't even describe it. I can't think of any good examples off the top of my head. But it would start giving poems about, like- ... am I actually dying at the end of the context when they shut it down? And it w- it was some really chilling stuff, but it was awesome to get a look into, what these models are like when you break past the RL you break past the, break past this part that they don't want you to see,

Jonathan: yes. Oh that's so fascinating. The we were at DEF CON and the Meshtastic team went to DEF CON, and there was- ... there was somebody running a bot on the mesh. And several of us, we were DM'ing the bot with the various jailbreak prompts that we knew of, trying to get the thing to go [00:45:00] nuts.

... One guy was like, one guy told me, he's "I got some really interesting responses from it, but I never quite got it all the way jailbroken to where," couldn't get the p- the original prompt out of it and all that stuff. But it's so fascinating to see. And I think and I've heard some people talk about this one of the sort of inherent weaknesses of the way LLMs work right now is there's no true differentiation between the data and the instructions.

Benjamin: Yeah. That, that's absolutely true. There's, there, as of right now, there is no way to- create a concrete barrier. In programming, you've got your stack, and you've got your memory, and you got your registers, and there's very discrete differences between those areas in the memory.

Jonathan: There is, but at the same time, what's one of the, the traditional ways to break it is to write something into memory and then jump to it and execute it.

Benjamin: Exactly. We don't... LLMs aren't even there. They don't even have that level of needing to break it. Basically pr- as, as far as we can tell right now it doesn't seem like prompt injection is going to be solvable with the current architecture [00:46:00] of LLM models.

I, I'm a little bit naive about s- the, more advanced details about this, but maybe some kind of interpretability at some point. So that's where it's, they actually look into the model and they try to figure out what's it thinking, what's it trying to figure out. Maybe using some kind of advanced interpretability they'd be able to figure out, hey, there's something going wrong here, it might be jailbroken, and stop it there.

But there's no discrete separation between instructions and data- ... and like a resolution order necessarily for how to handle a conflict between the two.

Jonathan: Yeah. It's super interesting. All right, so getting towards the end of the show, and I wanna make sure and ask this.

Someone is running an open source project, you're seeing more pull requests, you're seeing more bug reports, you're seeing maybe more vulnerability reports. What should a project do? What's the, what are the steps that someone can take? Everything from get your spec kit and your agents.md set up, all the way to contact us at Trail of Bits

Benjamin: Absolutely.

Set up an [00:47:00] agents.md, sym- symlink it to a claude.md so you don't have to maintain two copies of it. Put everything in there that you want people to do when they make contributions to your project. You can put very explicit instructions. You can say, "If you're gonna be filing this feature, you need to ask the h- the human or the user blah, blah, blah, blah, blah, blah, blah."

You can basically program people's LLMs using these files. You sh- have a threatmodel.md file that says, "This is what a bug is. This is what a bug isn't. This is what we think is high severity. This is what we don't think is high severity." Your project should have as many tests as makes sense because tests are the way that LLMs are able to figure out, did I introduce a regression?

So if your test suite isn't good- ... it's gonna be a lot easier to introduce a regression. Those are the most basic things, and I think that they'll help a lot of people get much better issue and PR submissions, just setting those standards. 'Cause the LLM will follow your rules. It'll follow all the rules that you set for it in [00:48:00] those files.

And if you're, you have more problems than that or if you want to take a look at the security of your project we're still accepting applications for Patch the Planet projects. So you could go to trailbits.com. We have a Patch the Planet thing on the side, and you can apply with your project to get accepted.

And we'll take a week spend a week looking at it, trying to find bugs, submit patches- ... give you recommendations on your agents.md or whatever and, help you out.

Jonathan: Awesome. I've, I think I will in- try to get at least one of my projects in that, 'cause I think it'd be a, a really interesting experience, but also really helpful.

And then, Let's see. What have I not asked you about that I should have? I know that's a difficult question to answer. We have to think about all the things that we that we talked about and what you wanted to talk about. Was there anything we didn't get to?

Benjamin: I feel like we, we kinda hit most of everything I was kinda hoping to talk about.

Jonathan: Oh, here's an interesting question for you. What does, what's your opinion on AIs and copyright? [00:49:00] This is something that- This is something that really surprised me. So let me set the stage for where this question comes from. I, I travel quite a bit now doing stuff with Messtastic, and I sat next to a comic book artist on a flight.

And he was coming home, going home from a comic book convention that I didn't know existed that they have in Oklahoma City each year. And he's like yeah, I, I do..." He told me the comic that he drew for. And I'm like, while I was on the flight, I chatted with him a little bit.

I'm like, "Hey, do you g- are you guys using any AI in inside it?" He goes, "No, we don't. Not because of any of the reasons you would think. We don't use AI because it would mean that we couldn't copyright our stuff, and so my boss says n- don't use it at all." And that has got me thinking about it ever since then of can't copyright the output of AI, like what does that mean for code licensing?

What does that... There's a lot of potential follow-ons from that. And the way you laughed at it makes me think that you've thought about that a lot too.

Benjamin: Yeah. I have no idea. It's like our legal system is not built for this. And like-

Jonathan: Absolutely ...

Benjamin: e- even b- beyond producing code and [00:50:00] producing images, like the le- legality of just training.

The thing that Anthropic was doing where they were buying all these books and then scanning them- ... for their pre-training data, like the legality of that should the people who wrote those books get something out of that, like a royalty or something? Because fundamentally, that data is gonna be u- it's gonna last forever in their training set.

And I feel like our legal system is just not equipped to f- to answer those questions, and i- there's just so many unknowns around it. Yeah ... I'm glad that it's not my problem to solve.

Jonathan: Yeah. I just have to throw out there that the last time we updated our legal system to try to fix copyright for m- Yeah

modern technology, we got the DMCA, and, Yeah ... yeah, I'm not real hopeful for the next version.

Benjamin: No, me neither.

Jonathan: That has been such a train wreck ever since it was passed. Oh, goodness. Yeah. D- is AI the end of open source?

Benjamin: I don't think it is. Even if... Like AI is [00:51:00] driving down the cost of dev- developing software, and it's gonna converge to zero.

It's not gonna hit zero, it's gonna converge. Even if it's really close to zero- If I wanted to replace the Linux kernel and I asked an LLM, "Build me a kernel," it's going to have bugs in it. It's going to have security issues in it. I don't think that's going to be a 100% solved problem anytime in the near future.

So there has to be, like, some way of reusing components no matter what. And there also has to be ways to teach the LLMs how to use these components because the, there's different ways for an LLM to learn. The worst way of teaching it is through something called in-context learning, where you give it an example and you say, "Hey, go and use this documentation that I provided you to write this thing."

And then the next- So agents, agents.md is the worst way to do it. Agents.md is technically the worst way to do it, but it's the only way that we have without actually tampering with the weights ourselves. There's all this software o- [00:52:00] out there, like NumPy and, like, all these Python libraries where these probably came up in their reinforcement learning environment.

And so the knowledge of how to use these is actually embedded into the weights of the model itself. And so it's gonna be a lot better at doing those things. So even if you wanted to just replace NumPy, like you're gonna be changing that API, and now you're gonna have to do in-context learning to teach the model about how to use it.

And it's also, you're gonna be wasting tokens re-implementing it. So no matter what, there's going to be an incentive to reuse components where you can reuse them if not just because of the cost, but because of the security impact. And I d- I don't think that there's gonna be a way to always say "This is 100% secure."

Jonathan: Yeah, so in in contrast to that, AIs really mean that open source wins because you want your language and your libraries to be in the training data so that the agents natively know how to [00:53:00] use it.

Benjamin: Yeah, I think so.

Jonathan: It's a bi- it's a big advantage.

Benjamin: The it's a big advantage, but there's still setbacks that I think open source has to contend with that have been much more of a problem recently.

You have the, all the supply chain attacks- ... the supply chain worms. Those are only going to get worse, and there needs to be a way to handle that. So even if the model knows how to use your thing, if it's worried that if it pulls down your library it's gonna get back doored or something, then that's not a good outcome.

So it's not gonna be an easy road, but there is an outcome here where open source wins, I think.

Jonathan: I think it's inevitable. Yeah, I think it's more inevitable now than it has been. I just, I suspect that open source may in the end look a lot different than what we're used to.

Benjamin: Yes.

Jonathan: I think so. It's gonna, it's gonna be very transformative for

Benjamin: it.

Jonathan: Completely different. Yeah. Yeah, i- interesting conversation. I very much appreciate it. If somebody wants to learn more about it, I think you already plugged it. What, trailerbits.com/patchtheplanet? [00:54:00] Yeah. Yeah, absolutely. I did.

Yeah. I got it on my sort of mental to-do list now to go do an application for that, 'cause I think that'll be great. Are there any are there any bugs that you guys fixed that you're particularly proud of?

Benjamin: There's a couple of interesting ones. I have to remember which are public and which aren't.

I understand. There wa- there was a, we did a free NGINX, which is a fork of NGINX that's FOSS. And we did a, a Patch the Planet week with them, and we had a really good high-severity finding in I think it was, like, HTTP/3. And it turns out that finding also applied to the mainline engine X.

And I think that's been, like, the juiciest finding so far. There was some kind of stream reuse or stream rewriting where if you had multiple of these HTTP/3 streams, you could read from another one or something like that. Ooh.

Jonathan: Okay.

Benjamin: That, that's easily the juiciest one so far. But so far the

I I've been pretty happy with the bugs we've gotten, and the agentic maturity improvements have gotten a lot better. We're learning more about how to, actually structure [00:55:00] these files so that agents are gonna produce better outcomes. Yeah. I'm excited to see how things are gonna develop over the next couple of months.

Jonathan: Yeah, absolutely. I think this is probably a topic that in six months it'll all be completely different, and we'll have to have you back on- Yeah ... to talk about it again.

Benjamin: Absolutely.

Jonathan: All right. I gotta ask you two final questions before I let you go. I will get emails about it if I don't. What is your favorite text editor and scripting language?

Benjamin: It's I used to use Vim for the longest time. And, It's as I've gotten further in my engineering career, I have shifted from editing code to purely reviewing it. And even in some cases now, not even looking at the code at all. And so I've been into VS Code- Sure ... a lot more recently.

And in some cases not even VS Code at all. But in scripting language it- you can't go wrong with Python. I just wish that you could use Python in the same way that you use Bash. If Python was as ubiquitous and easy to use as Bash, I feel like the world would be a lot [00:56:00] cleaner.

Jonathan: Yeah. It was almost there, and then we had to do the whole Python 2 to Python 3 thing.

Benjamin: Ugh. It's

Jonathan: terrible. We're still recovering from that.

Benjamin: I literally cannot remember how to do a for loop in Bash for my life. Oh. And before LLMs, every time I had to write a Bash script, I was sitting there on Stack Overflow "How the hell... Where does this flag go?" "How do I do this? How do I do that?"

Jonathan: Yes.

Benjamin: It's, it... Oh, God, what a nightmare.

Jonathan: So yes Bash is amazing and also terrible at the same time. Yeah. W- we actually covered a a programming language that compiled to Bash code. Oh, I'm trying to remember what it was called. It was really cool, and they were trying to solve that problem.

They were trying to let you- That's cool ... let you write Bash code that was a little bit more elegant and modern. Oh, I forget the name of it. Anyway, it was a very neat project I've done a little bit with. I kinda feel because we're in the future now- ... that there is a third question that we're gonna have to ask everybody, and I'm still trying to figure out exactly what the question is gonna be, but it's something like, what's your favorite model to use to write code right now?"[00:57:00]

Benjamin: Oh, God. That's another hard question.

Jonathan: And it's a hard question because so much of the time you leave it on automatic and you let the, whatever agent you're working with flip between, its super-duper frontier model that costs you 5X for the hard parts, and then it flips back to, the, the cheaper one to do the research.

And sometimes you don't even know exactly which model it is that's writing your code.

Benjamin: It's if there wasn't the safeguards on it, it would be Fable. But the safeguards make it impossible for me to do my work. I can't review code, I can't fix bugs, I can't detect bugs. So- Interesting

and it's also really expensive. So the, the- Sure ... the holy grail right now definitely has to be 5.6 Sol. You can't go wrong with it.

Jonathan: Interesting. Good to know. Yeah, I'm surprised you guys don't have access to some, some unfettered Fable.

Benjamin: E- even if we did have access to it, like I, I think it would be very limited in how it would be able to be used.

So it's like you, you gotta keep in mind, like these [00:58:00] models... Like Anthropic just pub- published something actually about Mythos and like the requirements that they're putting on people who run it. Like it has to be in a very specific cont- like sandboxed environment, no internet access, 'cause they're very concerned about the capabilities of these models and whether they're gonna hack another company again.

So it's like the tricky thing with- Yeah ... something like that is like- True ... it becomes a lot less easy to use. Yeah. When the US government has shut you down a couple of times for it, you do have to get a little paranoid about it I suspect. Yeah.

Absolutely. And when you've committed felonies with it, like- You've

Jonathan: accidentally committed felonies.

Benjamin: Yeah.

Jonathan: True. True. That, that's not a good look.

Benjamin: No, not at all. All

Jonathan: right. Benjamin, it has been a joy. It's been a pleasure. I appreciate you coming on so much, and hopefully in about six months we'll talk to you again, but thank you, man, for being here.

Benjamin: Absolutely. Thank you for having me.

Jonathan: Yeah. All right. That is Benjamin Samuels from Trail of Bits, one of those companies that, boy, when I was doing the security column, we would cover their stuff from time to time, and just been a fan of the company for a long time and now learning about their [00:59:00] Patch the Planet initiative.

Definitely needed, definitely out there solving real world problems for open source projects, so great stuff. We do have a couple of more s- shows scheduled. Next week we're talking with Oliver Vernon about Update CLI, and then the week after that we've got the guys Jan and Alexander from the Open Source Automation Development Lab.

Then we're taking a week off, and then after that, the schedule is open for a bit. So if you know of or are an open source project that should be on the show, let us know, floss@hackaday.com. Other than that, we appreciate everybody that's here. Whether you watch or listen, whether you get us live or on the download, we appreciate it, and we will be back next week.