Episode 876 transcript

FLOSS-867

Jonathan: This is Floss Weekly, episode 867, recorded Tuesday, March the 24th. Pangolin, people can lie. Hey folks, it's Time for Floss Weekly. That's the show about free Libre and open source software. I'm your host, Jonathan Bennett, and today we're going to talk well. A little bit of IO ot. We're gonna talk security, we're gonna talk about something that claims to be a replacement for A VPN.

Among other things we're gonna talk Pangolin, the open source solution for connectivity and updates and all sorts of, all sorts of other things. Um, I am not the expert on this. I don't know a whole lot about it yet, but I've got the guy I know, I know a guy, and I've got the guy here. Uh, we've got Milo Schwartz and he is the co-founder of a company called Faial and also behind Pangolin, software engineer by trade.

He's done IOT and ot. I'm not sure what OT means in this instance. Anyway, we've got Milo here. Let's bring him on the show. Hey, welcome. Welcome.

Milo: Hi, Jonathan.

Jonathan: Hey, how's it

Milo: going?

Jonathan: Hey, it's great. Excited to here. It's great. It's good to have you here. Okay, so first off, I know what iot, the Internet of Things is.

What's the OT sector?

Milo: Yeah, that's a good question. Um, it's operational technologies. Oh, okay. Right. So it is informational technologies, OTs, operational technologies. Uh, and that tends to mean there's some physical thing out in the world, like, uh, maybe it's a, you know, camera, right? Mm-hmm. And you're connecting to it and then providing digital services on top of it.

Jonathan: Gotcha.

Milo: Yeah.

Jonathan: Alright. So what, tell us the story starting, I guess, with Al Well, how, how did all of this come to be? Uh, what is this that came to be that we're talking about?

Milo: Okay, so yeah. So it all started without a name, right? I think most, um. Uh, projects, um, uh, that start as a hobby, right? You like the name is the last thing you're worried about.

Like, lemme build the thing first, lemme get it out there. Uh, so,

Jonathan: or, or you pick the name first and you realize five years later that it was a terrible choice, but you're stuck with it.

Milo: Exactly. That hurts a little little. That hits a

Jonathan: little close to home. Yes, I know.

Milo: Um, anyway, yeah. So I actually started this, um, project, this tunneling project, uh, about a year, a little over a year ago at this point, like maybe a year and a half ago.

Okay. The fall of 2020, uh, excuse me, four. Um, I was just kind of looking at open source projects to build, uh, in the connectivity space. Um, and, uh. With Pangolin, um, we're like trying to find names, right? We wanted to, the one thing you do is you try to find a domain, right? What, what domain can I get that has a nice tld, like a.com or a.net, right?

And there are very few like nouns available, unsurprisingly.

Jonathan: Yeah.

Milo: Yeah. So, uh, we, we were looking at just like what different, um. What's affordable, and then what can we use that gives us a nice space of names to name all of our products, because we wanted to be able to say, okay, this is the company name, and then the product name is X, Y, Z, and Fal is a, uh, classification of animals.

It's any animal that tunnels is a, is a faial animal.

Jonathan: Oh, nice.

Milo: Yeah.

So,

Jonathan: oh, that's really, I like that. That's really clever. And you've also got the, the Foss as the beginning that everybody's gonna recognize

Milo: e Exactly. It kind of was too good to be true. Um, so it, and the domain was like 2K, so it was within reach, you know, if, if we started to make some money from, uh, the, the project.

Uh, so we, we were just like, let's, let's do it. Um, and we, it allowed us to pick cool names like Pangolin or Newt, uh, or Om, which are all different types of Faso animals for our products.

Jonathan: I like it. You know, I've, I've been told that whenever you go to name an open source, a company in general, but open source company, open source project.

When you think you've got your name picked, what you need to do is go punch that into Google and make sure that you're gonna be able to get your hit in like the top three results. Yes. And so I, I had, I interviewed one time a group that, uh, scarf was the name of their project I think. And it was super cool.

It was like, you know, build your own API, I think, I think is what it was. But, uh, I went to Google for them and I could not find their project because, you know, scarf Team U and, and Amazon and Walmart and Target. Like, you get all those results instead. It's like no scarf programming and then you start getting results.

Here's how you build a machine that you program that'll knit you a scarf. It's like, oh, I didn't want that either. So go do the research, make sure you can. You can, you can take over one of those top spots on Google.

Milo: Well, you were talking about regretting the name later. I think that's our biggest regret is Pangolin.

You know, you get all sorts of cute pictures of Pangolins that come up. Right. Even there's a documentary that came out around the same time we, uh, launched the project that dominated. Um, and, uh, actually in COVID too, there was the whole Pangolin COVID controversy. Mm-hmm. So that, that tends to, to bury us.

But we're, we're working our way up the SEO ladder at the moment.

Jonathan: Yes. Um, didn't, did Ubuntu use Pangolin as their animal for one of their releases too?

Milo: Yeah, there's that, there's, there's the Ubuntu Pangolin, there's also, I think System 76 has a, a laptop code named Pangolin, so,

Jonathan: oh,

Milo: it, it pops up a lot.

Jonathan: Yeah.

Yeah. Good. Okay. So is, is Pangolin at this point kind of the, um, uh, the flagship product that you guys have?

Milo: Yes. Yeah. Pangolin iss the flagship product. Uh, we actually, our, our domain now I mentioned forso.com. Uh, we, we do own that, but. Uh, everything now goes to pangolin.net. Okay. Um, as the main domain. Yeah.

Jonathan: Got it. Trying to get that SEO juice. Uh, so what you mentioned tunneling, and this is something I've done a lot over the years in, in various ways. What's, let's start with this. What's like the, what's the problem that you needed to solve? Why, why did none of the other solutions for tunneling work and then what was the sort of differentiator with bangin?

Milo: Yeah. So yeah, there's, there's a bunch of ways to tunnel. Um, we, we started as an open source CloudFlare tunnels alternative, so,

Jonathan: okay.

Milo: Um, what the type of CloudFlare tunneling, um, is, is it's outbound only tunneling, so it's not peer to peer necessarily. Um, and where this was born out of was my background in the iot space, um, which, uh, I don't know if you wanted to get into that now, but it's, yeah.

We, we, uh, basically, you know, in the IOT space there's, there's, you put these devices oftentimes behind cellular networks mm-hmm. And they're behind CG app. Um. And the big thing there is how do you get, how do you expose something when you're behind a network that you don't control the firewall? Uh, and you just need to get access to some web panel for remote management, remote control, remote troubleshooting, whatever it may be.

Jonathan: Mm-hmm.

Milo: Um, so there really aren't any good well packaged like open source, um, reverse tunneling, reverse proxy tunneling, uh, solutions out there that have a, a nice little binary and then do forward off and let you manage your different sites. So we, we had looked for things to use at our, um, our former company, um, the iot company before starting, uh, penguin.

And, uh, penguin was kind of the thing we wish we had at that time. So that's what we started working on that as a hobby project and then, and pushed it out. Um. Yeah, that, that's, that's kind of the, the high level, like what bread the, or what was the need and then what led us to, to building it

Jonathan: that I I will just, I will just say quickly, that is a great way to pick a project and to even to start a business Yeah.

Is to ask yourself like, what thing do I really wish existed? What would make my life easier? And can I build that? If the answer is yes, go do it. Make money.

Milo: Yeah.

Jonathan: Yeah. Well,

Milo: and, and I'll be clear like there are, there, this is not a new idea. I mean, there, there is N Brock, there's FRP, right? Sure. Like there's things that do this.

Um, there, the, the big one for us was the packaging. Um, we wanted a simple like binary or simple container that you could just chuck onto Chuck anywhere, right. And then have. Visibility into that network and then be able to define resources on, uh, on that network to then both do CloudFlare tunneling and then later, which we can get into.

Uh, now we do peer-to-peer tunneling, so more of a traditional VPN.

Jonathan: You know, if I were to try to solve this problem right now without thinking about Pangolin, I would, I would reach for something like wire guard or even open VPN I've used to open VPN in the past. What's, what's sort of the differentiation there?

Milo: Yeah, so penguin's built on wire guard. Ah,

Jonathan: okay.

Milo: Yeah.

Jonathan: Very cool.

Milo: Yeah, it's, it's, uh, not smart to reinvent, um, uh, an excellent tunneling protocol on encryption protocol. It's not, why not do it? Right.

Jonathan: So, so when, when, um, oh, I can't remember his name. We've interviewed him too. The, the dev behind wire guard. Um, anyway, Jason.

Yeah, Jason, that's right. They, he talks about, um. And in some of the docs they talk about that wire guard does this one thing very, very well, but if you want to do, and then they've got this like DHCP over the, over, over wire guard or, um, all of your configuration stuff. It's like you need some other service on top of it.

And so that's, is that sort of where Pangolin fits in it? It is the other service that gives you all of that configuration and, and automatic deployment stuff?

Milo: Yeah, it's one of them. Um, yeah, I, I mean I think, uh, there's, there's all sorts of projects built on, on wire guard these days. I mean, famously tail scale dominates the mesh networking space at the moment.

Um, I believe. CloudFlare even uses wire guard for, uh, so

Jonathan: yeah,

Milo: for some of their internal tunneling. Um, we used to use that our, our former company, um, for also like getting into devices. But yeah, the, the big thing with wire guard is it's very low level. Mm-hmm. Or you have to manage keys, you have to manage, um, uh, you know, it's not based on users.

If you, if you wanted to like, bring on a new contractor to help you with something, you would have to generate new keys for them, send down SSH

into

Milo: the

Jonathan: server and run the script and generate the keys. Right. That's a pain. I, I, I understand. Been there, done

Milo: that. Oh yeah. And I, I've also set up countless, like you spin up a new cloud environment v PC or something, and you got, first thing you deploy is you play wire guard.

Tell them to get in there.

Jonathan: Yeah.

Milo: Um, oh,

Jonathan: absolutely. Okay. So, um, let's see. We, we, we've got, we've got this problem space. What was the, uh, what's, what's sort of the, you built, you built it on top of wire guard. What does the interface like, what's the, the actual step two process look like to actually use Pangolin?

Milo: Yeah. Yeah. So we, we try to, um, sort of break things down outside. We try to abstract the network, right? Um, the whole part of the whole inspiration for Penguin was people these days are thinking about applications. They're not thinking about, uh, and identities. They're not thinking about ips, they're not thinking about overlapping subnets, et cetera.

Um, so we wanted to boil access controls down to, um, sites, which are like your, these are your physical, physical networks. Mm-hmm. Um, or virtual networks, even if it's like a BPC. And then also resources and users. So resources are the applications, the things you wanna get access to, and users are the people, right?

The, your personnel that need to get access to those resources. Um, so in terms of step-by-step process, it, it's very quite simple. You just deploy the site connector thing, um, which is, we call Newt, just back to the naming scheme. Uh, newt's a little salamander lookin creature.

Jonathan: Mm-hmm.

Milo: Um, and then that gives you that entry point into the network, hence why it's called a connector.

You define resources for things that they don't necessarily have to be on the, the actual location of the site itself. It doesn't have to be on the, the wire guard pier. It could be in the address. It just has to be in the addressable range of the pier. Mm-hmm. Um, and you can, uh, then give your users access to those resources and the users when they, they can access to the web browser or they can connect in through, uh, client.

And, um, you know, it doesn't matter if you have 150 sites and locations, you could have 150 connectors out there with, you know, 10 resources on each of those locations. And the users will be able to access all of them, um, at once without having to jump between sites.

Jonathan: Yeah, very cool. Um, I've, I've seen, I've seen the need for this sort of thing firsthand.

Uh, actually for a little while I was playing around with the idea of building something similar to this, uh, but then building a physical, a very small portable physical appliance. And the, the, the point here, the thought process here was one of the other hats that I've worn through the years of my career was a telephone service.

And so there's been multiple times that I've rolled out to, you know, a hotel or what have you, and they've got, you know, they've got a big AVI switch or whatever, whatever brand, right? And the tech that I'm on the phone with at headquarters, he needs to be able to remote into it to fix something. And you, I mean, based on the fact that I'm there, you know, something is broken, so the, the backup modem is unplugged or somebody stopped paying for the, the POTS connection.

Something has changed with the internet connection. And so they don't, they no longer have an IP address to dial directly in. And so they just want to be able to get some kind of connection into that phone system to be able to do their thing. And my thought process at the time was, it would be cool if I had just like this little router that it would, on one side, it would get an IP address from DHCP.

Wire guard would connect it, go up, create the tunnel, and then on the other side you would have sort of this managed connection that you could plug into the switch. Yep. And everything would come up and they'd be able to jump in. It sounds like that's what you've done except in software. Have you thought about making a hardware version of this?

Milo: Yes, we have thought about it. Yeah. So this comes back to, again, to the roots. So, um, uh, the iot world, it's very, and and the OT world, it's very popular for people to deploy edge devices. Mm-hmm. And there's this whole world of. Of, uh, manufacturers. There's Tonica, there's a Van Tech. Um, the GLI nets one that's a little bit less in the industrial space, but the whole idea is, let me just buy this device.

I can plop it somewhere. Mm-hmm. Um, it, all it needs is an internet connection, and I can get in and I can access things. Um, so we have built a software, we built like a, we like to think of it as a hardware agnostic, um, version of what some of these providers provide. 'cause they're all, they're like tied.

Tonica has a kind of a solution to this, but it's tied to their box. Um, so we, we, we are considering hardware. Um, uh, it's, I don't think it's our top priority at the moment because, uh, because of the whole hardware agnostic thing, right? Mm-hmm. That's like, like it's part of our edge at the moment. Um, but it, it's certainly open, uh, to consideration like selling a, a cheaper box.

Uh, maybe, you know. Like a, a raspberry pie that's well packaged and then deployed something that's not quite as expensive as some of these hardened, like, edge devices that could be overkill for just simple access to a phone system.

Jonathan: Yeah. The, the deal with that, and this is what I ran into, is, was the chicken and egg problem, right?

So like there's the, there's the engineering requirements to make the thing, you gotta put some money and some time and some effort into it, and you've got it. But then that's not any sales. You've gotta also have somebody on the other side that says, yes, yes, we like that idea. We will pay you money for it.

Right? But to get that sort of commitment, sometimes you've gotta have the device in hand that you can show them. So that was, that was where I got stuck, is trying to

Milo: Yeah. Hardware's hard.

Jonathan: Yes.

Milo: Hardware's very hard. Uh, yes. For that reason. You have to manage inventory, uh, things break. It just adds a lot more variables.

Jonathan: Yep.

Milo: Um, I'm definitely much more of a software guy, um, at the, again, former company and mainly doing software. There's someone else doing the hardware. Um, but, uh, I, I like. I like seeing how they can, how they can marry.

Jonathan: Absolutely.

Milo: It's, it's, it's more exciting. Yeah. Yeah.

Jonathan: Yeah. Agreed. Um, so you've got a note here about, uh, Nat Traversal and corporate firewalls, which this is something that I also have fought with multiple times over the years.

Yeah.

Milo: Yeah.

Jonathan: Uh, have you seen that our, our corporations starting to try to block, uh, wire guard? Is that something you're running into?

Milo: Uh, actually yes. Uh, there, there is a bit of that, um, not necessarily blocking wire guard, but blocking just, just very hard nats where they're doing, uh, maybe port randomization or they, uh, are blocking all like, or UDP traffic, which obviously makes it very hard to, to handle the tunneling.

Jonathan: Yep.

Milo: Um, yeah.

Jonathan: Yeah. This, this was something that. Uh, another, another project that I was involved in years ago was a, uh, a port knocker. It was a FW kn, that's how we said it. Uh, or wha kn. Um, it was a, it was a port knocker that actually used real encryption and authentication inside of it, rather than just the list of ports.

Uh, but that was something we ran into is that, you know, all of the, we were gonna send outgoing UDP packets and all of these corporate firewalls would block them. And it's like, no. Yeah, that's kind of a downer, isn't it? Um, you guys, and, and I'm afraid that at some point the corporate guys are gonna realize that wire guard exists.

And that it is a potential avenue for exfiltration, which that's something that those guys, you, you have to, once you get into like a certain, um, a certain level of business or working with a certain type of data, you, you, you're required to care about this, right? So it's not, um, it's not entirely a ridiculous concern, but they, they care a lot about things like exfiltration.

They don't want a disgruntled employee or a spy or what have you to be able to send data off the network. And, uh, my fear is that at some point, these corporate guys are gonna realize that wire guard exists and they're gonna start looking for it and blocking it.

Milo: Yeah. I, I, I don't doubt it. I mean, I know, um, uh, many public wifi networks, uh, like famously schools will block VPN connections because they want to.

Track and block. Mm-hmm. Uh, internally and a v VPN's a way to circumvent that, but that's a, it's a different type of V vpn that's more of a, a nor Nord VPN or PIA that they're, they're going after and less of a remote access.

Jonathan: Right. Well, but I mean, the packets kind of all look the same.

Milo: Exactly. Right. Yeah.

There and there's ways to get around it. I mean, so there's, um, uh, there's on like the, the nor VPN side, there's ways to wrap packets in, uh, like an SSL wrapper, so that way it appears to just be normal web traffic.

Jonathan: Mm-hmm.

Milo: Um, or proxy it through some soc server. And then, um, it also appears to be web traffic.

And, and I think those are pretty successful. 'cause at a certain point, the, uh, the firewall has to make a, a guess. Right. And it's, I guess how, how lenient do you wanna be on that guess?

Jonathan: I, I'm trying to remember. I've, I've done some reading about technology developed for like the Chinese great firewall.

Yeah, people trying to tunnel, um, either, uh, either just VPNs or in some cases they were trying to tunnel, um, tour through it. And some really interesting technologies people have come up with, like hiding tour traffic inside of. Uh, telephone, like VoIP calls. Mm-hmm. Um, yeah, one in one in particular, like you would create a Skype call at first, and then you would, you would enter, exchange the internal traffic to be, instead of Skype, it would now be tour.

Um, do you, have you guys sort of borrowed any ideas from, from the stuff going on there?

Milo: Not really. So, uh, in, in terms of natural universal, I'll go back to the, the original question. Uh, we're, we're mostly doing, um, mostly doing port hole punching. So the classic it, it's what again, tail scale does this.

Mm-hmm. Uh, zero tier does this without wire guard, but you know, you fire packets from both directions, both the site and the, the user's client out to a central coordination server. Mm-hmm. Just fire a big stream of just random U DP packets. Basically, you look at the two ports that were opened on the, the two gnats, and then you just, you point them at each other, um, to do that peer-to-peer.

So that's, uh, in terms of nat reversal, that's uh. Really the extent that we go there, uh, really is it like a stunt server like

Jonathan: you would see in sip? So essentially what that is,

Milo: uh, it's very similar. It's actually, so it's our, we, we did not follow the stunt protocol because, uh, we, we thought we wanted to be special.

So

Jonathan: you said

Milo: it not

Jonathan: be

Milo: Yeah, yeah. No, we, we, uh, we thought it would be more, more fun and just a little painful to, to develop our own. So we have our own protocol, but effectively if you understand stun, you understand turn it basically works the exact same way.

Jonathan: Okay.

Milo: Yeah.

Jonathan: I mean there's probably some advantage to not being identifiable as a stun server and doing turn

Milo: right.

Jonathan: But, uh, it's, that's interesting. Uh,

Milo: yeah. And, and sorry to interrupt, but there's, there's also the, um, the outbound tunnels. So the, the peer-to-peer thing makes a lot of sense for a VPN connection mm-hmm. Where the person's running a, a client to be able to connect in. But when in situations where you aren't able to run a client, um, and you just have a web browser.

Then you, that's where the reverse proxy comes in. Mm-hmm. Um, because you can just establish an outbound only tunnel from the site connector to that coordination server in the cloud. All traffic will hit the coordination server basically as a front door or a gateway and then go down. If it's allowed traffic, it will go down the right tunnel to the right network.

Jonathan: Yeah. Very cool. And so you guys, you, you mentioned this briefly, but you have like the idea of sites and resources to where you can have one device sitting on, uh, a network somewhere, but then what you have sort of a, a, a little cloud of IPS around it that are accessible.

Milo: Yeah, basically. Um, so I guess one another difference between Pangolin and wire guard is, uh, you know, wire guard's all based on peers.

Jonathan: Mm-hmm.

Milo: Um, so you, you get access to the pier and what's on the pier, uh, unless you set up proper routing. So what we've done to sort of flatten the network. Is, um, we've added a proxy, um, in front or I guess at the end of the wire tunnel. Okay. So traffic can come outta the wire guard tunnel and then it gets proxied out to another destination.

So this is built into our, our new connector. Uh, and what that lets you do is yes, you, you can define different resources in the addressable range of the connector, um, that, uh, you can then get to and you can actually, um, refer to them or, or, you know, your, your users can connect to them via the familiar land addresses when connected into the network, um, even if they're on overlapping or, or d completely different physical networks.

Mm-hmm. Uh, which is a difference between us and something like a, uh, again, a tail scale or traditional traditional wire guard where you, you would just be addressing nodes within a network.

Jonathan: Mm. The other advantage I would think is that if you just set this up with wire guard and the routing, then you can get to all of the ports.

And it sounds like you guys, you, you give some administrative control to say, you know, you get to talk to port 4, 4 3 of this device and that's it.

Milo: Exactly. Yeah. Yeah. So you, you define a resource and you say exactly which ports are available on this resource. Uh, you could block all TCP block, all UDP, you give a range, et cetera.

Um, and then you set users of roles on that, on that resource. And, and yeah, of course you can do this with, you can throw a, a wire server somewhere and you get access to everything on that network. Um, but you, you can't be connected to multiple at the same time easily. Um, traditionally it's, it's either, you know, you pick a site, um, and, and that's your entry point.

So Penguin lets you sort of link together different networks in a, kind of, in a way.

Jonathan: Yeah. Very cool. So you guys, you needed this tool, you built this tool, you decided to open source the tool. What did, what did that conversation, what did that decision look like to open source? It? Was it just always the plan or was there at some point you had this business discussion like, this makes more sense if we release it under the no GPL or the MIT or whatever.

Milo: I think business came second or third or fourth. I don't, I don't even know. It didn't come first. Um, so we, I was, I always wanted to run an open source project because just been being involved in this space, um, for a long time. And, uh, most of my hobbies have tied back to the home lab in some way.

Jonathan: Mm-hmm.

Milo: Um, I think like many people in this community, so I, I was deploying servers and, and deploying image, you know, all that, all the good stuff that you would expect next cloud. And I saw like. Now I kind of like looked up to some of these maintainers as, as like role models a little bit and was like, I, I wanted to be one of them.

Mm-hmm. Um, I also liked the idea of actually having people use my software. Um, uh, again, coming from that other company, we, we were selling to a very select group of people. So we, we, we made money, but on very few people. So it, it was just less gratifying. Yeah. Uh, you get less feedback. Um, so we wanted, knowing all of that, we wanted to, um, create our own open source project.

We wanted to grow naturally, um, which is the ideal scenario. People share word of mouth. You get all sorts of people from all over the place using it, and that all feeds back into the projects. You now can, the community can basically pull it up into, um, a place where it's beneficial to everyone and gives you ideas, right.

For free. Mm-hmm. Right. I mean, people get very passionate about open source software, um,

Jonathan: to def Definitely,

Milo: yes. Yeah. Yeah. That, that was, that was the arc of it. It was the, the intention from the beginning was to open source it and we launched it on Reddit. Sorry.

Jonathan: Okay. So you said you, you'd always dreamed of being a dev a, uh, a maintainer.

You always wanted to have an open source project. You always wanted to have a lot of users that were using it. Now that you have those things

Milo: Yes.

Jonathan: Do you regret it? I don't regret it to, for those that have, have not been in this position, um, there are downsides to having, being the guy at the top of an open source project and having users that will sometimes ask for ridiculous things and get mad at you for no reason.

And like, there, there are, uh, it's, it's not all sunshine and roses. There are some thorns with running an open source project.

Milo: Yes. No, it's open source. Uh, well, I like to say it's, yeah, it's a very, very sharp double-edged sword.

Jonathan: Yes.

Milo: Um, yeah, you, it's, it's a double-edged sword. People are very opinionated, um, about software.

Uh, and they are, you know, they wanna share it, right? So I think the, the biggest problem is, is simply, is, is simply just managing, um, so much mm-hmm. Conversation. It, it becomes like a, a community management problem, right? Once you get to a certain scale.

Jonathan: Yep.

Milo: Uh, where do you have, how do you triage issues?

Like how do you. Manage support threads. Where do you manage support threads? Mm-hmm. Uh, how do you block spammers who are like coming in and are trying to take advantage of people? Um, oh, like now the founding team can't manage all the support. Do we need to pay someone for support? Like, there, there, there's so many, so many things there, um, that are make open source very hard.

Jonathan: Mm-hmm.

Milo: Once you reach a certain scale.

Jonathan: Yeah. Absolutely. Um, how about the, how about the pull request side of things? Have you guys, are you guys being innu in, in inundated with, uh, slot pull requests like everybody else?

Milo: Yes. Yes.

Jonathan: What, what,

Milo: unfortunately,

Jonathan: what, what have you done about that? Do you have a solution yet?

Or are you still waiting through it like the rest of us?

Milo: We're waiting through it, so we make people disclose, but there's, you know, people can lie. Right. It's, it's an honor

Jonathan: system. People can lie show title right there. Yeah. People can lie.

Milo: Exactly. Yeah. So we have a disclosure on the pull request and, and all that really ends up doing is it makes us just be heck of a lot more skeptical when looking at it.

Yeah. But we still have to scan for, um, scan for ourselves. Uh, we try to write a, a big sort of con contribution guide, contributors guide that includes best practices and even areas in the code to look where. Mm-hmm. Um, if someone were to vibe code, they, you know, they could probably just take that document and it would be a good source of context.

Right. For the, for the LLM, um, which would I improve to a small degree? Maybe the, the types of pull requests we get. Yeah. Um, but ultimately we still have to, um. You know, search for security errors. Uh, the big one for us is making sure, you know, the LLMs stick to ux UI patterns that we follow. 'cause they, they like to kind of go off on their own and like create their own components and, and ways of doing things.

But that's not good user experience and, and our whole like thing is user experience. So, yeah. Um, yeah. Answer the question.

Jonathan: Yeah. Yeah. Very good. Um, I, I pulled up the, uh, the Pangolin, um, GitHub repo here, and I see you've actually got a couple of licenses and I was curious about this. Um, at the very least I see the, the GNU, uh, A GPL, the aero general public license.

Um, but I also see that some, some files are also, uh, the faial commercial license. How do those two play and interact together?

Milo: Yeah. So, um, you know. As a business, you need to make money, right? So, um, uh, there's, there's a few ways you can do that. You have to, uh, sell something that is of valuable to people.

So at some point we decided we had to create features that are, uh, considered paid features.

Jonathan: Okay.

Milo: Uh, right. And these are licensed, not very GPL. However, um, we, we were very careful about this 'cause this is very touchy subject for the open source community as I'm aware. So, um, we, we did two things. One, um, the open source, we build two versions of the application.

One is fully built and distributed as a GPL three. So all the proprietary code is stripped out. Okay. Um, before it's distributed. And two, the part that is, um, licensed under the enterprise edition is free for, um, anyone using it for personal use and for non-commercial use under a certain threshold. Okay.

Uh, 'cause we wanted to make it as accessible as possible to people. Um, and so far I think this has been working pretty well. Mm-hmm. Um, it, you know, it. Uh, enables access to everyone. Um, the people who are, you know, need, um, it to be licensed a certain way for distribution under certain open source, um, restrictions, can use the H-H-G-P-L version.

And people who either don't care, want the extra features and are are hobbyists from home laborers like I was, um, when I started, can get access for free. And then the businesses who are getting real value out of the product, um, can pay us a little bit of money. Right. So we can fund development Yes. And fund all that support that I mentioned earlier.

Jonathan: Yes.

Milo: No, I, I,

Jonathan: I am, um, very sympathetic to that. Somebody's gotta pay some money because the programmers have to be able to eat and pay their rent. I, I'm right there with you. I, I totally get that. Yes. Um, ha have you found that, uh, so one of, lemme back up a little bit. One of the, one of the real concerns that open source projects have had, and I think this has sort of died down some, but therefore a while it was the big deal was.

A company like Amazon coming along and selling your product, selling the exact same services that you're selling for half the cost because they're Amazon and they can, um, and that, so that had to be like one of the things that you were thinking about. Um, but have you found, as I have that the A GPL is just, uh, terrifyingly effective at keeping someone like Amazon away?

Milo: Yes, I think so. Um, I think we're, yeah, like, I mean, I, I saw the whole Redis famously. That's, that's the one that kind of shook us, or even WordPress I think is having all sorts of issues

Jonathan: at the moment. WordPress has Yeah. Is kind of fighting between a couple. The, the, the WordPress company itself, and then WP Engine is the name of the other group.

I've not, I've not looked up and seen, seen where that is recently, but it is kind of the same deal. Right? It's an outside company offering some of the same services, right. And competing against the guys that run the open source project.

Milo: Yeah, I, yeah. I mean, um, do you know what the licensing model for WordPress is?

Is it. I licensees.

Jonathan: Um, I think the core is all open source and then they make most of their money by selling hosted services, I think.

Milo: Hmm.

Jonathan: I, I don't, I don't know all of the details, but I know that's at least part of it.

Milo: Yeah. Yeah. I think A GPL is, is quite effective at doing this. Mm-hmm. Um, it, it is one of the more restrictive licenses, that's for sure.

Uh, prevents, you know, uh, it's copy left. Of course if you, if you modify the source code and then you incorporate it in your product, or even if you access stuff over a network, that's the, the big one with, with A GPL and interact with the software. Right. Um,

Jonathan: it, it closes the, it closes the, what do they call it?

The cloud loophole, the web loophole of the, of the GPL. Yeah. So I, I, we should clear off the spot real quick and explain this. So like the GPL says that when you have it, when you have something licensed under the GPL and you give that binary to someone else, that person has the right to come to you and say, I want the source code that goes with the binary.

Well, when you're providing a service on a server up in the cloud, you're not actually giving the binary to someone, you're just giving them the service. And so the GPL doesn't have the same teeth. In that case, a user, you know, I use Gmail. I know that Gmail has a bunch of GPL code in it, but I can't go to Google and say, Hey, give me all of your Gmail code.

They will laugh at me. Um, because it's GPL, but the way it's written, I don't have a binary, so I don't have that right. The A GPL was the solution for that and, and it was come, it was written by the same folks as the GP L's, from the gnu. It's from the Free Software Foundation and it basically says not only do you have the right to get that source code, if you're given a binary, you essentially have that right to get the source code if you are a user of a service running this code.

And so it does exactly this. If, if, you know, if, if some part of Gmail was licensed under the A GPL, I could then absolutely do that. I could write to Google and say, Hey, I am a user of your service under the A GPL. Please send me a copy of all of your source code. And the license would require to do that.

And because of that, Google and Amazon and all of those big cloud companies have very, very strict regulations that no one in their company is to touch a GPL code. Like, don't even look at it for too long. Um, you know, I imagine in their, in their programming cubicles, they must have like the little eyewash stations.

Like if you look at at TVL code for too long, come run to the eyewash station and wash your eyeballs out. Um, it's almost, it's almost that serious, uh, because man, it would, it would poison all of their stuff in, in the way that it's addicted to, right? And so that's just sort of the, the, the tension between the two.

And so what a lot of companies have done is very similar to what you're doing. You have, uh, either a, a, a tiered approach with code, or you have a dual license with a, uh, uh, A CLA so that, you know, you're open source, that you're giving out to everyone. You can run with a GPL and nobody cares. It's fine. But if it's Amazon, well they gotta pay you the big bucks to be able to do it, which works out pretty well.

I mean, like, that's a fairly, it's a fairly ideal solution for everybody.

Milo: Yeah, no, I think so. I recommend the approach to anyone. Um. Of course you need to do a license if you ever do wanna do a commercial license. 'cause if people contribute code, you have to, um, you know, maintain the right kind of copyright.

Jonathan: Mm-hmm.

Milo: Uh, over that to redistribute it. Um, yeah.

Jonathan: Do, do you guys have a contributor license agreement?

Milo: Yeah, we, we have ACL a. We were very careful from the, careful from the beginning. Um, yeah. You, you asked about open source if that was our Yeah. First intention it was, but we always knew if this was going to become a company.

Um, that we needed to be transparent at the beginning. 'cause we, we've seen all the problems of like, changing licenses and, and making Yeah, like, right, like people contribute their, their time and their code and then you take the copyright away from them. Um, 'cause technically they own that when they, they.

Contribute.

Jonathan: Mm-hmm.

Milo: So we, we, um, even though the commercial side wasn't figured out at the beginning, we added a CLA just in case. Um, and there's pros and cons to that. I mean, obviously it protects the maintainers, um, uh, and the project, but it also means you, you, you get less contributions. 'cause some people don't wanna contribute if they have to sign ACL A.

Jonathan: Uh, something else that we found, I don't know if you guys have found this, some people can't legally cannot contribute with a CLA because they have signed an agreement with their company that all of that code, you know, hobbyist or whatever belongs to the upstream company. And so we've got some guys from a US company that would love to be able to contribute code to the, the one of the projects I'm involved in.

But we too have ACL A, um, for, for somewhat similar reasons. Ours is, ours is more, uh, I, for, I forget the name of it. Um, it's not so much about commercial use, but it's more about being able to fix a problem if there's ever a, an issue found with the license. Um, but anyway, we've had a couple of people that says, I've been talking to legal for months about this and we still can't figure out if I'm able to write this code.

Sorry. Do that. I think

Milo: that's, that's the big downside for sure. Um, yeah. Is is just concerns, I mean you mentioned it both on the Google right? They won't, they won't touch any A GPL software.

Jonathan: Yeah,

Milo: I think it's, um, fortunately I think there's, like, it matters more for dependencies of software, right?

Something like a library that you're going to integrate into your code. Right. And a little bit less for fully packaged ecosystems, which I think we fall a bit more into 'cause the software gets deployed standalone, um, and it's less interlinked with things, um, which I think helps us a little bit.

Jonathan: Yeah, yeah, absolutely.

Um. I just one more comment on this, on the idea of CLA and the business use case. You know, there's been, obviously there's been some real blowback in communities when, when businesses have done this, they've taken something and they've added either added A CLA or they've used a CLA to re-license code. Um, but I think if a project is started from the beginning and it's made clear that, look, this is open source, but it's also gonna be a business, we're going to have a CLA on it.

This is why, this is what the CLA allows us to do. Like most people get that, and particularly when it's. Stated from the beginning. I find most users are, are pretty open to that. They may not like it necessarily, but you know, they're not gonna

Milo: Right.

Jonathan: They're less likely to call. You mean words on Reddit as a result?

Milo: Yeah, I know the, the name of, I mean, the whole name of the open source game is transparency. So, um, right. Part of it's the whole idea get access to the source code, so you need to be transparent on every front. Um, and there's actually, you know, another inspiration of ours, but we haven't, um, we don't have the time these days to really go deep into this area, is companies like GitLab, uh, or Fleet dm, which are, have built this model around an open source company too, where all of their processes are, are fully.

Available on their website, even their sales processes, you know, you can see how they, they go after clients and stuff. Um, and as a, uh, uh, you know, as a customer right, of, of a company like that, um, I think that's as almost as almost more important than sometimes than seeing some of seeing the source code because you can see how they, they do, how they do things, how they operate their business, and do you trust them?

Do you, for lack of a better word, vibe with that?

Jonathan: Yeah.

Milo: You know,

Jonathan: I will say that there are some business models where that just does not work.

Milo: Yeah, of course.

Jonathan: I've got, I've got an email sitting in my inbox right now that I'm a little scared of that is marked not for, you know, do not distribute, do not blah, blah, blah.

It's like, uh, I would not, I would not be able to do that at all with that particular, um, that particular contract. But

Milo: yeah, no, I mean, there's levels to it, of course, right? Like. Um, I think, I think the big ones for me are roadmap, right? Roadmap is, is not necessarily, doesn't need to be held super close to, to the heart.

Sure. Uh, make that public fleet dm. I really like how they, um, do sprint. I think they, they do their sprint meetings, like their standup calls with the company. All of those are live streams so everyone can see. And I always thought that was super neat. Um, because usually that's internal. Like you have no insight into what the company's meetings look like.

Obviously you can't, you know, you can't have a confidential meeting on, on, on the public internet, and I'm sure they're not. They're picking and choosing.

Jonathan: Yes.

Milo: But it, it makes you trust them more because they feel a little bit better

Jonathan: about it.

Milo: Yeah.

Jonathan: Yeah, absolutely. So what, what does it look like to, to self-host Pangolin?

Milo: Yeah. So, um, uh, pangolin started fully self-hosted. Um, so open source, fully self-hosted. We didn't have a cloud. We had no idea how we're gonna build a cloud. The system's very complicated as you could imagine, with tunneling and relay servers and coordination servers. And when you're dealing with the reverse proxy now there's like, um, ways to sync certificates and ways to, you know, you have to move DNS and, and all that type of stuff.

Yeah. So the first version of pendulum that is self-hosted was just a stack of three containers. There's a, the penguin container, which is the sort of business logic.

Jonathan: Mm-hmm.

Milo: Um, all the penguin specific stuff. There's this other container called gerbil, uh, which is, uh, another animal named another Faso animal that handles all the tunneling.

The idea there is that. You know, gerbils, they, they move through. Like if you go, ever go to like a PetSmart or a Petco and you see all the tubes.

Jonathan: Yep, yep, yep.

Milo: Yeah, that was the idea there. 'cause like Gerbil is building that whole network. Um, and then finally we use traffic under the hood as our reverse proxy.

So it's actually handling, um, the routing and, uh, certificate generation with ryt and all that good stuff. So all that stuff gets deployed on a stack onto VPS and, and boom, there's your simple self-hosted single penguin server. Okay. Uh, but then you can graduate up, um, to what we call penguin clustering or multi-region deployments.

Mm-hmm. Um, depending on where, which one you fall into, um, this is where maybe you, you want high availability and fallback. If, if one of. One goes down where you want to put a point of presence on the east coast and a point of presence on the west coast. Um, and this is where it gets hard because if a tunnel disconnects Yeah, tunnel disconnects.

Like from one of those points of presence, it connects to another one. Well, now DNS was pointing to the first point of presence and how do you get it to the, to the other one? And the same with the certificate. The certificate was being served from one pop, served from another one. Um, and this is what CloudFlare does all day, every day.

Yeah. With billions of dollars. Right. So we were trying to figure out how do we do it with like.

Jonathan: Thousands

Milo: of

Jonathan: dollars at best. Yes.

Milo: Right. I

Jonathan: understand.

Milo: Um, so I think we came up with a pretty good solution. Um, there's, you know, a couple extra components you run. We have a DNS server, we have a certificate sinker.

We could, like if when traffic, when the tunnels connect, disconnect from one pop to the other one. If DNS is still pointing to the first pop, we use like an SNI proxy to actually send like traffic over to the second pop, uh, and then go back, like kinda goes out one pop and then into the next one. Huh. So that's all can be fully self-hosted, obviously.

I keep saying we, we also use this like in our cloud platform because we did eventually build a cloud platform, and that has to be like globally distributed and all that stuff that comes with the cloud platform.

Jonathan: So, so you guys were, you guys were containers from the very beginning. You are, as they say, cloud native.

Milo: Exactly. Yes, yes, yes.

Jonathan: If somebody says, I don't like those dang containers, is there a way to host it? Is there a way to get it running without containers?

Milo: Uh, technically yes, but it might be kind of hard. Yeah. Yeah. We, I, I don't think we have any official documentation on it, but I think you could reverse engineer it if you wanted to.

Jonathan: Yeah, yeah,

Milo: yeah.

Jonathan: Makes sense.

Milo: Uh, it's

Jonathan: funny.

Milo: Yeah. But we started a docker, um, and graduated app, I think as many do into like Kubernetes. Uh, the Real, real Men territory of, uh, cloud Native.

Jonathan: That's hilarious. Yeah. I used to, I used to be that guy for a while. I was like, ah, I don't like containers. Show me how to actually run the code.

And then one day it's like, okay, fine. I'll give this container thing a try. And I was like, oh, okay. Yeah. That's pretty cool. I guess I'll use it.

Milo: Yeah. Yeah. It's, um, there's pros and cons for sure. I, I, um, I guess we haven't received many requests for it, so we haven't like, put, uh, any real, real effort into documenting the process.

Jonathan: Can, can you scale it down all the way to run the, the Pangolin self hosted on something like a Raspberry Pi?

Milo: Yeah. Yeah. People are doing that for sure. Yeah. Uh, I mean, people, um, the way we started is, oh, you don't wanna use CloudFlare tunnels? Use Pangolin. How can I do it cheaply? Let me get a very cheap VPS, like one gig Ram one, one or two C bcps, right?

And, and deploy it. Um, that maps down to a small raspberry pie. As well.

Jonathan: Yeah.

Milo: Yeah.

Jonathan: About the same. Alright, so I've got a note here that you guys have 20,000 stars on your GitHub repository, which is a bunch. Um,

Milo: yes,

Jonathan: I am, I am tempted to ask if they're all real or not because there there was, there was some news stories about, uh,

Milo: oh, really?

Jonathan: Uh, yeah. Um,

Milo: what

Jonathan: happened there? I, I don't remember who it was that, and this, I'm, I'm, I'm not being serious here. Um, there, there was a few projects that some investigations showed up that maybe some of those stars were paid for and, and clicked to buy, you know, bought accounts of some sort. Uh, but I, I'm, I'm not making that allegation.

Not at all. It was, it was, uh, just an off the cuff joke. Um, but that, that is a bunch, that's a lot of people that saw this and went, wow, that's really cool. Um, what's it, what's it like to have that many fans? That many people watchers in the community.

Milo: Yeah. Yeah. Like, I mean, like I said before, there's a lot of people using the software, which makes us very excited.

Like we, we've pushed bugs out by accident before. Right. I think it's inevitable. And you're rem reminded by how many people use the software. 'cause you get inundated with, uh, I wanna like, you know, like, people reporting the book, right?

Jonathan: Mm-hmm.

Milo: So, um, it just, it just that many stars just bring scale, uh, stargazers are, uh, you know, they're a bit of a vanity metric for sure.

Um,

Jonathan: yeah.

Milo: And, uh, all it really means is, you know, there's, there's some projects that will get tons of stars and then disappear and not have any real usage.

Jonathan: Mm-hmm.

Milo: Uh, I feel like we've seen that a lot more recently with some of these viral like. AI products.

Jonathan: Yep.

Milo: Um, but I think generally, like on average, more stars correlates to more usage, so that's why it's like an important, important metric to see if a project is, has adoption, um, et cetera.

Yeah. So, uh, no, we did not pay for stars, but our, uh, unless you consider like yeah, we, we, we've exclusively just posted on Reddit, posted on Hacker News and, uh, our slash self posted has been where we, we've, uh, pretty much gotten all the stars and then people have shared it, you know, among themselves, which I think leads to a network effect.

Jonathan: Yeah, for sure. I wonder if I should start making that one of my standard questions. Have you paid for any of your sars on GitLab?

Milo: Yeah. I mean, I wonder, I wonder what the cost per star is.

Jonathan: I mean, it can't be much, but. Yeah. Still. Um, interesting. So in part of your background, you actually mentioned doing IOT and OT stuff.

Uh, you, you did traffic light infrastructure. Yeah. And so I'm curious if, if the circle has been squared, does Pangolin do any traffic light connectivity now?

Milo: Not quite traffic lights yet. Okay. Uh, but the, the industry is, um, very close. Like everyone's kind of doing the same thing that I used to do at this traffic company.

Jonathan: Mm-hmm.

Milo: Uh, you, I mentioned earlier, you know, you, you deploy an edge device somewhere to control physical hardware and for traffic lights. Every traffic cabinet has a traffic controller. These things are built in like the, the eighties and nineties. They're, they're, they're old, old pieces. They're older than me.

Jonathan: Right.

Milo: They're like, like, uh, these,

Jonathan: they're similar to some of the, the phone mainframes that I've worked on. Yes.

Milo: Right. Exactly. And, and, um. They like all the traffic lights, connect to the controllers and the controllers then have a management panel that you use the Edge device to get access to. So that's basically what I was doing at this other company.

Mm-hmm. Um, as well as many other things. But, you know, we, we have, we don't have any current customers in the traffic industry, but we have lots of people, you know, in industrials right? Where they're doing the same thing. There's A-A-P-L-C or some sort, there's some sort of like, um, you know, oil mining contraption, right?

Mm-hmm. Or like a, uh, BMS like Building Management Systems is another one where they're linking all these different pumps and like fans into a central sort of base station and then, um, getting access to it with an edge device. Um, so it it has all come Yes, full circle basically. Um, uh, I think part of the reason it's a bit of a self-fulfilling prophecy when you're part of an industry and then.

You build a product, it's very easy to sell, you know, and make a, a value proposition to that other industry. Sure. Because you know how to talk to them, you know, what the pain points they experience are. Um, the big one, the big pain point, right, that comes up a lot in these iot fleets is provisioning, right?

How do you provision these devices? Mm-hmm. Contain our site connector, contain any other software, do it on a repeatable basis, and then actually push it out into the field and manage it. So there's, there's management involved with that. Um, and existing tunneling solutions like tail scale, um. Are are not really built for this in mind.

Right, right. So we, we have some extra pieces in there that make it very easy. Uh, like a, a site just needs an ID and a secret, and then we have like a provisioning key, which you can use to, uh, you know, map to your devices and then it will pull down e femoral keys for, for the, the device at the point of provisioning as it comes off the conveyor belt and they get pushed out into the field.

So there's like little things like that that we do to optimize, um, that, um, kind of bring us back into the, the OT or IO OT sometimes called IIOT, which is industrial iot.

Jonathan: Got

Milo: it. Space

Jonathan: learning all kinds of acronyms today. Yeah, yeah, yeah. Um, is, is the, it's the traffic light industry and all of that. Are they still doing like raw IP provisioning in some cases?

This is something I, again, I've seen in the, the, the phone system world is, uh, you know, okay, well here's your phone cabinet and just give it this public IP address. Oftentimes no firewall in front of it or anything. It's like,

Milo: ah, yeah,

Jonathan: I'm sure it'll be fine. It's not always fine.

Milo: Oh, yeah, no, I mean, there's, um, it's crazy in the, we've heard this from people using Penguin in the building management space, uh, but also my experience in the traffic space, there was all sorts of just open ports for, um, public IP addresses.

You could, if you were scanning, you could find traffic controllers and get to the front panel. Um, and it is terrifying because mm-hmm. All this industrial stuff, all this industrial hardware and software is super old stuff. So it's, who knows what kind of vulnerabilities are in there, um, that can be exploited.

So part of the goal, the push right now in these industries is to close down ports, um, keep everything off the internet. And in fact, even sometimes, let's not let people connect in. Uh, like a VPN 'cause that extends like, you know, that, that to, uh, actually extends the network, uh, boundary a little bit. Right?

So they only wanna do browser based access

Jonathan: just to keep it isolation. Yes. So one of the, one of the most hilarious things that we used to do is, so when you're in, is this specific to a hotel, um, you would have your hotel management system and then you'd have your phone system and they would need to be able to talk to each other, but for similar security reasons.

It couldn't be an IP connection, and so you would have two separate pieces of hardware. Each of them were IP to their respective board, and then in between the two, it was Uart serial connections to get these two pieces of hardware to talk to each other for security reasons. Yeah. It, it never ceased to amuse me whenever we set one of those up.

Uh, very, very similar idea. So does, does Pangolin sort of, it's a VPN under the hood, but with all of the, uh, the admin stuff you have on top of it, does it sort of fulfill this need to not be a VPN?

Milo: Yeah. Yeah. So the, the browser based access is what really gets, is really applicable to that industry. It gets them excited.

Jonathan: Mm-hmm.

Milo: Um, because. Again, all these things are, all these industrial systems are just running a, like a web panel. Um, it's just a, a website you go to usually on the local network. Mm-hmm. You go to an IP address, call port, whatever. Right. And you click around

Jonathan: hopefully, hopefully not over the public internet,

Milo: right?

Not well. So

Jonathan: office

Milo: it is.

Jonathan: But

Milo: yeah, they would just open up that port and, and go to the web panel right from anywhere. Um, but with hangin they can just drop in the site connector. Mm-hmm. They can define the resources on, on that network. Um, and then the browser based access uses the outbound tunnel. So there's no client software You even have to run.

Contractors don't have to run, like install some new software on their computer that you then have to manage. They just go to their browser and can access the thing. And then it's also protected with a layer of authentication in front where you could give and grant temporary access as needed. Um, yeah.

Jonathan: You know, I hadn't been thinking of this in, in terms of this, but I could actually really use this. I've got Linux servers that I admin remotely, and I'm, I have a, they held together with duct tape and bubble gum wire guard installed that maybe I should go and just re replace with a self hosted Pangolin.

That would probably work.

Milo: Yeah. Let us know.

Jonathan: Um,

Milo: see proof of concept.

Jonathan: Yeah, absolutely. Oh, that's, that's actually, that's actually a really interesting thought. Uh, one of my to-do lists is to see if I can go find an old wire guard server and prop it back up because it's got keys on it that I otherwise don't have copies of.

If I end up having to just redo that, I will probably take a look at Pangolin and just see if I can do it that way.

Milo: Yeah, do it. Yeah. If you have a mix of, uh, public and private resources, uh, I think it, it could be a good mix. Yeah. Um, absolutely Simple access is, is all or four.

Jonathan: Very cool. Um, alright, it, we're getting close back to the bottom of the hour.

Is there anything we didn't talk about that I didn't ask you about that I should have?

Milo: Oof. Good question. Um,

Jonathan: this was hard to figure out. You gotta do some set math in your head.

Milo: Yeah, yeah, yeah. Um, I think we, we kind of, we kind of covered most, most everything. Um, yeah. So I think, I think that was good.

That's pretty comprehensive.

Jonathan: Okay. I did just, I did just see something I want to ask you about, and this is not necessarily an open source thing, although it kind of is. Um, you guys are part of Y Combinator.

Milo: Oh yeah.

Jonathan: Yes. How, how has that experience And so this is, this is sort of me taking off my floss weekly hat.

Yeah. Putting on my businessman hat. Um, although there's definitely overlap here. How has, how has that been? How has that experience been? Has it worked out fairly well?

Milo: Yes, it has. So, well, and you're reminding me of this. Yes. This whole quote we didn't talk about, like, my co-founder is also my brother, so we can talk about that.

Jonathan: Oh, well, I'm sure that's interesting sometimes.

Milo: Yeah. So, um, yeah, Y Combinator is, is a force like, you know, it's, it's an institution. It's pretty big. Um, it right now, you know, it's, I think we're in a weird spot. We were like the only. non-AI company right in, in our, in our batch. Good,

Jonathan: good for you guys.

Milo: Yeah.

Oh my goodness. We, we, we we're proud of ourselves. Yeah. Yes.

Jonathan: You should be.

Milo: Uh, we were the, we were the, for, for most people, we were the boring company. Um,

Jonathan: I have, I have learned, I I will let you, sorry to cut you off, but man,

Milo: no

Jonathan: worries. I've learned you're doing business, that there is really something to be said for boring.

Yeah. Like, like you, you review a contract and the contract is boring, man, that's a good thing. You have a meeting and the meeting is boring. Like you're doing something right. If your meetings are, are somewhat boring, you, you want the, anyway, that little, little money rant. Go ahead, continue on.

Milo: No, it's true.

It's like all, all the big companies are. Often boring companies, if you like, if you cut out like the, you know, the top three or four, like massive companies right now, or even if you do, like, you pick Nvidia, I mean, the core Nvidia is just building these chips and like selling these chips. It's not exactly all that sexy.

Jonathan: Yeah,

Milo: yeah,

Jonathan: absolutely.

Milo: Uh, but yeah, know the Y Combinator experience was great. We, we actually, um, we did it primarily because we wanted to quit our other jobs, so we were like, okay, um, what are our options? Uh, you know, we could find a new job. Um, we could, we could just quit, you know? Yeah. Uh, we need to pay ourselves some money.

So, um, a quick, a quick thing to do is shoot your shot on Y Combinator and, and they'll, they'll give you some investment to build a business.

Jonathan: Yeah.

Milo: Um, and, and, uh, we. Applied and got in. I think they liked, you know, that we had a ton of adoption. Um, right. Because that's a metric of do you have something that people want, which is their, the whole pitch with Y Combinators, can you build something that people actually want?

Jonathan: Mm-hmm.

Milo: Um, and I think we did that. So, um, that was about, that was last summer. That was summer 2025, actually just had our reunion. Um, they do reunions every now and then with the, the batch mates. So, uh, that was this past weekend actually. So it's funny timing.

Jonathan: Uh, one of the, one of the things that we sometimes see in on Floss Weekly and where I do some other coverage of like the Linux and open source space is that.

Sometimes venture capital is just, uh, toxic to open source projects. Yeah. Um, I guess Y Combinator is technically vc, but you,

Milo: yes.

Jonathan: It seems like you've had a very different experience than some of the other VC horror stories.

Milo: Yeah. No, you have to, you have to pick and choose who you get into bed with, uh, a hundred percent.

Um, you don't want to, uh, it's true. Yeah. You don't want to be dealing with the wrong venture capitalists. They, uh. So Y Combinator is because they're so big, they fund all sorts of stuff. Right. And they're very hands off. Um, they, they funded GitLab, right? Massive open source company. Docker came out of, out of Y Combinator.

Jonathan: Yeah.

Milo: Um, so there's, uh, it can be controversial, but at the end of the day, it comes down to, I think the founders and the, the roots of the founders. Are they gonna stick to, um, the ethos of what, you know, started the company? Uh, I think. GitLab has done that very well, in my opinion. Um, they're a massive co they're a public company, right.

They've taken on tons of venture capital.

Jonathan: Yeah.

Milo: Uh, but they're able to provide both a really good enterprise suite of products that are paid and free open source software that anyone can deploy and self-host. Mm-hmm. So, um, yeah, it's just, I think you just have to, it's like dodging a minefield in venture capital.

You just don't want to, you just don't wanna take money from the wrong person.

Jonathan: Yeah. Do you, do you, do you feel like you guys have, uh, turned the corner and turned it into a profit profitable business?

Milo: Uh, not, not yet. No. I mean, we're, we're still not profitable. We're focused on, um, growth at the moment it's still more adoption.

Uh, but we have, you know, I think the seeds that are, are leading to will be profitable probably sometime next year.

Jonathan: Oh good. Yeah.

Milo: Yeah.

Jonathan: That's, that's excellent. I'm, I am, I'm super glad to hear that

Milo: human. Sorry, hu. Human cost is very expensive, right? You, you have to,

Jonathan: I am aware. Yes. Am

Milo: aware. Right? Yeah. So

Jonathan: you, I I've, I've recently gone full-time with my, I call it a startup.

That's not entirely accurate, but that's the easiest way to describe it. I've recently gone full-time with my startup. Uh, it's, yes, yes,

Milo: yes. No, it's, it's very, very expensive. Especially if you hire more than just the founders. Uh, right. You can pay the founders less. Yes. Because you're, you're taking on the risk.

You're, you, there's equity to be had, but um, when you hire someone, you know, they, they are taking on sometimes even more risk. 'cause like

Jonathan: they're taking on a different type of risk.

Milo: Yeah. Like opportunity, cost risk almost. Right. Exactly. They could be working somewhere else. Um, so that's, yeah. And, and, and part of the goal of raising venture capital is to.

Hire more people up front to, so you can develop software as an investment in, in your later business. Mm-hmm. So what that means is high labor costs now to be able to sell more later. Um, and that's kind of, kind of where we're at the moment. Yeah.

Jonathan: Yeah. Very cool. Alright, uh, I've gotta ask you a couple of final questions before I let you go.

I will get emails complaining if I forget this. What is your personal favorite text editor and scripting language? It's a longstanding tradition of the show. I gotta ask

Milo: te so text editor and scripting language?

Jonathan: Yeah.

Milo: Okay. So Neo Vim

Jonathan: Okay.

Milo: Is the text editor. Yeah. So any of them. Um, I switched to that two or three years ago from VS.

Code. I tried Z for a bit.

Jonathan: Okay.

Milo: Um, have you used Z?

Jonathan: I've not, but I, I'm aware of it. I don't know what it is.

Milo: Okay. Yeah, it's, it's uh, it's like vs code but runs on a game engine kind of, so it runs at high refresh rate. It's a very nerdy thing to like mm-hmm. But um, yeah, I know Neo Vim is my go-to. And then scripting language.

It's hard. It's a hard choice between, I mean. I would say go, probably go is like the best, my favorite overall language. But it's not great for scripting. Like, you know, bash is, is what I think more of a scripting language. Sure. But I, I wouldn't say I, I'm not really super fluent in Bash uh, Python's also a great scripting language.

I'm pretty fluent in that, but I, uh, I like to pick up go where possible.

Jonathan: Sure. Yeah. Makes sense.

Milo: Yeah.

Jonathan: You know, the answer, the answer to that question used to be, you know, either VI or emax. Mm-hmm. And uh, then on the scripts and language side, you know, it was pretty much either Pearl or Bash way back in the day.

And the, this show has roots far enough back in the day that we've gotten a lot of those answers. And one of the things that I find fascinating is to see how people's answers have changed over time. Uh, we'll, we'll even have some people on the show that we've had on years ago, like in some cases, somebody we had on a decade ago and you know, they used to be a big emax fan and now they're like, ah, I pretty much use VS.

Code all the time.

Milo: Yeah.

It's

Jonathan: funny.

Milo: Well, there, it's like, sometimes it's like that curve, right? Where you're like, on one end you're using. Like VS. Code and then you're in the middle and you're trying to like Z and all these different projects and you just go back to VS. Code because it's, it's like the best overall.

Jonathan: It's the one that works.

Milo: Solution least amount of headaches kind of thing.

Jonathan: Yeah, absolutely.

Milo: Yeah,

Jonathan: time. Time is a flat circle man.

Milo: Exactly.

Jonathan: Alright, thank you so much for being here. It has been a blast to, to get to talk with Milo and talk about Pangolin and Faial, the I, ot, ot, and just all kinds of fun stuff.

A lot of stuff from my background and things I'm working on now too. So I thank you very much. I appreciate it.

Milo: Yeah. Thanks for having me.

Jonathan: All right. Uh, we do plan on having a show next week. I don't think we have a guest scheduled yet, although I've got some business cards from Embedded World. I'm gonna start sending emails out to hopefully later today.

So plan on being back next week. If you run a project, you know, somebody that does let us know so that we can give 'em an interview, uh, floss@hackaday.com or come jump into the Discord, let us know, get us in touch. We will get you on the schedule and chat about your project. Um, if you want to find more of me, there is of course, the Untitled Linux Show over at twit and you can come check out what we're doing at Mesh Tastic.

Other than that, thank you. Appreciate everybody that watches and listens whether you get us live or on the download, and we'll be back next week on Floss Weekly.