Transcript
D\&D - 2025/03/14 13:00 EDT – Transcript
Attendees
David Busby, David Busby's Presentation, david raistrick
Transcript
David Busby: Build versus buy in the age of AI. What's up now? You just said you rarely think of buy anymore.
david raistrick: Yeah, for small stuff I don't even go look for software anymore,…
david raistrick: much less go look for buying. And if I need to tie something together, I just get GPT to write the code for me. yeah,…
David Busby:
David Busby: Yeah. there was another thing we wanted to chat about was those integration automation tools things like N8 that is it then that even might turn on a little screen share on this one as we jibber jabber through the stuff…
david raistrick: I think that was we have that on our list.
David Busby:
David Busby: if is still around and you can hook all kinds of your other stuff, So, does GPT know how to integrate if notice…
David Busby:
david raistrick: Maybe probably it knows a lot of things.
david raistrick: I mean, it knows how the messages posted on the Samba are formatted without having to go look at them. It's not perfect on the Samba,…
David Busby: how the WS are wed?
david raistrick: which is a VW forum. It's been around for a long time. it spun out from one of the communities I run and…
David Busby: Why would you do that?
david raistrick: and I was like I need to take this 147 page thread and turn it into something that I can feed back into chat GPT to talk about, right?
david raistrick: So, I just had to write a tool to do that because I can't
David Busby: I mean, when I talk about build versus buy kind of stuff, I sort of think external services that you can do yourself now, GPT with lots of assistance if you're moderately good at stuff, a complicated one I've heard a rumor about, but I don't have the skill set necessary to, judge if it's working. But somebody, an S sur that I know is using the Kubernetes mechanisms in a cloud agnostic way and he's been using GPT assistance to rig his s*** together, let's say.
David Busby: And now instead of it being I want my beefy PostgreSQL database with replication in multiple availability zones that's now some magical again because it's out of my depth of knowledge some magical orchestration with his Kubernetes mechanism that is like yes we felt like that one was getting old so now we're going to upgrade this rolling PG version update and…
david raistrick: So I mean that's a good example.
David Busby: it's magically, through the scripts and the, magical feature sets that cube gives you let's just run a little test over here.
david raistrick: In systems and devops and we often think about builders by when we talk about logging, we talk about monitoring systems, we talk about database management. So do we use RDS or some other cloud database solution? Do we handroll our own database solution with replication and failover and backups and restores? Because a database does not exist if it doesn't have backups and restores, right? It's not a database otherwise.
david raistrick: So RDS integrates for example, RDS, the one I'm most familiar with, integrates all of those features. you have ways to scale it, ways to manage upgrades, you ways to manage backups, ways to manage restores. Somebody has built all the code around all of this and tested it extensively or you can build it yourself and learn how all of those systems tie together, which was how we did it before RDS. So we had to be experts in all those. And now the new model is this, right? So we don't have to learn it ourselves.
david raistrick: We don't have to actually learn all of the pieces. We can just have GPT tie all the pieces together and write us some code.
David Busby: Yeah, I suppose in this example J J J J J J J J J J J J J,…
David Busby: we'll just use the letter J to talk about this a. in Jay's circumstance, which now it sounds like it's his proper name, but it's not just the letter Agent J, knows that you need to do replication, knows knows that you need to do the thing, ready to go onto the system and manually put PG in there, and then get it rigged together and edit the necessary configs, read all the documentation, do all the thing. it's please give me the recipe for such and such.
00:05:00
David Busby: And I'm going to try that next. since I don't even know anything about when I say I don't know anything about Kubernetes, I mean I know generally what it does. And then maybe this would be another spot we could do a little experiment there. read the transcript and analyze with Open AI. then add a entry to a spread,…
David Busby: a Google spread sheet. It's harder to type when people are looking at me, even if it's just on the internet.
david raistrick: I know,…
david raistrick: right? Yeah,…
David Busby: Use the Zoom recording so it does know how to do this thing. Save your recording. Send yourself an email. Little bit of Python. I wonder this thing almost always defaults to making code for me in Python.
david raistrick: It is best at writing Python. Everything else it's worse at. that basically because it has a large large corpus of Python to work from. it can do other things and some things it can't do. Some things it will just make up. But Python is where it is kind of its expert zone.
david raistrick: And it has the ability to do pi to execute Python live inside an isolated sandbox environment. So it can run Python and produce images, produce results as long as those results don't require reaching out to third party services.
David Busby: Really?
David Busby: That's neat. So, that was cool. It does know how to do if I should have asked it how to do something else.
David Busby: what they call it DC2 and…
David Busby: what is called their VM thing. this is as far as we're gonna go.
david raistrick: Yeah, I don't know…
david raistrick: what Google calls it. I'm sure it wouldn't know what you're talking but
David Busby: Let's see how smart this thing is. All of this seems to make sense. So, how do I automatically make my PGs config know about each other? Put wire guard between them.
david raistrick: Still puts you way deep into debug land when you start implementing the solutions though, right?
David Busby: I mean, this is the kind of stuff you end up doing manually if you're not using this magic. You like a YAML example of this? baby. Yeah. Send me the YAML.
david raistrick: Right up…
David Busby: Apparently you just take the debug spew and…
David Busby: feed it back into here and say, " it didn't work for me and here's what I'm seeing.
david raistrick: until you get into the circular you hit it's exhausted all of its ideas and so the next idea is the first idea and…
David Busby: No, you run all this stuff in a testing environment.
david raistrick: then it just keeps working through it and if you're not paying attention and I am guilty of this extremely, right? I'll just keep feeding it now like okay feed it because I'm not looking at the results I'm trying the results right in the ecosystem in the appropriate place where it's safe to do this right I wouldn't do this against production right in a testing ecosystem that I can absolutely completely destroy then and sometimes because I'm doing this while I'm doing five or six other things in between conversations whatever and so I'll just take that output feed it back and I don't recognize until three or four times
david raistrick: later that are the same changes it made last time and so you end up just doing the same thing until you recognize all right now we've got to take a different approach. Let's inject some logging into this. Let's do something. Let's figure, let's isolate this down to a smaller problem set, whatever it is, and start working through it. My favorite tool is the one that I built to take Terraform's spew and feed it through just pipe it directly through GPT and have it summarize it all and tell me what went wrong and what went right and count up the number of resources that changed as a very simple summary instead of me having to go read and look at all the spew and then if what it says is hey there's a problem then I can go look at it but it's
00:10:00
david raistrick: explain why there was a problem, too. manual setup. Can't we automate this?
David Busby: So here I did talk to me like I'm an idiot.
david raistrick: It's sometimes good to revisit your assumptions.
David Busby: How do I do that step by step? I don't know anything about how to do this. I mean, I do know how to do this, but this is now something that would be days of possibly documentation reading and is only in seconds. And what I'm replacing is, an RDS cost that is like 4x, I mean, this is what agent J has told me is that this gives them the same, level of assurances that their stuff is going to stay on because EC2 and these cloud instances are very, very reliable and they're obviously more complicated.
David Busby: in this example, this instance is having a problem. Let's magically take it out of the loop and…
David Busby: tell everybody to go over here with a whole bunch of other orchestration s*** that's involved in there, right? …
david raistrick: It's interesting to hear that right…
david raistrick: because 10 years ago we would have always considered that EC2 instances are unstable and…
David Busby: I mean,…
david raistrick: but it's true you can have four or five years uptime on an EC2 instance now it's possible Exactly.
David Busby: yeah, I don't mean to say that They're the same level almost I think as if you managed a big ass data center with hell of servers in it. s*** crashes, drives fail time.
david raistrick: Where it used to be expecting to lose an instance in…
David Busby: Right.
david raistrick: if you had 10,000 instances, you would expect to lose an instance every couple of hours, And you don't have that expectation now, especially not if you're only running like five or 50.
David Busby: Or you've got this mechanism that's just like I have a thing here. you can if you were so paranoid about things like this,…
David Busby: you could auto rotate your nodes in and out of this grouping with the way that the replicas can turn themselves on magically.
david raistrick: And that's definitely in the buy versus build,…
David Busby: It's less auto scaling and more of a auto rotate.
david raistrick: Because we would build in the data centers because we could build reliable that we would get a couple years of uptime other than the things that screwed up, but we could expect some consistency around it. we would break it or after 3 four years you would start to see real breakage…
david raistrick: but now EC2 instances are reliable enough that we can just consider them to be in that same category of reliability. Yeah.
David Busby: close enough.
David Busby: Yeah. Not 50%, it's 90%.
david raistrick: Yeah. It's just wire guard with a bunch of automation and…
David Busby: And if you have your infrastructure rigged up with the necessary replications and the magic wire guard VPN, which is tough to configure initially, you could just buy tail scale, and with some finger wagging. Tail scale is just wire.
david raistrick: UI around it. Yep.
David Busby: Yeah. That just is doing a lot of work in there. But you can then at least from my understanding none of it is outside of…
00:15:00
David Busby: what you can now figure out how to rig up yourself and then crap you have to maintain it but crap you have to maintain everything sort of anyway…
david raistrick: Everything you buy or…
david raistrick: build, you have to maintain. That's very true. Right. And the 300% problem kicks in here a lot because the 300% problem.
David Busby: what problem 300%.
david raistrick: So every single thing you use adds 300% of overhead of how much knowledge you have to have.
David Busby: How do you get to 300
david raistrick: There's a really good article about it and I don't have it in front of me.
david raistrick: But that idea is that so if you want to use EC2 if you wanted to okay let's start at I'm going to run Linux on a machine under my desk right I need to understand the machine I need to understand Linux now I want to run that same machine on an EC2 instance now I need to understand EC2 the machine and…
David Busby: …
david raistrick: Linux But EC2 brings in a bunch more problems because do I understand everything starts to escalate.
David Busby: you have to know how to use the Amazon console and click through there. And then you'll want to make sure how to pick the right server size. You have to do something with a security group.
david raistrick: I have to understand VPC networking, all and understand how to remotely access it, how to someone else from remotely accessing it because after all I'm just replacing this tool that was sitting under my desk and…
david raistrick: I already knew how to manage and control access to it, And so every single thing if we say okay now we're going to adopt the database. we could install the database on the EC2 instance. I have to understand how the database works enough to get it running enough to manage backups enough to do all those things or I use RDS where I have to understand less of how it runs but I still need to but now I have to learn a different service how to use its UIs how to use its APIs how its configuration mechanisms work right I forget the Postgress version of it but I don't create just a mysql
David Busby: All right.
david raistrick: my SQL postgress config file and put it on the machine, I have to run it through UIs or APIs that align specifically to how RDS manages it. And so the amount of knowledge we pick up then we're going to use Kubernetes on top of EC2. we've got to understand that. We've got to understand Kubernetes. We've got to understand using Postgress inside Kubernetes. We got to understand Docker, And we just escalate and all we were trying to do was run some JAR file,…
david raistrick: right? What were we doing again?
David Busby: just running some jar file.
David Busby: This one says that cube admin does manual setup full control though which is almost always the part I bump into ties into that other thing we had on our topic list about vendor driven churn being things like they bumped your version of you or whatever it is you need to support and so now without a lot of wiggle room you've got to move into the next piece there in an unexpected time I think one of the benefits on if you're building your own crap and you're working at
David Busby: the lower level piece, you don't have a framework that moves a whole lot out from under you. the most recent one I have to do some finger wagging with was a project I observed that was making an attempt to act. but early in the React and there was not really a super hard demand for this multi-developer kinds of feature sets and those benefits that React made promises about. you don't need that on a small team and you don't have this fancy super awesome real time dynamic something or other that has this complicated even page state to maintain. never minding the fact that the back end of the app is like some legacy
David Busby: two different implementations of some legacy stuff. One of which is still like a Pearl script, which I love. Pearl C, yeah, It's still getting the job done, but you've decided to adopt React because it was cool. and then you ran into some little issue that everybody was running into because you were so early and now you had to go from 13 to 15 to 17 and there were breaking changes so you end up rewriting a bunch of stuff.
00:20:00
David Busby: It's sort of easy to pick on that example because sort of anybody adopting something early runs into that that's not new and fancy but even if you pick something that's more mature another group I was advising had a dev team that already had loads of PHP experience. So they're going to build a new app for this dude prototyping some stuff out. They decide hey we're going to do laral Marvel 10's great. Marvel 11 is now released, and I'm doing the finger wagging saying, "Hey, hey, hey, hey,…
david raistrick: It's balancing that investment,
David Busby: just stick on 10." it has this terrible gaping security hole, knock on wood, that we know about that, there's not a security patch requirement on this one. And yes, it's better for reasons, but those are all hard to measure. And you can keep going on 10 right now and then maybe just land at 12 or something, right?
david raistrick: Right? Because every piece of software you pick is you have a longtail investment in that goes into managing it and everything else. but what are we trying to do right now? If we're trying to get a product out the door, if we're trying to figure out, do we even have product market fit? Maybe We don't need to worry about that because let's do that when we might throw this out next week.
David Busby: Listen, let's just call it tech debt and…
David Busby: move Just call it tech debt and…
david raistrick: Right. Yep.
david raistrick: Exactly. the thing I always tell companies is that on day zero,…
David Busby: move on. I read something funny on the internet. The new cool name for tech debt is you just call it founder code. And okay, it's the early team who was building this stuff.
david raistrick: you have no tech debt. On day one, 100% of what you wrote is tech debt.
david raistrick: on day two, 100% of what you wrote is It's all tech debt. If you're building it, you're creating tech debt. The question is when do you reinvest in that? And when you reinvest in that should be somewhere aligned with when did we start making revenue because our cost of doing business goes up when we start making revenue because we can spend money on places and we should spend money on places. you're muted.
david raistrick: No, you're muted.
David Busby: I am. Yeah. so this is that article you're talking about, the 300% production problem,…
david raistrick: And every one of those things is scaling infinitely.
David Busby: 200% now it's 300%. Because the deployment target and the deployment methodology. So I do I mean winding all of this back and talking about tech debt too. There is a significant number of operations that never ever ever have to worry about some of these massive scaling problems that a lot of this infrastructure was designed to maintain.
David Busby: So winding this back to some other little startupy ideas that you can run little web apps and things like this and you want to avoid the 300% problem of learning tool A B C D E again with it increases your burden at some geometric rate you can still find the stuff that you could deploy all of your stuff on a single node and then run forever and then for any of the build versus buy stuff, you always just build your own. I know another agent is ever so firmly in the camp of build your own, never move to cloud.
David Busby: I'm going to just not build on my own data center,…
David Busby: but I'm buying a rack in a good data center and I'm filling my gear in there. And I'm buying a rack in another data center and I'm filling my gear in there and I'm doing my stuff automated deployments, but it's not Kubernetes or Terraform or Magic Sauce at this point. it started off 20some years ago with here's…
00:25:00
david raistrick: But what's his…
David Busby: how we would do these automated deployments now like inch forward year by year.
david raistrick: but What's his business plan? How does that align with His business plan should have some expectations of growth in it, So he can define upfront…
David Busby: Yeah. they process information in the finance space. there is not any kind of exponential growth that happens in that space,…
david raistrick: how big his operation needs to be.
david raistrick: He's not going to pick up new customers every day, Where else is scaling the inflection point? Yep. So,…
David Busby: right? Yes.
david raistrick: he can say, "Okay, we start out here. Tomorrow, we pick up another customer. It adds a little bit to us. We're going to pick up a customer a year.
David Busby: when they scale they buy some beefy hardware that they're pro overprovisioning. They say everybody says that and it's a reasonably good server that you can purchase couple thousands of dollars put this in there you've got the tools already…
David Busby: because they started the investment on this infrastructure 20 years ago again opted not to I'll just buy a managed hosting provider yeah piece by piece I mean it started off like we got a new server.
david raistrick: They've already designed and…
david raistrick: built a framework for managing these things. Yeah. Yep.
David Busby: The process is ship it to our facility here. We put it on our test network. We get it all dialed Ship it to the data center. because you don't go there to do that because the data center is not in your town, you're in Seattle and the data center is somewhere in eastern Washington and then your other data center is like Chicago or Denver or something and then you would provision the thing and ship it over there and then they plug it in and they turn it on and you're like okay cool maybe it's connected to your IPMI kajjigger when you buy this rack or half rack or something you have to buy the full one if you want a cage around it and you want a cage around it because you're doing finance crap
David Busby: app. you got to put your own router at the top. You got to put your own, local switch, whatever quality you want to put at the top Put your own little backend network and all of that stuff you're doing. or putting that hardware in. And some of the data centers will have somebody who's like,…
david raistrick: Right now,…
David Busby: I'm familiar with I Cisco IOS. I'll provision your router for you or
david raistrick: let's take that team in and put them in a new context. We take that same team out of the business that they're in. The business folded for some reason. It's done. Right? They've lost all their IP.
david raistrick: They start a new business. How many years is it going to take for them to rebuild all of that? They don't have any of the code. And that's…
David Busby: I think that's…
David Busby: where you would just like, you want to yeah,…
david raistrick: why we start today with buy and we just use cloud because we can get that started in five minutes instead of buy hardware wait a couple of weeks ship it provision it get an remotely get routers installed do all that stuff and…
David Busby: that would
david raistrick: that's because 20 years ago we couldn't and so that was the default cost of doing business.
david raistrick: we don't have that right even then we would start by people would build things on the local and get it running get and get some things going right while we were building out the infrastructure for it but we don't need that baseline infrastructure we don't need to start we need to buy five pieces of hardware we can just throw it on one EC2 instance and see what happens
David Busby: I mean there's a lot of folks that I mean we're saying build versus buy and how does that and how does AI maybe help mo move that s towards a little bit of build your own for a bunch of the stuff where maybe the baseline of…
David Busby: what you have to build is definitely moved up. Don't buy hardware in a rack. just buy generic Linux server nodes in whatever provider you feel Scatter them across Oracle and Google and Amazon and…
david raistrick: And …
David Busby: Aubai. Right.
david raistrick: 20 years ago, our developers are sitting together in a room, hashing out the ideas, figuring out what they're going to do. They're frameworking and wireframing and doing all the things to come up with the basic scaffolding of the application they're going to start building, right? And then figuring out how to solve the business problem they're trying to solve with that software today. And so during that time, our operations guys can start ordering hardware, figuring out what the infrastructure is going to look like, where our scaling points might be, right? Buying hardware based on best estimates of what that's going to be.
00:30:00
david raistrick: But today all of that has been compressed because we have tools that manage that. We have distributed teams that are working in many different places using different tools and then now that was five years ago right now those teams are one person using AI to generate their wireframes to generate their mockups come up with and then generate code from those mockups. And so by day two we're expected to have something for them to run this on. It's not longer three months or even six weeks. It's two days. But they can do it themselves.
david raistrick: They don't need us to do it because that same tooling can give them just enough to get that product out to market without having to worry about. And then if they're doing it in cloud, then at least they have the expectation that they can bring somebody in behind them to help them figure out and fix the thing that they built that shouldn't have ever existed.
David Busby: Yeah, I don't know if there's a lot I've heard more than one horror story of a poorly implemented for lack of better terminology or poorly implemented Amazon setup that maybe is costly because you're have too much in the Amazon space and so you're just paying for the service itself too much
David Busby: but also it's very messy the way that it's rigged together because we had folks that decided I don't need to hire a consultant to help me design my stuff and I can figure this out with AI but I'm working my way through the 300% problem here and I thought this was the right way to do
David Busby: the private networking between my nodes in Amazon and my RDS and my elastic cache and something and my something and then I thought, I had to do this with S3 and then I decided to do this other weird thing with it and…
David Busby: and then you have somebody else slide in later who's just an expert on Amazon and AWS infrastructure looks at your stuff and it's like what a mess here. Now we have
david raistrick: It was the same in data centers,…
david raistrick: I mean, we'd come in and follow up behind what a team had built and you're really trying to not be accusatory and be like, "What is this pile of s*** you built because we need to fix this now?" Because usually by the time they would bring someone in, it's already on fire, right? They've lit servers on fire. It's just exploding. And then they're like, "Let's hire someone and make it their responsibility." and I hate to say it's still actually the right answer until you can get generate the revenue, it's not worth making that early investment.
david raistrick: If you can start a business where you already start with revenue, Let's start the investment early. But yeah.
David Busby: That's the way to do it.
David Busby: Just get money.
david raistrick: Start with a business that already has customers and already has money coming in. Then we can do it the right way from the ground up. Otherwise, building the pile of s* first, getting customers to spend money with you that but know this upfront. Know you're doing this though, you're making that build versus buy decision because you could buy data dog but you've decided to implement your own API and you're stacking some or you're all what API APM so you've got some Loki and…
David Busby: on there, too.
david raistrick: some Prometheus and some this and that and I don't even know what the open source you're doing open telemetry based stuff graphana and you're scaling all these little things but they're only having to scale a little bit you only had to build them once
david raistrick: really know how they all work, but you did get them working together. Then you start making money and you have to realize that the time invested in maintaining that now because you probably did one of the classic interesting decisions of taking some metrics that you were storing in graphana that you were using internally and…
David Busby: You take a step back.
david raistrick: exposing those metrics maybe even the graph to your users which meant your entire graphana stack became customerf facing production and you're like, " how do we scale this?" You don't. That wasn't what it was built for. But yeah, you have to take that step back. So recognize it when you're building the business, recognize when you're making the early build decisions that you may need to revisit them constantly. Re same thing when you make a buy decision.
00:35:00
david raistrick: you say, "We're going to jump in and we're going to pay Gitlab because GitLab's hosted service gives us the thing and we're doing it at their free tier, so we didn't have to pay it for anything." And then you turn around and you've cross that threshold and now it's $19 a user a month and you're like, " that's a lot of money. We could host our own GitLab, so maybe you pull it in house." And then you realize that getting people to manage that stack is expensive. Maybe we should go to GitHub where we can spend half as much money and…
david raistrick: get the same thing, but it's still all transition work, right? It's not terribly difficult at a certain scale.
David Busby: I would say that hosting your own GitLab is not terribly difficult though.
David Busby: One of those roll your own thing. Yeah. So if you're under …
david raistrick: I mean, you remember the story of Mercedes, right?
david raistrick: So, Mercedes rolled out GitLab internally and they just used default configs which meant that all of the repos were public. They thought it was private because it was inside their own GitLab. nobody looked at the configs and then they accused GitLabs and said that it's GitLab's fault that all their data got exploited and their code got shared.
David Busby: sort of buried maybe.
david raistrick: I'm pretty sure it was Mercedes and they accused GitLab of and…
david raistrick: said it was their fault and their default configs need to change, but they picked up a product, hosted it themselves, and didn't understand that this was a product built for public and private repositories. If you wanted to make everything private, you needed to check config.
David Busby: Yeah.
David Busby: So that went down to 2020.
David Busby: Pulled down there, put the lead on the article says misconfigured GitLab some access control something or…
david raistrick: It was default config.
david raistrick: Yeah, It was 100%. It was just misconfigured.
David Busby: something interesting.
David Busby: I mean there you go. That's part of your 300% Oh no.
david raistrick: You see you pick up you pick it.
David Busby: Oops. I mean,…
david raistrick: if they'd used the paid hosted GitLab, then someone would have already made those decisions and all you did you had a obvious distinction between public and private you might hope so.
David Busby: at a company that size, if you're going to be doing a GitLab deployment or something and you're going to be paying them and you have how many users that need access to the GitLab instance? Let's pick an easy number. 2,00. I feel like you might even have access to tools and feature sets from their support team. direct phone number to somebody.
david raistrick: But then again, going through acquisitions and getting a product purchased and all of that is so much harder than spinning it up in your own data center and without support,…
david raistrick: without anything. Right. Just we can turn it on.
David Busby: Just run the Docker thing,…
David Busby: man, and put traffic in front of it.
david raistrick: And then you're like, we didn't even think about p where we were going to store persistent data. I still love people that run databases in Docker and haven't thought about the part of a…
David Busby: Do the volume.
david raistrick: where the data is d, Running databases inside Kubernetes is a perfect example.
david raistrick: They don't think about where that data is stored. it's just in the cluster and then next thing you know, something happens and they've lost their database.
David Busby: start on the cluster.
david raistrick: You got to understand how it all works. Please
David Busby: You had to put in your …
David Busby: we didn't get the chat robot about it, but we had that conversation with it about this thing Kubernetes and something
David Busby: Keep the backups on a rsync.net. Why not? I love rsync.net. those folks have some fun technical blog posts just like Yeah.
00:40:00
David Busby: I wonder…
David Busby: why it says AWSS3 is recommended.
david raistrick: because it has the most buckets with that with most unencrypted publicly accessible buckets of your data.
david raistrick: They just change it so that I think S3 finally changed it so the default mode for a bucket is not public.
david raistrick: It's not private. It was relatively recent and…
David Busby: I think that was ages ago,…
David Busby: Make some crown job.
david raistrick: there's still tons and tons and tons of data out there that's public that should have been private.
David Busby: This is all the stuff you want to do,…
David Busby: I do want to know why it recommended AWS, but then give me the step by step so I can test all of my own magic cube deployment now so that I can replace checking the box for AWS RDS multi-AZ, change taking my database cost from, $12,000 down to $6,000.
david raistrick: And how many man hours do you spend implementing it and…
david raistrick: maintaining the solution?
David Busby: Far I've spent agent Jay already knew a lot of the stuff,…
david raistrick: Yeah. Yep.
David Busby: right? So now it was just have me go faster on my build stuff. I know how to evaluate a buy decision. We've already made that. We are trying to figure out how to use more lower level optimization.
david raistrick: Absolutely.
David Busby: One of the objectives was not having the So RDS is objectively a vendor lock in you chosen the Google one or the Oracle one or whoever's, right? So they want to be able to move around not because they're sensitive to price shopping or any of that other stuff. It's just a resiliency choice they want to make. And so moving out of we have to use Elasticash. No, no, no.
David Busby: Now we just spin up a node and…
David Busby: that's our Reddus implementation node and our whole little infrastructure knows Move it around, destroy it when we need to. Let's just make another one. All of you guys use the new one. Everybody kind of stop using that one and then we'll feel like it's empty and then we'll just …
david raistrick: and…
david raistrick: usually when that happens of course is when you discover how many people use their cache as a database. Yeah, of course.
David Busby: that is a different problem. And whether or not you are outsourcing some of that management to Elasticash in AWS or doing your own Reddus nodes that you move around, misusing air quotes misusing the Reddus is a different problem. And also you can why don't you just have your Kubernetes add another Kubernetes cron job that backs up your Reddus or something like that.
David Busby: It seems to have done pretty okay for this one here. Where's the question I asked already? why highly available, scalable, cheap. That said, Cloudfare R2 is a great lowcost alternative and rsync.net is better for privacy focused off-site backups. to each here's how to do your own local cube cluster with kind PG deployment backup and verification scripts. This is some boring s*** to install the thing. Here's the larger deployment. Probably want to set those passwords somewhere.
00:45:00
David Busby: Cube apply. Do the thing. And then what is this one? that's man.
david raistrick: …
david raistrick: there's another part of this and that's So, if we wanted to do this once, we wanted to do this once, then we can invest and do all this work, but what if we needed to do it twice or five times or we decided to silo every single customer and so how many times do we have to recreate this and build this? How much do we have to invest in making this automated so we get consistent process or do we just go with manual and hope we get consistency through manual process?
David Busby: In Agent J's circumstance and they were building this stuff out, I guess they had to do this.
David Busby: They didn't end up having to do another problem, but they did move things in a more like gradual way. But
david raistrick: Right? They built one place,…
david raistrick: then they started to replace that tooling,…
david raistrick: investing in the slow process of making it something they could maintain and migrating to it.
David Busby: Correct. But also,…
David Busby: unfair advantage is that agent J is not a startup founder CTO who's trying to do eight different things that are eight different roles. He's already at a larger organization. He's on the S sur team. He's on this.
David Busby: He's already spent loads of times in the AWS console, writing auto other management scripts with shells that run on your own infrastructure that monitors stuff, various YAML from various tools, the Terraform thing.
david raistrick: He expects his day job is to manage and…
David Busby: So when he's looking at, I understand what is going on in this log. I'm not trying this from scratch with no knowledge and then running into a confounding arcane error message in a completely new domain. that's yeah.
david raistrick: maintain this thing. It's not a ancillary part of his job. It's the thing. It is his job. And so he's like, "This This is my career. I'm going to spend the next five years working on this."
david raistrick: maintaining managing it. eventually this knowledge will get transferred to more of More of the team will take ownership of this and we as a team will manage and…
david raistrick: own this for the life of the company or the life of this particular piece of the product.
David Busby: Yeah, that's pretty much it.
David Busby: I mean, that's mostly it.
david raistrick: Yeah. I don't know.
David Busby: If you're the small startup, right? Build yourself use ExpressJS or whatever framework you think is cool, Laravel or Django or Express or is Express the Node one of a Go one? I don't know. But pick one of those things for your favorite stack, ship it on a single EC2 node and at your first scaling problem just double the size of the node and it's all running on that one box and that box is persistent. So you can solve your backup and store my database problem later. you can run your Reddus and your DB and…
David Busby: your app all on the one box probably for your first thousand clients depending on what you're doing, the cannabis software that I do it's very very niche …
david raistrick: Yeah. How…
david raistrick: how you scale with clients. Yeah.
David Busby: but even the data sets from those largest clients is not terabytes and terabytes of data I think there's some nervousness somebody else I'm chatting with right now there's some nervousness about what if we need to get big all of a sudden, which is such not the problem that you're really going to have, and the way that get big all of a sudden thing happens is it's an exponential kind of thing when it occurs.
david raistrick: Yep.
David Busby: So you have to get big all of a sudden very very slowly and then sort of all at once, and now you're already focused on what am I going to do when I have a 100,000 users. Hey, why don't you worry about getting a 100 users or 1,000 users or something along those lines. Anyway, all the way back one more point for just build it yourself because you can end up saving loads of money. I know we had talked about some other things you could do that with.
00:50:00
David Busby: and possibly avoid some vendor driven h But I think maybe in 200 ages ago, definitely by 2013, my phone system, I plugged it into Twilio and I had had another small business phone provider that was costing me some number of dollars. and I'm pretty good with writing code and I have some experience with telefan in the asterisk and free switch kind of space and I've coded systems against that. Then I was like, I'm going to move my business phone. I'll move it over to Twilio because now it only cost me a dollar a month for the phone number, which I think it might be a$120 now. I don't know. Trivial is my point.
David Busby: because the other phone service was like we'll do your thing with an IVR with some open hours with some stuff some feature sets hunt group blah blah blah and it's minimum 40 a month or something I'll move over to Twilio again with my unfair advantage of prior experience with telefanany and making web apps to making a system that can store a little bit of state and respond with XML come on dude I'm like no AI in 2013 13 for this good documentation from Twilio and a little bit of a test playground on a Called that phone number. I got a service running on, my site. It resps, I just threw some PHP still running Apache prefork on that old box and bar and some PHP endpoints for Twilio to hit.
David Busby: logs my status, gets my transcript, shoots me an email from messages, keeps my open hours, and that was code. We opted to move away from just build your own solution and also the scalability of it is not really my problem. I didn't build my own telefan infrastructure. I'm using Twilio, but I built my own IVR, which Twilio made very easy, and this and…
David Busby: this. And now on months that I have low traffic on the phone, you still pay per minute, but every single month has been way less.
david raistrick: …
david raistrick: build versus buy isn't an exclusive decision. You can build and buy at the same time.
David Busby: You're always doing that anyway.
david raistrick: And that's a good example, That I'm going to buy from Twilio and build using their integration support to suit my specific needs. because you could have gotten the telephone line plugged it into asterisk, done all the things, but you didn't want to do that. that's too far. Right. Right.
David Busby: I'd done that before for, in a different role. you got to be ready to do a lots. You want to talk about plus 300% knowledge. I feel like some of that telefan stuff at that level would be plus 500% knowledge.
David Busby: And I'm not ready for that.
david raistrick: It really is right.
David Busby: I think that settles it. Case by case basis every single time. No easy answers.
david raistrick: It's your team. Is your Is your people's time free? if it is, because sometimes it is or that's because, for all the reasons, then invest that time. If it's not, then maybe it's the other or maybe it's a combination. Nope. Go.
David Busby: Until next time. that stops the recording. I think the transcript is still going though. I can't tell. That worked out. cool.
david raistrick: Yeah. Yeah, it does say this call is being transcribed at the top.
david raistrick: So, yep.
David Busby: So, then the thing that was next on the list is probably vendor driven churn. That's this thing that annoys me where just "f***, I have to upgrade my Iuntu again. f***, I have to upgrade something or other again." somebody called me up and they're like, "Hey, remember that app you built for us 12 years ago?" No, of course anyways, we need to upgrade the MySQL on it. And so it's like MySQL 5. Can you help us out? My SQL 5. Yeah.
David Busby: I mean, sure I can. And that's very likely going to be a pretty trivial upgrade.
david raistrick: There's some tricks in what happens with eight though. So it works. Absolutely.
David Busby: I mean, I don't know. You can bag on my SQL all you want, but it works. So, I don't know that that's the hard part,…
00:55:00
David Busby: but there's some front end on the stuff that's also built in an old way. and the way that old thing is built, it's very likely stuck with an old MySQL driver.
david raistrick: right? Yep.
David Busby: And so I have no idea what's going to happen there. So you're saying, "And now, we want to upgrade this. So we have to upgrade this. So now we have to upgrade this." That's my rant on vendor driven churn. So maybe something to think about for Monday, St. Padre's startup, annoying UI, Pomo, all the browsers. Do you know about Linux server IO?
david raistrick: I think we mentioned it the other day and I probably have an open tab and that's as far as I
David Busby: I mean, I would basically run an advertisement for Linux Server I mean, they wouldn't even have to pay me. It's so cool. It's like those guys are doing some that crew is doing some really good work,…
David Busby: Anyways, I got to go figure out what I'm going to do about my pie pi related emergency some c I am curious like it was and…
david raistrick: One lemon,…
david raistrick: one lime. There you go. I think the lemon idea will work out. You still got the lime pie to eat.
David Busby: I'm happy that this experiment I wanted to try sort of fell in my lap and hey just do a halfass one and see if it's good or not and if it's terrible guess what it's terri Terrible.
david raistrick: Still got There you go.
David Busby: Still got I'll have two lime pies. one small lemon. I'll let you know how it turns out. Happy weekend.
david raistrick: right. Bye.
Meeting ended after 00:56:58 👋
This editable transcript was computer generated and might contain errors. People can also change the text after it was created.