Transcript
D\&D - 2025/03/19 12:59 EDT – Transcript
Attendees
David Busby, David Busby's Presentation, david raistrick
Transcript
David Busby: B Okay, cool. I guess that's live now. so it was good. Last time actually the caddy intro was, thanks for that because I went through on one of these other projects that I'm working on and it's using either Apache or…
david raistrick: Right.
David Busby: EngineX as the case may be, right? And some of their Apache is still legacy legacy, Apache pre-fork with mod PHP, which is bulletproof, right? But, you're not going to get to 600 requests a second on something like that. there's only so much you can do when you throw more hardware at it. You still got all these processes. And then we'd have to still have some engine X in the equation because some of the stuff on the front end wants to make some websocket connects to the back end and prefork is not really compatible, right?
David Busby: So we ended up with this I would call it a tangled mess of engine X doing the statics that it has doing reverse proxy to some websocket provided services engine X doing some stuff itself with PHP's FPM mechanism CGI and…
david raistrick: The way to make it simple is by choosing all of them.
David Busby: then some stuff getting routed to you So, the backend Apache prefork PHP services. and there's another thing in the backend on this one that's a Django API. But, why choose a simple tech stack when you can just smash a bunch of stuff together and put Engine X in front so folks on the outside don't ever really know the difference. I mean, I guess but the caddy in the demo in the test beta, workshop environment in the lab environment, we were able to throw this.
David Busby: It's one config now for Caddy and it does the statics just like we want and it has the multiple site handling and it's routing the stuff we have it rigged so that it's routing to the backend reverse proxying into the Django mechanism and the Apache prefork. But the thing that we are planning on being able we think we're going to be able to move forward is actually that Apache prefork stuff can move into being FPM based PHP thing. There's a little bit of a we need to figure out how to run different instances of FPM…
david raistrick: Right. Yeah. And
David Busby: because one is this other version, right? So we still have this requirement this one is on 7.4 it is not ready for PHP8 this one is PHP8 it's ready to rock and roll this is just you build the thing and then it gets old right and interestingly enough the legacy is not really the right word but the older app that exists on PHP7.4 before is a good thing to highlight. today's topic, driven churn slash the 300% problem. I have that blog post up here and I'll just sort of I guess remember to put a link to it in the notes. Ivr Lee Briggs from some guy from the UK briefly. He talks about this thing, the 300% problem. You can't just learn one thing and then be done with it.
David Busby: We can't just deploy to AWS. You can't just upgrade from PHP 7.4 to 8.2 without some other unknown at the moment, thing cropping up is kind of…
David Busby: what happens and it becomes this problem multiplier.
david raistrick: Yeah. Right.
David Busby: I had been calling this kind of mechanism in my own head vendor driven churn. I want to just make an up quick update to my DNS entry, but I sign into my AWS console and my UI is different because I don't sign in every day, right? And then I have to navigate over to this thing and now the UI is slightly different and where do I go to do this? I mean, I know DNS. Why can't you guys just hand me the text file and I'll edit it and…
david raistrick: What's wrong with the bind zone file? Right.
David Busby: And then I upload it and you just pre-flight check the file to make sure that it's not ridiculous and then I'm deployed, and that worked 20 years ago and now I can't just edit a file and save and reload. I'm clicking through a dozen different things. One of the frustrating times of this vendor driven churn b****, I had to sign into my AWS to do something
00:05:00
David Busby: but you've just enabled MFA and now you don't have your thing and you don't have your key and is your key on this device. Do you have your authenticated code on your mobile phone that's with you because you're on your laptop and you're away but you saved your code on your Prime Workbox because reasons and I just wanted to get into Amazon to do something. It obligated me to do the F MFA. I set up the MFA so I could get in this time. I saved my stuff here, but I was done with my task of updating the DNS and I had this new task of grab the TOTP codes and put them into your authenticator on your mobile device, put them on your second box, put them in your normal archive. Okay, Made it into my primary one. But my primary one is not visible to me when I'm thousands of miles away.
David Busby: Not trivially available to me.
david raistrick: Right. and…
David Busby: I'm so far away. Let me just get onto my VPN into that network, connect over to the box, look at the stuff, do the things, get my MFA code.
david raistrick: if your VPN's down, fine. You SSH into your private bastion host that has access. And from there, you SSH tunnel or you just SSH in and find the files. But it's still it right.
David Busby: I'm going to tell you something.
david raistrick: It's all…
David Busby: My built or…
david raistrick: because wait, I might have lost my phone, but I still need to log into AWS.
David Busby: my bank sent me one of those little things.
david raistrick: My bank my battery has died. Whatever it is,
David Busby: And so every time now I go on vacation. I want to leave the office and do something somewhere, and that little key that's on my desk all the time. Got to take that little key that's on my desk with me when I'm traveling…
david raistrick: Yep. right?
David Busby: because every sign in is you have to do this key gizmo. I sign in from this box right here. That never happens. But if I lose that thing, I'm basically hosed, for banking until they send me a new one. And they did send me a new one. And I s*** you not, bro. They sent me a new one. It sends in a FedEx overnight. There's a hole in the package. There's the paper in there that says we've sent you the new key, but the hole in the package in the damn key fell out. So, I didn't get a new key. and my old key is running out of time, in fact, the old key I still have the old key. It says off, right? But I didn't have the new key. And I'm like, "Oh crap, I'm going to be totally locked out." And, this is all just extra crap I have to do to do the crap I want to have to do. I'm now all the way over here to pick on this 7.4.
David Busby: for PHP 7.4 app, which is, in great shape. It's working. It's making money. Who really wants to mess with it? we don't want to run 7.4 anymore. Although, I still have a client that I work with that has a PHP 5 app that is still use in their facility making money.
david raistrick: the only PHP I'm still running.
david raistrick: Last year maybe it's the year before I finally invested the time in moving it up to I think probably about 7.4 actually.
David Busby: Yeah,…
David Busby: Bull Debian Bullseye or whatever.
david raistrick: It was a large lift. I mean there was a lot of even for this very very old I mean this thing started out probably PHP2 so the last time it had been updated was for PHP 5 somebody else had done a seven lift but it didn't support the modifications I'd made to it as a dead open source project so …
David Busby: Still works when you're running 5.54,…
david raistrick: but of course it still worked yeah
david raistrick: But I wanted to run it in Docker and…
David Busby: but you upgrade and then you've got to fiddle a bunch of stuff because Right.
david raistrick: I wanted to put it in a new environment so I could get it off of the machine that was installed in 2004 that it had been running on, and get it a little bit more manageable.
david raistrick: That and getting PHP 5 running in Docker was really painful. So it was easier to spend the week porting this thing forward to seven.
David Busby: I mean I agree that's the right choice there is move it forward when you can so the folks I know…
David Busby: who are still running the 54 5 something app that one started off as a deploy on a bare metal in their facility right like they set up a Debian machine, put the thing, got it rolling. We're like, "Here we go." and somebody internally who's not a developer, was like, " I can drop PHP files and interact with a database on a web server let's build our own internal tooling that we use." and does this very simple thing, generates a nice HTML page that comes out that almost every single time fits on a single piece of paper and…
00:10:00
david raistrick: right?
David Busby: generates this printed document is that organization's very valuable work product. That's that piece of paper and the work that's necessary to generate that piece of paper. But that piece of paper they bill $4,000 and they make a couple of these pieces of paper on a daily basis. They're like, " so the last time I talked to them, our server Help us recover all of the stuff off there." Fairly easy. You just pulled the stuff off the drive. the drive wasn't corrupted or anything. The server just fell over the hardware. I didn't bother to do any diagnostics when they called me up.
David Busby: It was like, "Hey, we got the server. It's dead. We got the drive out. It looks like it has stuff on it. Can you help us set this new thing up? We have a new server. Just move it on there." their new server is running various services on it already. An existing web server, so we can't put Apache on there. It's got a newer version. maybe we'll have to do something, but probably not there. Apache is mature and very stable. Doesn't wiggle too much from 2.0 0 to 2.4,…
david raistrick: Hey, heat.
David Busby: but we have PHP 5.4 on this thing. The bare metal server we want you to go on is running 7.2 or whatever Debian had popular in two Debians ago. Maybe that was Debian 10. Are we on 11? The one before Bullseye, I think, was Buster. Anyways, we have that. It's 7.2. So, it's probably compatible with the necessary tweaking of the PHP configs, maybe a little bit of updating some stuff, in terms of some routines used to be able to pass a null in or…
david raistrick: Right.
David Busby: an empty string or an empty array and it would do the right thing. PHP would allow a lot of slot back in the day and now it says, "Nope, you can't pass in a null. You have to pass in an array or it must be a string." sometimes there's some other stuff there. although that's not so heavy and that type requirement is not so heavy in seven but it is definitely an eight. so then we had to do the thought process like what are we going to do? We put it to make it work as easy as possible for us without any do fights with Docker. We just made a new VM and made a new VM with this very old Debian with 5.4 or the legacy Apache put the thing up on this box. It's got a proxy server on the front end that now proxies to this thing. We had to change one or two lines of code at the top of the thing so it knew where it was, right? Because now I'm on a reverse proxy.
david raistrick: right?
David Busby: I'm not bare metal. I can't use things like environmental variable of my server name to tell me whom I am because my server name is wrong and is not my externally visible origin. So we just couple lines of code fixed that problem sorted. Then they said we should upgrade this to 7 whatever which was the thing at the time. Said sure we could do that. We'll get to it when we get to it. and now it's not going to seven now it would be let's go to eight.
David Busby: So now we'll get to it when we get to it. there's been some interest added, let's say, right? That's a little rough.
david raistrick: Right. Right.
David Busby: So then that one basically They avoided this vendor driven churn. They avoided a 300% type problem because they didn't bother to really get in and didn't do anything. The easiest work is nothing, I mean, put it in a new VM and done. and how often I have had to do that for folks. I think spend some time at some larger organizations and…
david raistrick: If it's not broke, don't fix it, right?
david raistrick: I mean,
David Busby: I spend some time I would say they are small and medium-sized businesses where, there's not eight people on the tech team, there's two. and one of them's been at the company for 20 years, knows everything about all the systems off the top of dome, stuff like this. And then the other person's been there for a year or two, and this is their stairstep job into, being their own IT manager or something and they end up having to do a lot of that outsourcing, code us up our app real quick. Hey, patch this app that somebody who worked here wrote so that it's less crashy or whatever. Those kinds of things. And I see really a of let's just keep this legacy system moving forward. Let's just keep this legacy system moving forward.
00:15:00
David Busby: let's set up virtual box they got upgraded desktop computer but they had this really legacy app from their old machine that they were running that was still a Windows 95 let's just put Windows 95 in a VM and that's how you use that app now it's like you can copy and paste through the clipboard through Virtual Box it's sort of an awkward experience for someone, but the staff spends three or four days getting used to, I run that app in that other Double click VBox, doubleclick this. Now I can doubleclick bespoke proprietary app that we wrote that runs in Windows 95 but was originally written for DOSs, right?
David Busby: run a start Windows 95, then open the terminal window and full screen And that's what you're doing, Tab to move between fields. Don't type in exotic characters, still support only supports a two twodigit year, so it still thinks everything's happening in 1925. so they're avoiding that urn. But these other folks that I'm working with now have the 7.4 PHP and…
david raistrick: right?
David Busby: it's Docker. So that's cool. And they want to move it to 8.2 or whatever's on the latest Debian, which I think is 8.2 ain't that something anyways and we couldn't just let's upgrade the PHP engine and everything's done no it starts throwing barf everywhere deprecated that warning here type hints got slightly more strict which is a good thing but the code wasn't ready for that kind of stuff they did some pretty sloppy things with regards to name spacing in
David Busby: there. So, fun mechanisms composer don't work as tidy as they could be should be to build the big autoload list. They had this other thing now. we've upgraded this. Guess what? We went to 8.2. I have all these dependencies. So my dependency that I use to connect to or generate QR codes, I got to upgrade that But that library when it moves from its compatible to PHP8 version, now it's got a new API interface.'t The old way was really static classes and functional, and now the new one's all OO.
David Busby: That's just a contrived example, but stuff like that happens. This one's changed some method procedures. This one's got the new thing. So now you have to upgrade your dependency A, B, C, D, E, so that they're compatible. Now you got to upgrade your own code to be compatible with your dependencies as well as,…
David Busby: the engine that you just moved to. And as if PHP7 to 8 is the only time that this happened, my least favorite experience with this vendor driven churn was the early React stuff.
david raistrick: Yeah.
David Busby: Yeah.
david raistrick: Changing every week was something was
David Busby: On a sort of a side project, I was just trying to help someone who's getting into coding doing front-end stuff. And there's this other front-end UI I called Riot, which is very much a React-l like thing. Build a little custom component, throw it in the web page, and it's all JS controlled, and it renders itself nicely, does its thing. Great. and I was sort of just saying there's React and there's these Pact and you could have spelt and you could have Vue and you could have blah blah blah.
David Busby: Riot was one of the ones in the list. I pointed this group at try Riot. It seems to be moving less quickly, which is true, or at least it was true. and then guess what? Even that one that was trying to be narrower scope than what React does still ends up moving from Riot 2 to Riot 3 to Riot 4,…
00:20:00
David Busby: which are mostly the same. And then we got stuck with a thing that was like, " no. You got to go into every one of these tag files that you make." And just make sure to add export default to every single thing and…
david raistrick: right? we had a build failure that started happening on one of our C++ apps that it would only happen in debug,…
David Busby: that's fine. It's no big deal. It's only a 100 components. Let's just run that in there. I don't know what's the worst I don't know. the prime driver of the vendor driven churn is those** OS manufacturers keep producing new versions.
david raistrick: which of course it's not like you can't build and debug because this thing's hundreds of thousands of lines of code with hundreds of developers supporting it.
david raistrick: And so you have to have debug builds because we use the debug objects all the time in development. We don't ship them to production, but we can't run development without production without dev. Right. Yeah.
David Busby: You got to debug.
David Busby: You're going to debug without debug.
david raistrick: And this happened when Red Hat had updated a library and we're picking it up down COS land, I guess. but we're running COS and all of this stuff, but we also have Red Hat support contracts across the organization, so it's not like they're out of reach. and they'd updated a library for the debug build. I forget what it was now, but I know that package had changed. And the reason I know that package had changed is because I reversed the update, right? So, we applied updates, did the things, and we were stuck at that point.
david raistrick: And I had to go and build a box and back out packages and install old versions of packages until I could figure out what broke, Still I could actually get a working build again going back toward the original target. Isolate what caused it to break.
david raistrick: Go up to Red Hat and tell them, "Hey, let's open a ticket. Let's let them know that they've broken this." And they're like, "Great. Give us your code. I'm not giving you a 25 year old game platform.
David Busby: That's a little rough,…
David Busby: but come on.
david raistrick: Sorry, we can talk to our legal team and see how long this takes to get approved, but that's not going to happen. So, our solution ended up being that we had to fork GCC. so we forked the GCC RPM, And all of the other and the couple other RPMs that we needed to patch to apply our patch so our code would build, which it built before.
david raistrick: And then because we're doing this at the RPM level which still made sense based on all the way we needed to distribute these changes that we had to rebuild we had to make world the entire RPM tree because everything depends on GCC.
David Busby: Right. Everything does.
David Busby: I use Gen two on my desktop and there's a thing that occurs every few years that is a profile update I was on 15 then I was on 17 then I was on 21 and a new one came out and big things happened in the whole file system space across the whole universe of the Linux world merging user so that we're getting rid of slashbin like the FHS specifications or standards or suggestions have changed since 2015, and now my Gen 2 update, and this is the same Gen two box I've been running for, it's effectively the same Gen two box. It started off as a 32-bit box. because Gen Two, you have to rebuild s*** all the time.
David Busby: had a mechanism where I was not able to promote this box itself from a 32-bit box to a 64-bit box by itself. because how am I going to do all this compiling 64-bit ready when I'm still 32-bit kernel, but I'm on 64-bit hardware. So, I had to, boot from an external system rescue CD, which was actually on a System rescue CD goes in the thing, gives me the 64-bit kernel, and then you're able to move some stuff and then, mount the thing and then change route into the other system. And now I'm using these Gen two tools and they're compatible because my kernel can run 32-bit s*** and I have 32-bit s*** on that box.
David Busby: but now I'm in the gen two space and so I say so you run one emerge of some of the core tools GIBC GCC there I can't remember them off the top of my head. There's three things that I had to rebuild and then one of those three and now they're all 64 bits and smart and then I had to take another one of those things and rebuild it again to be built with itself. I probably was build the other things that it likes GIB and libc then rebuild GCC again.
00:25:00
David Busby: and then it was still having to build it in multi-lib mode. So it would support 32 and 64-bit. And then you say emerge world and then that rebuilds GCC and all of that s* a third time and takes forever. But this is the life we chose. and then I think I ran that multi-lib one for a bunch of years. And then there was another one where you want to switch and get rid of multi-lib, so you're only 64 bits only, which I had to delay because of some 32-bit crap I needed to work on, not anymore. and then it was rebuild everything to drop multi-lib support, which boy, I think that only happened in 2017 or something like that.
David Busby: And then there was another big profile update on here which changes a bunch of the default build flags. Lots and lots of other stuff. Some things in the file system had changed. But to Gento's credit, these upgrades work more successfully for me than any Auntu upgrade I have ever done. every Auntu upgrade had a terrible problem and I reboot the machine and it comes back up and something didn't complete or something big is config is weird or something something and you got to look it left all these old packages in there and you have to undo them somehow they got installed but not upgraded weird stuff like this I had two boxes that were identical Iuntu servers
David Busby: circa 2020. Update this one and it bounc it and it comes back up and everything was fine. I was like, wow. I'm finally good at this one." And then I do the same stuff on the other box and it won't start running. And it's got some complaints about files system permissions and weird stuff and it's like how the s*** did that…
david raistrick: right? Right.
David Busby: how did that even happen? to Gen 2's credit, Emerge World, I mean, I got a beefy machine now. it takes all day. But mine runs through mostly most of the time I got to do that. the time it doesn't work is it crapped out rebuilding the kernel because it had done three kernel updates and filled up my boot partition with, because it doesn't delete the old ones and it's saving them. So now I've ran out of space on boot partition.
David Busby: Okay, that I feel like is on me because I started this update process with not enough room on my boot partition for a new kernel. Okay, silly Maybe the package manager should check. maybe. Sure, sure, sure. But I've been using Gen 2 for more than 15 years.
david raistrick: right?
David Busby: So really, I'm like, I should know better because this is a very manual distribution. compared to I use Debian in all my production everywhere else. the Iuntu shop that was somebody else's choice. My own choices are I'll put Debian, which incidentally has had the smoothest upgrades ever. Every time I have to do one of those, they're like, "You should start another shell in case this thing has a problem." f*** no. Let's go.
David Busby: In fact, what? I'm gonna hit enter and then I'm gonna start drinking. I know this is gonna work out. and yeah, I guess it generally does except for weird where my app layer comes down to it. But this is all the way down to your not yours, but Lee Briggs 300% problem. To successfully get the app in production,…
David Busby: I got to know the app as an expert. I got to know my They say deployment target as an expert and I got to know the method for deploying also sort of at an expert level because my deployment isn't drop a PHP file into the web route and I'm running.
david raistrick: right? It's somewhere from okay I've committed my change now I push my change goes somewhere my change…
00:30:00
David Busby: We're 20s something years away from that being reasonable. you don't just update php and now millions of people clicking the button have their code changed. It doesn't work that way. it's got run through your CI to CD pipeline.
david raistrick: then right sure deploy right I know it's all got to be colors
David Busby: Then Maybe it goes to your what do they call them these days? blue green employment used to just say AB but hey cool man I guess maybe I worked on a team…
david raistrick: No, gone through the whitelist change and…
David Busby: where somebody got really mad about the choice of the terms blacklist and whitelist that had been in the code for 20 ears.
david raistrick: the master slave change because you can't call a primary and a secondary a master and a slave. I mean, that's fair,…
David Busby: I mean that's…
david raistrick: but that's a lot of code to change.
David Busby: what it was a lot of code to change. I think that's a problem. I understand that problem connotations. words mean something and stuff like that. But I feel like sometimes the engineering work of …
david raistrick: it was probably six months of work for the entire company to do the master slave blacklist whitelist changes for a firewall. Yeah.
David Busby: can't we just leave this one and going forward will be doing stuff that is more inclusive? there was a lot of work I observed at a bunch of places like backporting changes that were primarily about feelings. Again, I respect feelings a lot. I was about that change. They said, "We need you to sit here and implement that change." I said, "Yes, let's do it. It's not a black white list anymore. It's the accept list and it's the reject list because that's what you wanted." Those are better words. Really wild.
david raistrick: because you've got documentation, you've got training,…
david raistrick: you've got the actual code, you've got to come up with new words, right? we had a set of words that were standard that everybody understood in this application of…
David Busby: You got the other 300%.
David Busby: So, right.
david raistrick: what they meant. you had to come up with all new sets of words for all of these things. Do we call this a block list and an allow list? But, okay, I mean, we found words, right? We call them primaries and secondaries, but then again, you might have multiple secondaries. What do you call it when you have three or four secondaries?
David Busby: read replica.
david raistrick: Yeah, but it's a firewall,…
David Busby: I don't know. the left and server on the right. FWL and…
david raistrick: right? Secondary one,…
David Busby: FWR matrix zero.
david raistrick: secondary two. Wait a minute. Secondary, tertiary. What happens when they change relationships? Who's third?
david raistrick: let's face it, when it was slave, we were interchangeable,…
david raistrick: all the bad connotations, but the actual application of the term worked very well when we're talking about how a highly available set of firewalls or a highly available set of routers apply are working. there's a lot of human effort that goes into all of that.
David Busby: Right. But…
David Busby: but there wasn't any breaking changes from any of this, right? we moved all of our legacy code from legacy words with negative connotations and…
david raistrick: No. Right. Yep.
David Busby: it was just another little bit I started off complaining about vendor driven churn and now we've were like this isn't even the vendors this is not an OS update we did this one to ourselves I mean I'm about it I spend a lot of time in the cannabis space and…
david raistrick: Sure.
David Busby:
David Busby: terminology in the cannabis space is that you can't right but marijuana is now negative term and has some negative connotations and I won't fully understand them but I was at a standards group sit down and they're like we got to say J don't spell marijuana with an H call this thing cannabis we should call it cannabis everywhere I think the only vendor
David Busby: driven churn that put for me is a bunch of the state regulations went through and changed the agency's name to take marijuana out of the agency name and now it says cannabis and then a whole bunch of the data in the system they used to classify stuff as that's usable marijuana now it's classified as usable cannabis and there's spots where interacting with those systems you don't give it some immutable identifier fire that represents usable marijuana or usable cannabis or whatever new term we are going to come up with later.
00:35:00
david raistrick: Right.
David Busby: Immutable thing that really means smokable flour. Nope. You have to send it a text string marijuana. you send it one that says usable space cannabis. if you send it the wrong one, the system rejects it, and that's hard that's hardly a 300% problem, That was very much a string replace.
David Busby: The agencies in the cannabis space we're going to do a stage deployment. We'll accept both words for two weeks and…
david raistrick: Right. Yeah.
David Busby: then they're just like,…
david raistrick: But you still got multiple deployment targets.
David Busby:
David Busby: hey, guess what? Starting on Sunday night, we are gonna do this thing. So, everybody else who interacts with that agency is " cool." So,…
david raistrick: The way we should do this This is V2 of the API and V2 accepts cannabis and V1 accepts marijuana and the two don't mix and…
David Busby: right. You could do that.
david raistrick: then we put it in the date. We deprecate it and then we depreciate it.
David Busby: Yeah. I mean, but it doesn't happen that way a lot of the time,…
david raistrick: But no one does that right.
David Busby: right?
david raistrick: The right way and the way that our vendors enforce upon us.
David Busby: I mean, there's no API versioning at all. one of the other ones they just extend some of them you're just pushing JSON up and down, And so they're just like, "Hey, we'll add new fields. f*** it." know, because who cares, right? Jason,…
david raistrick: It's seamless. It's good.
David Busby:
David Busby: that'll work. No problem. if you try to check, I want to do a little schema validator, that causes a bit of a problem, but all of that stuff is just sometimes I feel like how am I going to get forward when I spent today, I had a little project today for some web app that I helped some folks on and it was to just go through their code, a new Chrome version came out, somebody who doesn't need to be was looking at the developer console in Chrome and it starts telling them, " is deprecated. This is deprecated."
David Busby: cookie same site lacks versus strict was one of those things that came up for the same group because the same non-engineer looking at the dev console opens the dev console because showed them one time here's how this dev console shows up. So if you are working on this app or our web deployed thing and you want to see and help the engineers if you find something open the dev console and then take a screenshot and it'll show a bunch of s*** right there that it'd be better if you could get a more complete bug report.
David Busby: But I am not expecting a sort of senior manager to do that kind of thing.
david raistrick: I don't know…
david raistrick: how anyone debugs web problems anymore in the world where browsers hide URLs by default. So, we don't even know what page you're on.
David Busby: My shows me…
David Busby: what URL I'm on.
david raistrick: You changed that config,…
David Busby: No, it's just Chrome out of the box.
david raistrick: bro. Are you sure? because Chrome changed the default so that it hides URLs. Now maybe they unchanged the default…
David Busby: And I've got back forward. Look at my screen here,…
david raistrick: but there is a configuration for it.
David Busby: There's my thing right there. I'm looking at your 300% problem on I haven't changed any config on anything.
david raistrick: I know and I bet you have the config turned on.
David Busby: I'm on Chrome 134 blah blah blah blah blah.
david raistrick: This happened a couple of years ago.
david raistrick: I actually had to fix it on a browser recently. I think when I set up a new laptop maybe on Linux they default it differently.
David Busby: I don't use a different platform, so I would not know.
david raistrick: Let's see. I don't remember what they call the control.
David Busby: I don't even know where to look for it under here.
david raistrick: I know.
David Busby: Maybe you're thinking about I don't know. But I still have back forward reload my URL bar, the little star so I can put it
david raistrick: It's the thing where They strip the HTTP off and they strip the thing off.
David Busby: They don't take the whole URL off, but they do have HTTPS. That tricked me two days ago when we were doing the Caddy one, and I was like, " it's giving me an SSL error because Caddy hadn't got theert yet, but it was trying to hit it on HTTPS.
david raistrick: Yeah.
David Busby: I go into the address bar. I took the S out. Then I hit enter and then my site loaded on that demo caddy that we were doing." And I was like, " okay, cool. Now it's working." And I was still assuming I'm on plain text. And then you're like, " yeah, and there you got SSL out of the box." I said, "No, I didn't do anything." Blah, blah, blah. And then I go back into the URL, put the focus and look, the scheme is HTTPS. That was wild and it automatically fetched it. I did another test with This other project's demo environment or their sandbox deploy that we're working on wants to have I think 16 different host names pointed at that one box.
00:40:00
david raistrick: right? Yeah.
David Busby: So I end up with this caddy file that is way simpler than my The Apache one is whole Each file is very very big. Lots of stuff I got to set. And then on this one, it's like the three lines. Here's my web route. do a file server thing. Do the fast CGI thing. three lines for a single one of those. And turn Ran that one, in the foreground when it first kicked on.
David Busby: And it took I would say four maybe five seconds to resolve all of the SSL searchs that it needed to get for that host.
David Busby: Took…
david raistrick: Okay, it does look like they took away hiding the URL part.
David Busby: but now every time I start it's fast.
david raistrick: They hid it for a little bit, and so it's all show full URLs or it's in the view menu and it's the same thing that turns on off HTTP.
David Busby: I don't even have a view menu.
david raistrick: I don't know where Linux puts these things. you don't have a view menu. Where's your history menu?
David Busby: Why do I want to know about the history that is in the past?
david raistrick: where is your file menu your profiles?
David Busby: Only do this. It comes dot off of the bro.
david raistrick: Yeah. No, The stuff that's not in the hamburger really like…
David Busby: There is only hamburger. This one also I do bookmarks and…
david raistrick: where do you turn on and off your bookmarks bar because you have it turned off.
David Busby: lists. …
david raistrick: Show bookmarks bar there.
David Busby: show but I don't like it's ugly.
david raistrick: No, that's no. Show bookmarks bar there. So, I wonder Where's your Zoom is there.
David Busby: So there's Zoom.
david raistrick: I don't know where they have it. If it's not because I don't find it anywhere except in the view menu on Mac.
David Busby: Whoa, look at this. Here's a feature. More tools. Name window.
David Busby: Get f*****. What did that do?
david raistrick: It probably doesn't help for incognito windows.
david raistrick: It might show up in your window list except you don't have a window menu.
David Busby: I do have a window menu. I do the super key and tab and then it tells me I have an onscreen display of what all my things are with the icon and…
david raistrick: Cuz that's the tab. Yeah.
David Busby: it has so the title in the title bar is the 300% production problem, But then the window title when I'm tabbing around the universe has the profanity that I typed in. that's pretty cool. That is a neat feature. I wonder if it is not keyboard shortcutable.
David Busby: That is disappointing because I've got 20 open windows. It would be really awesome to be able to
david raistrick: a Windows app that I was automating the other day.
david raistrick: One interface of it was fully keyboard shortcutable. So including the file menu you would always expect to be able to open the file menu, the next part of it. So the first part I do that and then it opens a new part. The new part has no keyboard shortcuts at all.
david raistrick: file menu isn't access at all. None of them. And I'm like, so I had to switch to pure screen macros at that point.
David Busby: Yeah, I don't see anything about hiding this.
David Busby: But also, this Chrome profile is attached to my managed Google domain for Adosio. So maybe that's something that set there.
David Busby: So this mid-level manager gets to her open URL.
david raistrick: Right click.
david raistrick: Right click on your URL bar. shows. Yeah, at one point when that was turned off,…
David Busby: Look at you.
david raistrick: it would hide everything after the first slash as well as everything before the first, in the protocol.
00:45:00
David Busby: Okay. I don't like it.
david raistrick: They must have changed the default on that, which was good because it made debugging problems impossible.
David Busby: gets kind of difficult. I get a lot of reports that here's a screenshot and it's only a part of a page and I'm like all of those components appear in six different spots in the app. which page were you on? They're like I clicked on the thing. Yes. Why didn't you say so they were doing this in the dev console and it was like here this deprecated this and that was like some vendor driven churn just like going through this mostly front-end app to pull out spots where different themes are putting in various meta tags for Apple and metatags for MS application all that stuff's now more or less standardized and then Chrome got updated
David Busby: Chrome starts warning you about these other legacy web s** are deprecated,…
david raistrick: right?
David Busby: then your management sees that and says, we've got to fix this right now. This is the big bug we've got to look at because, What if our clients who are also not technical visit a web app, open the inspector, and start seeing deprecated warnings in the console let's fix this team doesn't have time for it. I'd helped them out, once or twice. They're like, "Hey, we need you to come in for a couple hours, do this one little thing real quick." I mean, on the one hand, all this vendor driven churn frustrates the heck out of me…
david raistrick: It's nice to be the person they call.
David Busby: because now I'm doing this thing and I'm not making forward progress. I'm just Sweeping up behind the thing again. But at the same time, it's nice to have somebody call up and say, "I urgently need you to do this four hours worth of tedious." Tedious, not critical work, but we wanted to get it done because the person who writes the checks in the organization thinks that it needs to happen. But the technical lead is like, I can't take anybody off of anything else to have them muck about with that.
David Busby: make a branch, clean that s* up, run it through CI, and if it checks out and you've done all that stuff and debug it, blah blah blah, and all that checks out, send me the patch, send me a build, let's move on, and that's actually kind of handy. Being the person that they call anymore is a little difficult, there's a lot more people available to call in March 2025 than there was in for example March 2022 let's say or…
david raistrick: Yep.
David Busby: pick sometime in the past I mean just to reminisce a little back in the old dot blah blah blah blah blah I was working in Pioneer Square in Seattle there was a Quiznos sandwich shop down there cuz Quiznos was blowing up about the same time. Yeah. There were people who worked at companies that didn't compete with each other, but two different companies and lots of folks would end up at this quiz at the same time and at 11:30 nobody's At 11:35 the place is freaking crowded until 1:30 and then it's a ghost town again, right?
David Busby: So, a couple times I was in there and one of the folks in the story, the person who found a new job was at the moment on my team at company not my subordinate or anything, just a person I worked with on the team, nice guy. we're in Quiznos hanging out, somebody is in line with somebody else on their team blah blah blahing about something. And so my guy, Jeff says, " hey, here's something you should take a look at. Pay attention to blah blah blah blah blah when you're, trying to implement your own, search parameters for your own s* hack of a database mechanism you have. Try this stupid stuff that we've had to do in this circumstance." two weeks later, we're in the damn Quiznos again. She's the same dude. They're like, " that was really helpful. Thanks." Blah, blah, blah.
David Busby: Hey, do you know about such and such? He's like, " yeah, of course I know such and such." Cuz Jeff is super good at databasy things. I should say specifically super good at MSSQL databasey things. They're like, " yeah, here's what you should do. You should do You should do this." It tells them what to do. let me get your email if I can, maybe ask you a couple questions after this, something. Man, I s*** you not. two or three days later, Jeff comes in and he's just like, "So, today's my last day, I slid into this new gig, 20% pay rise, less hours, they're going to let me not have to be in early you guys want me to be, all this other stuff." And, everyone's like, "So, folks are like mad, mad."
00:50:00
David Busby: I'm like, "Yo, Jeff, they hiring, I'll make a pave up and not have to be there at 8 a.m. because somebody wants to have an all hands with a team that's remote that,…
David Busby: we'll be early to accommodate the remote team. We're the ones paying the remote team. They should accommodate A rant for a different time. Anyways, I think that's a wrap on this.
david raistrick: I hear you.
David Busby: 300% problem, vendor driven churn, nothing to be done.
david raistrick: Keep production simple. Everything you add to production adds complication.
David Busby: Yeah. I mean, I think this goes back to maintenance Monday and spending time to Just clean up some stuff. If you had maintenance Mondayed your PHP 5.4 app, you would have less of this problem. If you had maintenance Mondayed your Django and Engine X and two or three different Apaches on these backends and…
david raistrick: Yep.
David Busby: now it's like six servers. If you had maintenance Monday that a little bit, you could have paid down that debt in moderate increments instead of going, "* sake. now it's not going to cost me $200 to fix the problem. Now it's going to cost me $2,000." Make up bigger numbers. Agreed. yeah, that's a wrap.
Meeting ended after 00:51:45 👋
This editable transcript was computer generated and might contain errors. People can also change the text after it was created.