WEBVTT 00:00.031 --> 00:07.025 [JM]: Hey everyone, this episode marks the first time I'm recording my side of the audio while on the road, and boy oh boy does it show. 00:07.045 --> 00:11.514 [JM]: You will definitely notice it and hear the difference, and I hope it's not too annoying. 00:11.895 --> 00:15.702 [JM]: We will learn from this and improve our on-the-road recordings over time. 00:16.063 --> 00:21.955 [JM]: In the interim, thank you for your patience with the audio anomalies in this episode, and we hope you enjoy the show. 00:22.103 --> 00:36.326 [JM]: I saw the other day that David Attenborough turned 100 years old, and I can only imagine what it feels like to be 100 and to look back at your life and see just how much has changed over that time span. 00:36.947 --> 00:37.969 [JM]: I am... 00:37.949 --> 00:44.984 [JM]: roughly half his age, and it already feels like the world is very, very different than when I came into it. 00:45.665 --> 00:55.907 [JM]: And I mention this because there's a cool site I came across called sinceyouarrived.world, and it has a few different volumes, and the 00:55.887 --> 01:06.582 [JM]: original volume that's available at the base domain tells you not what has happened since you arrived into this world, but what has happened since you arrived on that particular page load. 01:07.023 --> 01:10.107 [JM]: So I loaded this page right as we started recording. 01:10.388 --> 01:19.060 [JM]: And since that time, however many seconds it's been, it seems that 738 children were born into the world. 01:19.040 --> 01:23.566 [JM]: And sadly, 357 humans have departed. 01:24.107 --> 01:47.258 [JM]: And it goes on to provide these counters, including how many hands were held somewhere on the planet, how many dollars were spent on war, how many songs were listened to, meals were prepared and shared, the number of questions that were asked of so-called AI instead of people, and how many species have sadly vanished from this planet. 01:47.238 --> 01:51.208 [JM]: And there's a couple others here, but it's an interesting collection of statistics. 01:51.669 --> 01:54.496 [JM]: Ten categories thoughtfully chosen. 01:55.037 --> 02:04.420 [JM]: And it's a cool visualization as these numbers increase so quickly that at least the right-hand portions of them are more or less a blur. 02:04.653 --> 02:26.505 [DJ]: This is a cool site, and I really encourage people to go take a look at it and not just listen to Justin's description of it because I'm not saying he did a bad job, but the design and the layout of the site are so well done and so intentional and careful that it really is a work of art. 02:27.086 --> 02:32.314 [DJ]: And the various sub-sites that you might also be about to mention, Justin, are... 02:32.294 --> 02:36.399 [DJ]: Likewise, I was really impressed by the work that's gone into this. 02:36.720 --> 02:46.752 [DJ]: One of the things on that main page alongside the various metrics it shows you is a link to how did they come up with these numbers. 02:47.393 --> 02:51.018 [DJ]: And that itself is very interesting and artful. 02:51.298 --> 02:55.744 [DJ]: So I really suggest taking a couple minutes out of your day to check this site out. 02:55.784 --> 02:56.665 [DJ]: It's pretty magical. 02:56.982 --> 02:59.967 [JM]: Indeed it is, and I'm glad that you highlighted the design. 03:00.067 --> 03:06.156 [JM]: It is very cool the way you have this light-colored side on the left and this dark-colored side on the right half. 03:06.557 --> 03:09.462 [JM]: The typography, it's a really well-designed site. 03:10.022 --> 03:14.890 [JM]: And as you mentioned, some of the other volumes are also interesting. 03:14.870 --> 03:21.487 [JM]: One of them has descriptions of what world skies look like on a particular day. 03:21.547 --> 03:27.903 [JM]: Volume three, I went to briefly, but then my browser asked me to share my location in order to make it function. 03:28.043 --> 03:32.254 [JM]: And I decided that that was not something that was going to happen this morning. 03:32.234 --> 03:39.784 [JM]: I wish there were an allow once option for these browser dialogues, but I don't believe there is such an option. 03:40.024 --> 03:42.968 [JM]: Because if I could just say, sure, this one time, go for it. 03:43.368 --> 03:45.951 [JM]: But I think it's pretty binary. 03:46.132 --> 03:49.536 [JM]: You either allow always or deny always. 03:49.576 --> 03:52.219 [JM]: And that just seems like a silly choice. 03:52.740 --> 03:56.605 [JM]: So Safari, thumbs down on that particular implementation. 03:56.686 --> 04:02.275 [DJ]: I suppose you could allow it and then right away go into your browser settings and remove that site from the list. 04:02.315 --> 04:03.717 [DJ]: But who wants to do that? 04:04.198 --> 04:18.520 [DJ]: And if what you're saying is true, it does feel silly because like the iPhone, if you go into an iPhone app and it requests your location, it does give you the opportunity to allow it once, I believe, or to allow it whenever the app is running. 04:19.020 --> 04:21.985 [DJ]: So, yeah, why don't they extend that to the web browser, too? 04:22.218 --> 04:22.859 [JM]: Absolutely. 04:22.900 --> 04:24.383 [JM]: So I didn't see volume three. 04:24.663 --> 04:38.855 [JM]: Volume four, the most recent one, presumably, when you visit it, shows you a bunch of information that websites can glean from your browser environment and the Internet connection you are using to access the page. 04:39.276 --> 04:40.699 [JM]: So it'll tell you. 04:40.679 --> 04:48.812 [JM]: roughly where it thinks you are, the kind of browser that you're using, the things that that browser is leaking to the website. 04:48.852 --> 04:56.144 [JM]: If you know anything about the concept of browser fingerprinting, then this shouldn't really come as a surprise to you. 04:56.304 --> 05:05.539 [JM]: But the bottom line is that your browser environment, when combined with your IP address and a 05:05.519 --> 05:11.189 [JM]: a unique identifier that sites can use to track you across the web. 05:11.750 --> 05:21.046 [JM]: And the point of the site is, of course, not to track you across the web, but to show you that other sites can do that and to show you what this particular site was able to collect. 05:21.186 --> 05:24.372 [JM]: Because if it can collect this information, well, so can 05:24.352 --> 05:25.433 [JM]: every other site. 05:25.854 --> 05:29.858 [JM]: And when I visited this at the bottom, it said 32 data points. 05:30.399 --> 05:38.008 [JM]: That's how much information this I was able to get from this MacBook Pro that I'm using to browse this site in Safari. 05:38.489 --> 05:49.802 [JM]: And at the bottom of this page, there's wording that essentially says this page has identified your device with enough details to distinguish it from most others on the 05:49.782 --> 06:00.095 [JM]: So if you aren't familiar with the concept of browser fingerprinting, and even if you are, this is a cool reference for seeing what kinds of stuff gets leaked when you visit sites. 06:00.577 --> 06:02.543 [JM]: All right, moving on to some follow-up. 06:03.130 --> 06:15.878 [JM]: In a previous episode, I talked about switching from MacVim to another text editor based on NeoVim that, unlike MacVim, is cross-platform called NeoVide. 06:16.359 --> 06:20.228 [JM]: And NeoVide is relatively new and is... 06:20.208 --> 06:28.722 [JM]: Slowly but surely adopting some of the more Mac specific features that I have become accustomed to over the many years. 06:28.883 --> 06:40.242 [JM]: And so one of the things that it didn't have that I missed is proxy icons, which are these little icons that show up at the top of many document centered application windows. 06:40.222 --> 06:45.169 [JM]: And you can drag that icon to other parts of macOS to drag and drop. 06:45.549 --> 06:56.544 [JM]: As if you were dragging the document from the finder, you can command-tap on them in order to navigate the hierarchy to see where that file is located and let go, and it'll take you to that particular folder in the hierarchy. 06:57.045 --> 07:03.194 [JM]: So I posted an issue just saying in the Neovide issue tracker how it would be cool to have such a feature. 07:03.234 --> 07:09.402 [JM]: And a mere 48 hours later, Alexander Falkucci posted 07:09.382 --> 07:16.839 [JM]: who I presume is one of the Neovite maintainers, submitted a pull request that implements window proxy icons for Mac OS. 07:17.260 --> 07:28.065 [JM]: And it's not available yet in a shipping version, but I downloaded a nightly, tested it out, and in a very peb-cac moment realized that... 07:28.045 --> 07:38.139 [JM]: The reason I couldn't get it to work is because I didn't actually read the documentation that says you need to add this setting, you know, via proxy icon equals true to the relevant configuration file. 07:38.259 --> 07:43.366 [JM]: So, you know, like most good software, you opt into this feature, particularly when it's new. 07:43.867 --> 07:50.396 [JM]: And as soon as I did that, it works like a charm, works just like every other window proxy icon on macOS. 07:50.416 --> 07:51.117 [JM]: So... 07:51.097 --> 07:52.298 [JM]: Hats off to Alexander. 07:52.318 --> 08:00.045 [JM]: I am floored sometimes at how open source works that 48 hours after I requested this, it was shipped. 08:00.345 --> 08:06.351 [JM]: So bravo and footnote because podcasts don't have footnotes, at least not when you're listening in real time. 08:06.851 --> 08:17.641 [JM]: A pebcac moment is an acronym that means problem exists between keyboard and chair, which means the reason that didn't work is because I was the problem. 08:17.721 --> 08:19.703 [JM]: The human was the problem in this equation. 08:19.919 --> 08:30.491 [DJ]: In this case, you forgot the fundamental rule of vim and therefore neo-vim and therefore neo-vide, which is that there is a setting for everything. 08:30.531 --> 08:33.841 [JM]: There is always a setting for everything. 08:33.961 --> 08:34.603 [JM]: Absolutely. 08:34.887 --> 08:47.894 [DJ]: And in fact, as a Vim enthusiast, I like that they did it this way because arguably the proxy icon is a feature of the Neovide application shell specifically on the Mac. 08:48.254 --> 08:50.078 [DJ]: It's not a feature of NeoVim. 08:50.058 --> 08:53.822 [DJ]: But I like the fact that there's only one way to configure the app. 08:54.243 --> 09:06.596 [DJ]: If you are going to configure things about the Mac OS application shell around NeoVim, you configure that the same way that you configure everything else in NeoVim, which is through your init, like your config files. 09:06.996 --> 09:17.668 [DJ]: I think that's a very clever way to keep the experience cohesive, even when they're providing something that isn't really inherent to, you know, what is fundamentally a terminal-based text editor. 09:17.885 --> 09:23.915 [JM]: Yeah, that's an interesting point because one can always argue for either side of that equation, right? 09:24.136 --> 09:34.854 [JM]: Where, okay, in a Mac environment, if there are Mac-specific features, why not put them into a preference setting in the Mac application like every other Mac application? 09:35.555 --> 09:37.058 [JM]: And I think... 09:37.038 --> 09:56.504 [JM]: I think in this particular case, I agree with you that that's probably not the right move in part because in a day and age where at least I am using multiple Mac computers, it's nice to be able to have settings that easily can be synchronized between those different environments. 09:57.045 --> 10:02.893 [JM]: So in theory, I only have to set this in my configuration file once and then the other Mac will just pick it up. 10:02.873 --> 10:14.496 [JM]: Whereas if this were done in a preference setting in a traditional Mac application sense, it would be much trickier to synchronize that setting across my different Mac machines. 10:14.637 --> 10:15.859 [JM]: So yeah, agreed. 10:16.020 --> 10:16.881 [JM]: It's the right way to do this. 10:17.603 --> 10:18.705 [JM]: All right, moving on. 10:18.865 --> 10:22.232 [JM]: In a previous episode, I talked about how 10:22.212 --> 10:37.420 [JM]: When running local large language models, I prefer to do that via Lama.cpp or LM Studio or a number of other tools, except for the one tool that seems to get the most attention. 10:37.400 --> 10:53.637 [JM]: recognition and mentions across the internet and that tool is called olama and i made my case in that episode as to why i don't care for olama and why i feel like you dear listener should also not care for it 10:53.617 --> 11:01.326 [JM]: And I don't know if I did an adequate job of explaining all the various reasons why perhaps you shouldn't use it. 11:01.786 --> 11:05.090 [JM]: Obviously, I can and have talked about why I don't use it. 11:05.350 --> 11:11.117 [JM]: But thankfully, I don't need to anymore because someone has done a very thorough takedown. 11:11.157 --> 11:15.642 [JM]: And I'm just going to read the opening paragraph of what is a... 11:15.622 --> 11:23.191 [JM]: long and very detailed site replete with so many link receipts it will make you dizzy. 11:23.752 --> 11:26.715 [JM]: "Ollama is the most popular way to run local LLMs. 11:27.216 --> 11:27.817 [JM]: It shouldn't be. 11:27.837 --> 11:30.300 [JM]: It gained that position by being first. 11:30.800 --> 11:38.449 [JM]: The first tool that made llama.cpp accessible to people who didn't want to compile C++ or write their own server configs. 11:38.930 --> 11:41.393 [JM]: That was a real contribution. Briefly. 11:41.373 --> 11:57.120 [JM]: But the project has since spent years systematically obscuring where its actual technology comes from, misleading users about what they're running, and drifting from the local first mission that earned it trust in the first place, all while taking venture capital money. 11:57.100 --> 11:59.544 [JM]: This isn't a both sides piece. 12:00.125 --> 12:01.087 [JM]: I've used Ollama. 12:01.527 --> 12:02.209 [JM]: I've moved on. 12:02.489 --> 12:03.471 [JM]: Here's why you should too." 12:03.911 --> 12:18.055 [JM]: And the author starts out by talking about the very distinct and intentional lack of attribution regarding llama.cpp, which is the whole thing that made Ollama work in the first place. 12:18.556 --> 12:23.344 [JM]: And I think that that should really be a big red flag for anyone watching. 12:23.324 --> 12:43.459 [JM]: Looking at any project, if a tool is almost entirely being powered by another tool and you go out of your way to never mention that other tool, that might not be the kind of maintainer and the kind of leadership that you want to invest your time and energy into supporting by using their product. 12:43.692 --> 12:45.535 [DJ]: Yeah, because what are they up to? 12:45.755 --> 12:47.457 [DJ]: Like, what's the actual value add? 12:47.858 --> 12:51.182 [DJ]: And then what is it going to remain over time? 12:51.463 --> 12:53.866 [DJ]: There's always been reason for skepticism here, right? 12:54.006 --> 13:01.617 [DJ]: I think as you laid out and as the author of this excellent article really lays out, that it's felt like a... 13:01.597 --> 13:16.618 [DJ]: It reminds me almost of how Microsoft's behavior was described many decades ago with the phrase embrace, extend, extinguish, with the idea that you take a thing and wrap your stuff around it and make it like... 13:16.598 --> 13:38.978 [DJ]: easier and friendlier, and yeah, come on into our little house, you know, and then once you've done that you can obliterate the old thing, and now only your thing exists, and that does kind of feel like Ollama's move to some extent, and to some extent they've been successful because it is so popular, like there are so many self-hosted LLM-driven tools I see that... 13:38.958 --> 13:43.527 [DJ]: when you look at how to configure them, it just goes like, oh, use, use Ollama to do this. 13:43.867 --> 13:50.199 [DJ]: And, and I find myself because I think you, you very early on talked to me out of, out of using Ollama for this reason. 13:50.239 --> 13:53.846 [DJ]: I find myself going like, wait, no, how, but how do I do it? 13:54.066 --> 13:54.908 [DJ]: You know, a different way. 13:55.175 --> 13:55.656 [JM]: Yeah. 13:55.676 --> 14:05.151 [JM]: And I think the angle that I was taking as it relates to this particular part about not mentioning llama.cpp anywhere is just this notion of trust. 14:05.471 --> 14:11.901 [JM]: It creates, at least for me, a sense of real distrust of the people that are running the project. 14:11.881 --> 14:22.278 [JM]: And I think that there's probably a moral and ethical argument for, yeah, you should mention the tool that allows your tool to even exist. 14:22.639 --> 14:25.684 [JM]: But it's more than just a ethical concern. 14:26.084 --> 14:32.134 [JM]: It's not just a matter, as the author of this article puts it, a matter of open source etiquette. 14:32.114 --> 14:45.913 [JM]: because the MIT license that Lama.cpp is released under has exactly one major requirement to include the copyright notice, which, as you may have guessed, Lama did not do. 14:46.434 --> 14:47.878 [JM]: And when someone brought it up, 14:47.858 --> 14:55.834 [JM]: It took well over a year for maintainers to acknowledge this problem and to rectify it. 14:56.335 --> 15:07.217 [JM]: If you consider proper resolution, adding a single line to the bottom of the readme that says that llama.cpp project founded by Georgi Gerganov. 15:07.197 --> 15:12.571 [JM]: But this lack of attribution is just one of many problems that this author delves into. 15:12.591 --> 15:14.957 [JM]: The link to this article will be in the show notes. 15:15.057 --> 15:16.662 [JM]: I highly recommend reading it. 15:17.103 --> 15:22.236 [JM]: And you're right, Dan, to point out this Microsoft strategy of... 15:22.216 --> 15:40.866 [JM]: Embrace, extend, extinguish, because it is a similar strategy to the extent that ulama takes the tool, doesn't provide any attribution, gets popular, and then replaces that tool so that they no longer have to provide any attribution. 15:40.846 --> 15:48.657 [JM]: and can claim to have added some kind of significant value because they're no longer basing it on llama.cpp anymore. 15:48.937 --> 15:59.832 [JM]: But the other thing, and this is totally predictable, is this pivot from running local models, which again is the whole reason that Alama exists. 15:59.913 --> 16:02.716 [JM]: The whole point is to run local models, presumably. 16:03.137 --> 16:06.702 [JM]: But because venture capital needs to make a return, 16:06.682 --> 16:15.817 [JM]: Unsurprisingly, they are pivoting to a cloud hosted closed source model that you presumably then pay Ollama for. 16:16.238 --> 16:22.328 [JM]: And the author does a great job of calling this out, saying that this follows a venture capital pattern very predictably. 16:22.308 --> 16:41.739 [JM]: where you launch on open source, you don't provide or at least minimize any attribution, you create lock in with proprietary model registry formats, you then launch closed source components, such as the graphical user interface app around Ollama. 16:42.139 --> 16:43.962 [JM]: And then you add the cloud services 16:43.942 --> 16:45.808 [JM]: which is the monetization vector. 16:46.069 --> 16:53.412 [JM]: And like any good takedown, the author provides a list of tools you should use instead, including of course, llama.cpp and llama-server, 16:53.628 --> 16:55.350 [JM]: as well as a bunch of others. 16:55.891 --> 17:10.288 [JM]: And finally, I'll note that it was really gratifying to come across this article because for a long time with seemingly everyone using Ollama except me, at some point I kind of felt like, is this me? 17:10.328 --> 17:13.191 [JM]: Am I the old man shouting at clouds? 17:13.732 --> 17:15.815 [JM]: Is this just another get off my lawn moment? 17:16.175 --> 17:17.897 [JM]: But to see someone else 17:17.877 --> 17:25.368 [JM]: See a lot of the problems that I saw two years ago or however long it's been since I originally decided this tool wasn't a good fit for me. 17:25.829 --> 17:30.275 [JM]: And then to come up with a list of a whole bunch of other things that I hadn't even considered. 17:30.676 --> 17:32.078 [JM]: Yeah, it was very satisfying. 17:32.158 --> 17:39.008 [JM]: So thank you, Zeta4, for writing this very detailed and well-written article. 17:38.988 --> 17:45.561 [JM]: Exposé, another related article that I came across is called Local AI Needs to be the Norm. 17:46.062 --> 17:49.569 [JM]: And much like the other article that I mentioned, well written. 17:49.589 --> 17:53.096 [JM]: The site design here is really cool. 17:53.316 --> 18:00.490 [JM]: I'm a big fan, including the blinking cursor at the top right next to the title of the article. 18:00.470 --> 18:06.963 [JM]: which is in orange, the only bit of color on this page, or at least one of the few bits of color on the page. 18:07.343 --> 18:19.407 [JM]: And the idea here is if you are designing software and in your configuration or preferences, you have a place to just paste an API key for OpenAI or Anthropic, 18:19.387 --> 18:39.117 [JM]: then you have effectively failed us by creating, as the author puts it, a generation of software that is fragile, invades your privacy, and fundamentally broken, resulting in applications that stop working the moment the server that responds to those API calls crashes or your credit card expires. 18:39.097 --> 18:48.250 [JM]: And essentially makes the argument that we should return to the habit of developing software where our local devices are actually doing the computation. 18:48.650 --> 19:06.054 [JM]: And I mentioned this not that long ago where I have this dictation software that I use and I wanted to use the cleanup feature of that dictation software to use a large language model to fix any problems with the detected language in that dictation. 19:06.455 --> 19:08.918 [JM]: But the API calls were just taking too long. 19:08.898 --> 19:16.872 [JM]: And I can't wait for 10 seconds for the dictation result to come back after it's been all cleaned up because it's just too long. 19:16.892 --> 19:18.756 [JM]: At that point, it's faster for me to just type it. 19:19.116 --> 19:26.890 [JM]: And it's only when I switched to a local model that I realized that the problem wasn't in the computation. 19:26.870 --> 19:31.899 [JM]: Or if it was a combination of the computation and the network latency. 19:32.360 --> 19:32.701 [JM]: I don't know. 19:32.821 --> 19:34.584 [JM]: But the bottom line is 10 seconds was too long. 19:35.085 --> 19:37.510 [JM]: And using a local model solved the problem entirely. 19:37.910 --> 19:47.528 [JM]: But this habit of just plugging everything into some remote LLM API call is indeed an anti-pattern. 19:47.508 --> 19:55.115 [JM]: And ends up being this distributed system that in the end costs you money that you don't necessarily need to spend. 19:55.175 --> 20:01.340 [JM]: You already have adequate resources to run very competent local models. 20:01.981 --> 20:17.515 [JM]: And if these features that software developers are adding, you know, can be done locally, bolting on the stuff that makes these remote calls is just, it's an own goal. 20:17.495 --> 20:24.643 [DJ]: From the business standpoint, I was just having a very similar conversation with a friend of mine the other day. 20:24.663 --> 20:39.080 [DJ]: We were talking about how, you know, in people's rush to embrace large language model powered features, I think maybe people are like losing sight of what is a very questionable business decision, which I put this way. 20:39.140 --> 20:47.350 [DJ]: If you take so-called AI out of it for a moment, if I just told you, okay, for the SAS company that you want to build, 20:47.330 --> 20:56.263 [DJ]: Do you think it's a good idea to build a piece of software where every time any one of your users use it, you have to pay another company money? 20:56.744 --> 20:58.607 [DJ]: Does that sound like a good business model for you? 20:58.647 --> 21:00.710 [DJ]: I think your reaction would probably be no. 21:01.270 --> 21:11.245 [DJ]: But all the companies building software whose features rely on calling other companies' APIs to get access to a large language model... 21:11.225 --> 21:13.087 [DJ]: are pursuing that business model. 21:13.687 --> 21:15.189 [DJ]: And that's not great. 21:15.589 --> 21:20.054 [DJ]: And then it's also not great for all the other reasons that you described that are in this article. 21:20.494 --> 21:37.131 [DJ]: The other thing this article reminded me of is the broader concept of local first software, which I have often seen referenced in an essay by, I think they describe themselves as a lab company. 21:37.111 --> 21:40.856 [DJ]: Inc and switch who are at Inc and switch.com. 21:40.916 --> 21:59.080 [DJ]: And they have this sort of essay manifesto about what they call local first software, which is the idea of, you know, to summarize software that principally stores its data and does its computation on your device, as opposed to all of that stuff being in the cloud. 21:59.521 --> 22:02.545 [DJ]: And if you have multiple devices that synchronize with each other, 22:02.525 --> 22:09.994 [DJ]: The synchronization is like downstream of doing the computation and storing the data on your device. 22:10.615 --> 22:17.744 [DJ]: And this has become a fairly popular concept in certain circles, which I've been happy to see. 22:17.784 --> 22:22.710 [DJ]: In fact, there is a local first software directory at 22:22.690 --> 22:30.004 [DJ]: lo-fi.so that you can explore to find various apps that take this approach. 22:30.485 --> 22:45.694 [DJ]: And not that concept is not limited to using large language models, but the large language model obsession that we have in current software development feels like a subset of it, where if you have software that's powered by a large language model and 22:45.674 --> 22:47.920 [DJ]: It is much better for you. 22:47.980 --> 22:49.705 [DJ]: It's better for your business. 22:49.785 --> 22:54.618 [DJ]: It's better for your users if those models can be on your users' devices. 22:55.200 --> 22:59.110 [DJ]: So I'm really a big proponent of this idea also. 22:59.191 --> 23:01.236 [DJ]: Like, I've invested in... 23:01.216 --> 23:20.828 [DJ]: personal computer hardware that lets me run LLMs locally, and while I still use tools from the big so-called frontier AI providers, as much as possible I try to build tools whose use of large language models can go through local hardware that I control. 23:21.044 --> 23:24.049 [JM]: Yeah, and in theory, there are two ways to do that. 23:24.149 --> 23:35.167 [JM]: So if you are developing software, the argument that's being made here that I think is 100% correct is ship a feature instead of shipping some distributed system. 23:35.548 --> 23:39.154 [JM]: In other words, instead of essentially outsourcing software, 23:39.134 --> 23:47.330 [JM]: some feature to some external API call to Google's Gemini or Anthropix Cloud or OpenAI's ChatGPT. 23:47.711 --> 23:50.115 [JM]: Do one of two things, or preferably do both. 23:50.536 --> 24:00.315 [JM]: Allow the user to choose one, okay, use an OpenAI compatible API so you can connect to a model that's running on your local machine. 24:00.295 --> 24:01.557 [JM]: That's a great place to start. 24:01.977 --> 24:03.419 [JM]: It's relatively easy to implement. 24:03.780 --> 24:10.048 [JM]: And then as a second step, look for APIs that are exposed on the device that your software is running. 24:10.449 --> 24:28.733 [JM]: Like we have these incredibly capable pieces of hardware, which are increasingly shipping with software that provides these APIs to software developers so that you can just say, instead of connecting to some external large language model, whether it's running on your device or not, 24:28.713 --> 24:39.988 [JM]: You can just use these built-in APIs to tap into the generative models that are available right on your own devices, thereby, once again, shipping a direct feature. 24:40.028 --> 24:45.175 [JM]: So to take an example of my dictation software, I can already do the first of these two things. 24:45.275 --> 24:49.882 [JM]: I can already and do connect this to a local model that's running on my Mac. 24:50.242 --> 24:53.927 [JM]: But sometimes I forget to start that server process that's running. 24:53.907 --> 25:05.663 [JM]: And so the dictation software will come back to me and provide an error and say like, hey, the cleanup feature didn't run because I couldn't connect to local host server running the model. 25:06.104 --> 25:20.503 [JM]: Really what I want is for this dictation application to use Apple's intelligence APIs to run the cleanup for me so that I don't have to connect it to some model running as a separate process that I have to remember to start and keep running. 25:20.964 --> 25:22.486 [JM]: That would be the ideal way to solve this. 25:22.806 --> 25:23.307 [JM]: So... 25:23.287 --> 25:34.765 [JM]: I agree with the fundamental premise in this article, and I hope that folks who develop software will take notice and start shipping features instead of distributed API calls. 25:34.930 --> 25:53.715 [DJ]: So I am a big fan of this notion of running local large language models, Justin, but I do want to encourage people not to use those models to write really crappy written work and then put it on the internet because there's too much of this already. 25:53.695 --> 26:05.776 [DJ]: And just like you were relieved to find that stop using Ollama article because you had these you have these strong feelings about this topic, but someone else has done such an excellent job of articulating them. 26:05.816 --> 26:13.088 [DJ]: I feel the same way about AI writing because I came across an article on 404 Media. 26:13.068 --> 26:17.593 [DJ]: the other day called Your AI Use is Breaking My Brain. 26:18.174 --> 26:27.965 [DJ]: And this was such a well-written and comprehensive explanation of something that, again, I feel really strongly. 26:28.326 --> 26:31.269 [DJ]: And I'm sitting here going, I'm not the only one who feels this way, right? 26:31.750 --> 26:32.290 [DJ]: I can't be. 26:32.851 --> 26:34.553 [DJ]: And of course, no, I'm not. 26:34.533 --> 26:50.078 [DJ]: And the writer here does an excellent job of describing like what it is about the writing generated by large language models that rubs many of us the wrong way for a variety of different reasons. 26:50.058 --> 26:53.464 [DJ]: I won't go through all of the arguments in the article. 26:53.484 --> 26:55.368 [DJ]: I think you should read it. 26:55.468 --> 27:03.402 [DJ]: And to that end, we will put a link in the show notes that should allow you to read the article. 27:03.883 --> 27:10.395 [DJ]: But for me, one of the things that frustrates me about this form of composition is 27:10.375 --> 27:12.318 [DJ]: That is becoming ubiquitous. 27:12.438 --> 27:14.040 [DJ]: I see it everywhere at work. 27:14.060 --> 27:15.562 [DJ]: I see it everywhere on the Internet. 27:15.943 --> 27:18.707 [DJ]: It's not just the tells that are in the writing. 27:19.228 --> 27:23.614 [DJ]: Incidentally, one of those tells is what I just did, which is it's not just X, it's Y. 27:23.634 --> 27:31.185 [DJ]: As the writer of this article points out, like you'll notice that sentence construction has become so common. 27:31.165 --> 27:42.737 [DJ]: And unfortunately, there's this weird feedback loop where the more that AI writing, so-called, sounds a certain way, it makes us question each other's writing. 27:42.777 --> 27:54.229 [DJ]: Like, wait, did you use AI to write this just because a person chose a sentence structure that is commonly found in the output of large language models, which, after all, were trained on what people write in the first place? 27:54.249 --> 27:57.051 [DJ]: They just over-index on certain patterns. 27:57.612 --> 28:00.615 [DJ]: The real pattern that I don't like in AI-generated writing is 28:00.595 --> 28:13.874 [DJ]: is it uses a lot of words to say very little and one of the classical indicators of good writing is exactly the opposite, which is saying a lot in only a few words. 28:14.474 --> 28:19.161 [DJ]: Doing that is very, very difficult, which is why most people are not good writers. 28:19.141 --> 28:21.484 [DJ]: Because writing well is very hard. 28:21.524 --> 28:27.832 [DJ]: That's why there are entire courses on it and entire books about it and bodies of work that are about this topic. 28:28.232 --> 28:30.475 [DJ]: It's hard to write well. 28:30.855 --> 28:34.039 [DJ]: It's hard to communicate effectively in writing. 28:34.580 --> 28:36.522 [DJ]: And large language models are bad at it. 28:36.903 --> 28:39.546 [DJ]: They're not good at communicating in writing. 28:39.526 --> 28:45.453 [DJ]: And so the more that people use them to do that, the more bad written communication we have. 28:45.893 --> 28:49.617 [DJ]: And the problem is, I'm afraid that the people using it think that it's good. 28:50.178 --> 28:58.267 [DJ]: In software engineering, for example, you can have a large language model generate a summary of your pull request instead of writing one yourself. 28:58.688 --> 29:03.373 [DJ]: I find these summaries excruciating to try to understand. 29:03.353 --> 29:05.096 [DJ]: Because they use so many words. 29:05.136 --> 29:15.695 [DJ]: They just go on and on and on and on about how the changes to this file leverage synergies to like to redact the improve the blood of the maintainability to the like. 29:15.796 --> 29:18.160 [DJ]: They're just they're full of buzzwords and stuff like that. 29:18.581 --> 29:26.475 [DJ]: When as a reviewer of your work, what I need you to tell me is you change this module to do X because Y. 29:26.455 --> 29:29.297 [DJ]: There was a bug in here and I fixed it by doing this. 29:29.658 --> 29:30.759 [DJ]: That's what I need to hear from you. 29:31.179 --> 29:33.982 [DJ]: Large language models do not help you communicate that effectively. 29:34.462 --> 29:46.953 [DJ]: So the biggest frustration I have about the ubiquitous use of generated text is the degree to which people think they're doing the work, but they're not. 29:47.393 --> 29:51.397 [DJ]: Because the hard work of writing is making sure that you're communicating effectively. 29:51.417 --> 29:54.960 [DJ]: And that generally requires you to write something and then edit it. 29:54.940 --> 29:58.045 [DJ]: And edit it and edit it and edit it. 29:58.606 --> 30:03.974 [DJ]: And instead, most people are just getting an LLM to generate text and then not editing it. 30:04.314 --> 30:15.492 [DJ]: And what I find so obnoxious about that is basically what you're saying when you do that is instead of me doing the hard work of getting you to understand what I'm saying, I'm going to make you do all of that. 30:15.992 --> 30:17.134 [DJ]: And I object. 30:17.114 --> 30:18.256 [DJ]: I object. 30:18.916 --> 30:23.282 [DJ]: So now you've heard my main objection to AI generated writing. 30:23.963 --> 30:37.722 [DJ]: And this article in 404 Media does such a good job of describing the many other objections to the ubiquity of LLM generated writing. 30:37.702 --> 30:43.271 [DJ]: That, again, like there's not that much more I can say about it other than please, please go read it. 30:43.391 --> 30:46.396 [DJ]: And then you too, you too shall understand my pain. 30:46.957 --> 30:51.845 [DJ]: And perhaps we'll also feel like someone else understands your pain. 30:52.065 --> 30:59.457 [DJ]: And really, Justin, isn't that all that we want out of life is to know that someone else understands how we feel? 30:59.690 --> 31:00.591 [JM]: Absolutely. 31:01.112 --> 31:02.855 [JM]: And I think that's the point, right? 31:03.316 --> 31:13.733 [JM]: I think that's the chief objection that some of us have to this notion of just interacting with a bunch of bots is that there is no empathy. 31:13.833 --> 31:18.901 [JM]: There is no, Oh, you feel what I feel and vice versa. 31:19.222 --> 31:22.247 [JM]: You don't get that when you're interacting with software. 31:22.767 --> 31:26.794 [JM]: And that's, I think the crux of the issue in general is, 31:26.774 --> 31:31.903 [JM]: and specifically here when interacting with writing that we see on the internet. 31:32.384 --> 31:43.062 [JM]: And you're right to point out that there is this evergreen now argument that's going to be had where you read something and someone posts a comment that says, oh, this is 31:43.042 --> 32:00.088 [JM]: ai generated and someone else posts actually i wrote that by hand and i didn't use software at all and then there's going to be people that say that this was real when it was actually generated that there's going to be this endless debate as to whether or not something was generated by software or not and we're not really going to know the answer 32:00.068 --> 32:09.742 [JM]: But when you feed a prompt into a LLM and you tell it to generate stuff, it's going to produce this homogenous slop because that's what it does. 32:10.323 --> 32:11.665 [JM]: And sometimes it's useful. 32:12.126 --> 32:12.967 [JM]: Sometimes it's not. 32:13.367 --> 32:17.033 [JM]: But almost entirely by design, it's homogenous. 32:17.593 --> 32:19.917 [JM]: And that's generally why I think we object to it. 32:19.937 --> 32:24.844 [JM]: That's generally why we can tell sometimes that it's not generated by a human prompt. 32:24.824 --> 32:34.381 [JM]: And one of the things that the article mentions is that someone created a browser extension called Sincerely, which is spelled wrong. 32:34.421 --> 32:37.266 [JM]: And I'm not even going to bother telling you how it's spelled wrong because who cares? 32:37.706 --> 32:46.782 [JM]: But the point of this browser extension is that there's three modes to undo your LLM homogenized writing. 32:46.762 --> 32:49.607 [JM]: where you can choose between subtle, human, and CEO. 32:50.128 --> 32:52.793 [JM]: And the idea is it will get rid of em dashes. 32:52.833 --> 32:54.375 [JM]: It'll add typos. 32:54.836 --> 32:56.098 [JM]: It'll add slang. 32:56.158 --> 33:01.247 [JM]: It will put words in all lowercase, just like people often do because they're lazy. 33:01.728 --> 33:08.500 [JM]: And this person wrote this because they got tired of everyone in their inbox sounding like LM-generated text. 33:08.480 --> 33:11.625 [JM]: And the intention apparently was that this was satirical. 33:11.805 --> 33:17.595 [JM]: This was meant to hold a mirror up to people who use these tools as a way of not actually thinking and writing. 33:18.196 --> 33:20.520 [JM]: But apparently it's like a legitimate function. 33:20.740 --> 33:28.132 [JM]: People actually use it to make their LM-generated slurry sound like a human actually wrote it. 33:28.393 --> 33:33.441 [JM]: And there's just something infinitely comical about a tool that adds typos. 33:33.421 --> 33:36.770 [JM]: So that when you read it, you think, oh, okay, this was not a large language model. 33:36.810 --> 33:37.552 [JM]: A human wrote this. 33:38.033 --> 33:43.127 [JM]: I mean, I'm left somewhat speechless when considering that this is the world that we live in now. 33:43.293 --> 33:50.941 [DJ]: It feels like a warm bath to me because any good satire is going to go over the heads of some people and they'll take it seriously. 33:50.981 --> 33:52.743 [DJ]: So this is a great example. 33:52.783 --> 33:56.227 [DJ]: Yeah, the author of this tool is quoted. 33:56.267 --> 34:03.315 [DJ]: This isn't a direct quote, but basically of saying something like, you know, I created this as satire to try to get people to think about what they were doing. 34:03.856 --> 34:06.038 [DJ]: But of course, that's going to be lost on some people. 34:06.278 --> 34:13.266 [DJ]: But yes, like the fundamental nature of I am so unwilling to 34:13.246 --> 34:28.386 [DJ]: Do the work or I so disbelieve that I'm capable of doing the work that I'm going to use a large language model to generate some writing and then I'm going to use a different large language model to make the writing generated by the first large language model. 34:28.366 --> 34:31.714 [DJ]: sound less like writing generated by a large language model. 34:32.134 --> 34:32.776 [DJ]: Y'all got that? 34:33.217 --> 34:43.520 [DJ]: I think people like me and presumably the author of this tool would just like want to virtually grab people by the lapels and say, what if you just wrote it yourself? 34:43.961 --> 34:45.244 [DJ]: Come on. 34:45.224 --> 34:46.385 [DJ]: Like, what's the point of this? 34:46.446 --> 34:56.578 [DJ]: And of course, the point is, for most people, the point is just no, I have to send 100 cold emails to people to try to make sales so that my boss pays me my salary so I can pay my rent, right? 34:56.619 --> 34:59.903 [DJ]: Like, that's the reason most of this content exists. 34:59.923 --> 35:09.595 [DJ]: But I continue to find something enervating about the degree to which people rush to stop doing their own work. 35:09.575 --> 35:22.758 [DJ]: The degree to which people don't care about craft in what they do, which I'm sure has always been true for the vast majority of people for a large number of reasons, many of which are valid, like the aforementioned need to pay your rent. 35:23.258 --> 35:27.145 [DJ]: But as someone who does care about craft in what I do, 35:27.125 --> 35:41.420 [DJ]: Yeah, I find it really obnoxious and a downer when people say out loud, because sometimes they do say the quiet part loud, and they say stuff like, you know, when I point something out, they're like, oh, yeah, I generated that with ChatGPT. 35:41.560 --> 35:42.703 [DJ]: I'm like, oh, okay. 35:42.683 --> 35:44.926 [DJ]: So like this isn't your work anymore. 35:44.966 --> 35:45.627 [DJ]: It doesn't matter. 35:45.647 --> 35:48.591 [DJ]: You're not responsible for it because a robot did it for you. 35:48.651 --> 35:50.554 [DJ]: No, no, my friend. 35:50.574 --> 35:57.163 [DJ]: No, please be responsible and exert your human agency over the things that you produce. 35:57.664 --> 36:00.808 [DJ]: Even if you use a large language model to generate some of them. 36:01.289 --> 36:02.831 [DJ]: That's that's what I want from people. 36:03.052 --> 36:05.535 [DJ]: I know I'm not going to get it, but that's what I want. 36:05.667 --> 36:07.991 [JM]: It is understandable that you would want that. 36:08.071 --> 36:22.478 [JM]: And I think that that's really where the author of the article is going with this, because they summarize their feelings very succinctly in the last paragraph when they say that what's driving them crazy is not the idea. 36:22.458 --> 36:25.162 [JM]: that these tools exist, or that people use them. 36:25.662 --> 36:44.888 [JM]: It is, as the article says, And this feels very similar to what I said before when we were talking about 36:44.868 --> 36:50.699 [JM]: LLM generated pull requests and issues in project repositories that I maintain. 36:50.759 --> 36:53.504 [JM]: The whole reason why I'm there is to interact with other humans. 36:53.905 --> 36:55.368 [JM]: It's not just to build software. 36:55.408 --> 36:56.730 [JM]: It's not just to ship software. 36:56.770 --> 37:03.062 [JM]: It's not to merge pull requests that just land without any interaction with a human. 37:03.042 --> 37:04.283 [JM]: It's the human part. 37:04.383 --> 37:05.284 [JM]: That's why I do it. 37:05.825 --> 37:25.906 [JM]: And I think that's why this person is so frustrated that we live in a world now where you have detection tools that detect whether something is written by an LLM only to have people use the humanizing software to make their LLM-generated output sound more like a human to bypass the detection. 37:26.407 --> 37:31.632 [JM]: And then it just becomes this ongoing cat-and-mouse game, which the... 37:31.612 --> 37:41.769 [JM]: vendors of the LLM generated text detection tools themselves say make it look kind of grim for the future of the internet. 37:42.230 --> 37:48.981 [JM]: And yeah, I'm just glad that there are a few of us that are calling forth the idea that really it's interacting with other humans. 37:49.041 --> 37:49.863 [JM]: That's why we're here. 37:49.903 --> 37:51.966 [JM]: That's what we're here for. 37:52.148 --> 37:56.755 [DJ]: Yes, and what really leapt out at me and what you just read was the phrase, couldn't be bothered. 37:57.256 --> 38:08.773 [DJ]: I think that actually puts the finger exactly on what I dislike so much is when someone goes in one way or another, hey, Justin, I would like your time and attention. 38:08.994 --> 38:16.846 [DJ]: I want you to spend some of your limited time on this earth interacting with, reading, consuming, using, reviewing, et cetera. 38:17.266 --> 38:21.092 [DJ]: This thing that I couldn't be bothered with 38:21.072 --> 38:24.478 [DJ]: To actually put any of my heart and soul into. 38:24.498 --> 38:27.785 [DJ]: I think that's a really gross thing to ask. 38:28.286 --> 38:35.519 [DJ]: And I'm sure without explicitly meaning to, it is what so many people are asking so many other people to do now. 38:36.000 --> 38:36.722 [DJ]: And it sucks. 38:36.742 --> 38:37.263 [DJ]: So stop it. 38:37.563 --> 38:37.984 [DJ]: Cut it out. 38:38.740 --> 38:40.445 [JM]: All right, everyone, that's all for this episode. 38:40.525 --> 38:41.448 [JM]: Thanks for listening. 38:41.468 --> 38:44.216 [JM]: You can find me on the web at justinmayer.com. 38:44.236 --> 38:46.803 [JM]: You can find Dan at danj.ca. 38:47.244 --> 38:51.737 [JM]: Reach out with your thoughts about this episode via the Fediverse at justin.ramble.space.