AI Grab Bag 2024

S-curves, open source, "artificial intelligence," Exadelic, and (probably) more!

Out with the old, in with the new, and a whole passel of things to write about:

1. Requisite Self-Promotion
   Skip to the next point if disinterested, obvs. Having said that: I am pleased to report that my novel Exadelic has been selected for an Amazon Kindle Promotion this week and thus until Sunday, is on sale in the US Kindle Store for a mere $2.99, a very temporary 80% discount(!). (From the authorial perspective this is slightly eye-popping, but it’s good for sales.) As such, if you have a Kindle and even the least interest in the book — which boasts multiple starred and rave reviews, along with praise from Josh Wolfe, Jo Walton, John Carmack, and even some others whose names do not begin with “Jo”¹ — ‘tis the week to buy.

   

   “A picture of a wizard withdrawing the following items from a Bag of Holding: - an S-curve - an open-source model - the phrase "artificial intelligence" - a book entitled ‘Exadelic’.“ Look, I had very low expectations, so it exceeded them.

2. The S-Curve
   This time last year astonishing new advances in the field were dropping not merely every week, there were weeks when they seemed to arrive every day, as I wrote back then. Stability AI — remember them? — were the talk of the town after dropping Stable Diffusion and raising $100 million. Then on Pi Day, March 14, after six months of rumors and red-teaming, GPT-4 officially launched to the public. And since then, of course, we have seen…
   
   …not actually all that much? I mean, sure, some advances, some interesting papers, but if you were to have told people last winter that this winter the state of the art would be basically unchanged, they would have laughed at you contemptuously. And yet … here we are. Sure, Claude 2 and GPT-4 Turbo have 100K/128K context windows vs. GPT-4’s “mere” 32K, but … longer isn’t quite the same as better. Meanwhile, Google spent an estimated $1 billion building and training Gemini, and its most advanced advanced model — which hasn’t shipped yet! — is … basically as good as GPT-4.
   
   In other words, we’ve seen no significant advances in LLM foundation models. Maybe they’re happening somewhere, behind the scenes! But judging from what’s been released — well, “treading water” is unfair, but we certainly aren’t on the kind of exponential curve that people were dreaming of / having nightmares about this time last year. The one arguable exception being entirely separate from LLMs: credit where it’s due, Midjourney was the locus of AI’s most mind-blowing improvements in 2023.

3. Open Source
   Instead, I think the biggest LLM advance of 2023 was the rise and rise of open source. Now, I think we need to be clear on what we’re talking about: “open source” LLMs are not (at least not yet) the product of bazaars rather than cathedrals. They, or at least the best of them, come from very, very well-funded establishment entities such as

   1. Meta (who gave us LLama)

   2. Abu Dhabi (who produced Falcon)

   3. and Mistral (a startup with some $500M of funding, co-founded by a former French cabinet minister, who released Mixtral.)

   
   But the mere fact that it’s possible to download and run these models yourself is pretty remarkable! The further fact that, in the aforementioned absence of basic foundation model breakthroughs, they keep getting closer to “genuinely competitive” is excellently surprising. And the possibilities — from on-phone LLMs to on-premises LLM applications to, eventually, personal amanuenses and analysts — are potentially endless.

4. “Artificial Intelligence”
   I’m going to pick on the American Dialect Society, which just announced “artificial intelligence: computerized simulation of human intelligence that is not actually intelligent” as one of its euphemisms of the year, but people outside the field are constantly complaining about the phrase, and have been doing so for many, many years. Simon Willison has a typically excellent post explaining why “It’s OK to call it Artificial Intelligence,” to which I wanted to add one point: these terms come from inside the industry, and they tend to emerge because engineers need to distinguish between things which really are taxonomically quite different. It’s hard to overstate how just different AI is from traditional software engineering as a field, for instance. We could call it “machine learning,” but then LLMs and diffusion models would be “generative machine learning,” which is basically a contradiction in terms. “AI,” while used differently in different contexts (like so many English words…) is in practice an extremely helpful both as a larger umbrella term and as a distinguishing term.
   
   OK, fine, while I’m ranting, a second point: the criticism that it enshrines “intelligence” would be more valid if its goalposts didn’t keep migrate. If, twenty years ago, you had asked a person “If I built a machine that could turn ideas into extraordinary art, write compelling essays on any subject, pass the bar and medical boards exams, analyze and write software, and translate nearly flawlessly from any language to any other, do you think it would be reasonable to refer to its outputs as the creation of a kind of ‘machine intelligence’?” After they finished looking at you like you were crazy, they would almost certainly shrugged and said “Yeah, sure, I guess, sounds reasonable.” But, (in)famously, as soon as computers can (semi) reliably perform an activity, that activity suddenly stops being perceived as a sign of intelligence. I leave you with this question: given that the Turing Test is no longer really fit for purpose, what would a computer have to do to ensure no one would again grouse about it being called “artificially intelligent”?

1. Though all the best names do, obviously.