My new new thing
You'll be shocked, shocked, to learn it's an LLM play.
[What follows is mostly navel-gazing about my new startup. If that’s not your thing, keep calm and carry on; regular speculative programming shall resume forthwith.]
Let’s talk about me
Once upon a time, I was the CTO of an ~80-person consulting company. We did strategy, design, and engineering. We helped to conceive and build whole products and companies (like these) from scratch.
Much of my job consisted of rapidly analyzing software projects so as to explain their real status and what they needed. If projects threatened to go off the proverbial rails, and got past our first defensive line, I was also the “free safety” tasked with tackling them and bringing them back.
Now, like half the Bay Area, I’ve cofounded an LLM startup, Dispatch AI. I’m half-reminded of Full Metal Jacket: “This is my startup. There are many like it, but this one is mine!” I do think, though, ours differs in several important ways:
It’s not a chatbot. (My chatbot skepticism, as the Brits say say, has form.)
My co-founder and I aren’t building tools for engineers. (Yeah, yeah, picks and shovels … but when everyone pivots to that, one should consider panning for gold.)
Having worked with hundreds of software clients, we understand the software industry — in particular, the 98% of it which is not the Big Five and/or elite Valley startups — better than most, and have a good idea what its managers need/want.
Specifically, I’m automating my old job. Which was both useful and lucrative! …Which makes the idea of “an AI software analyst,” which is what we’re building, seem awfully promising…
Let’s talk about software
Most software engineers think their teams are inefficient. I’ve seen hundreds of projects in action, and can confirm; most engineers are right. Meanwhile, software is crazy expensive! Engineers are very well-paid. Even a small team has a run rate of tens of thousands of dollars a month1. A large one can cost millions a year.
Of course we have collectively tried to address this unfortunate combination of high expense and low efficiency. We have tried to address it for thirty years. Agile development, Jira tickets, Kanban charts; continuous integration, end-to-end testing, static code analysis; stand-ups, scrums, story points; etcetera, and so forth, ad nauseum. They all help … well, most of them … and yet, to this day, most software teams remain awfully inefficient.
Much of this is fundamentally a communication issue. Understanding even an approximation of what’s really happening in the guts and at the many coal faces of a complex software project is hard. Understanding the ramifications? Even harder. Managing and directing such projects? Harder yet — especially when the managers and directors, very understandably, lack technical background and context themselves. Managing multiple concurrent projects2? Oof. Ugh. Sorry to hear it.
So we try to communicate. We write Jira or Linear tickets, and Notion or Confluence documents; then we add comments. We confer on Figma designs. We discuss pull requests on GitHub. We have calls and meetings and stand-ups on Zoom or Google Meet. We engage in long conversations on Slack or Teams. We send emails; we read emails. We collect error reports in Sentry, and customer feedback in Zendesk. .Almost every artifact of a software project that is not code is instead communication about code — spread across so many platforms that merely keeping track of those communications can be a full-time job in itself!
No wonder so much still falls through the cracks, inevitably, consuming time & money.
Let’s talk LLMs
You may expect me to now explain how we will be saved by LLMs writing code. Surprise! I, and my startup, are actually pretty agnostic about / orthogonal to that. I think LLM code generation is fantastic — it probably accounts for 25%+ of my own output nowadays! … but it doesn’t address that communication problem.
And while I’m incredibly bullish on AI, there are still significant near-term obstacles between LLMs and real-world adoption by people other than devs comfortable with their occasional wonkiness. Modern AI has very high output variance. Crafting a mindblowing demo can be surprisingly easy, but coercing it into consistently generating quality output from that chaotic flux called “real-world data”? That’s hard.
I should know—that’s what we’ve been building for the last few months. What we’ve done isn’t nearly as technically impressive as the autonomous LLM coding agent “Devin,” from Cognition, unveiled earlier this month … but it is instructive, and unsurprising, that a full year passed between the launch of GPT-4 and that of Devin, essentially a GPT-4 orchestration suite. Many of the finest technical minds on earth were working on LLM coding agents! And yet it took a whole year for someone to build one that actually works in real-world conditions … sometimes.
I do think Devin is, not the, but a way forward. LLMs are the new microprocessors. Building your own hardware was an effective path to success in the Intel/Motorola days, just as training your own models can be today. Some even quote Alan Kay: “People who are really serious about software should make their own hardware,” replacing the last phrase with “train their own models.” But looking at Cognition … and, analogously, Facebook, Google, Microsoft, Netflix … it’s clear you can accomplish much within the enormous possibility space opened by every new foundation model.
So, as I said, we’re building an AI software analyst. Oh yeah. And it’s not a chatbot.
Let’s talk money
What we do is very simple. It’s just like having an independent, objective analyst assess and report on your project, as I used to do myself. You don’t even need to ask or answer any questions. Our product, which we call The Dispatch, connects to your GitHub, Jira, Notion, Slack, Figma, etc.; assesses the code / documents / designs within; and sends you “dispatches,” a.k.a. reports. (At whatever cadence you like, but generally weekly makes the most sense.)
These reports are for managers and executives, not engineers. The key, of course, is that, like those I used to write myself, they can contain insights which help projects save time, and therefore lots of money. (Copious real-world examples available upon request.) But the other key is that they do so because software is still so complex, and our meshes of communication about software still so patchwork, that gems of insight and understanding still, always, inevitably, fall through the cracks.
Just as I was a “free safety” as CTO, keeping projects from going off the rails, the purpose of our AI analyst is to catch what would otherwise fall through those cracks, assess all the data, highlight the insights, and flag the risks.
(and so, when LLMs write a lot of the code … managers/execs will need this even more.)
One big difference, though; I cost a lot more than it will. Again, software teams are super expensive, tens of thousands of dollars a month, per team, even for small teams. …We plan to charge less than $25/week to help spend those $10,000s more efficiently.
The most economically interesting thing about LLMs is that they slash the cost of production in previously impervious fields like software. A year ago I would have charged you far more than $25, indeed far more than $250, to study your project and write you a report. It wouldn’t have made any sense to pay me to do that every single week. But as economists will tell you, when the cost of production drops by 10x or much more … this induces whole new categories of demand that didn’t previously make any sense. This is one of them.
Let’s talk about consensus reality
Listen to me. Here I am talking about efficiency and slashing the cost of production. I sound like a neoliberal World Bank consultant, I know. But I am actually not doing this to make a lot of money3.
Let’s go back to “Understanding even an approximation of what’s really happening in the guts and at the many coal faces of a complex software project is hard.” Which is really the fundamental problem here. All too often, managers, execs, devs, designers, QA, and customers are to the project as the fabled six blind men are to the elephant.
In other words what projects really struggle with is establishing a consensus reality. (In fact all organizations face this struggle … but it’s especially true of software.) My startup is trying to address that by crafting independent, objective, data-driven, verifiable, LLM-generated reports on the true state of the project, which in turn will help sync everyone’s “project reality” to something at least closer to a consensus.
…I don’t know if you’ve noticed, but there are other, much larger, consensus-reality problems in the world today. Groups who choose not just their own beliefs but their own facts, making it impossible to even establish a basis for mutual communication. I believe that independent, objective, data-driven, verifiable, LLM-generated reports — based on curated datasets that reflect reality well — can help with that consensus-reality problem too. (If you’re thinking “there’s a word for that”; yep.) That’s far enough away I won’t muse about it here just yet … but it is my genuine belief, and it’s what got me working on this in the first place.
(And let me quickly add that if all you know of LLMs is that they’re the creation of evil Big Tech companies and they hallucinate wildly when you ask them questions, then you presumably trust LLM-generated reports about as far as you can throw Salesforce Tower, and are baffled by my belief that hardheaded businesspeople will find them credible. Briefly, their actual use is … quite different from what you have heard. In particular, we don’t ask them questions! Instead we do “in-context learning,” i.e. use them as “anything-from-anything machines,” to transform data into reports.)
It’s genuinely thrilling to be excited about technology again, and to believe, for the first time in a decade, that living in interesting times can be a blessing not a curse. LLMs are incredible (if sometimes incredibly frustrating…) and, honestly, The Dispatch is already far better than I expected it to be. I can’t wait to see what the AI frontier brings next.
Let’s talk
Needless to say, if you’re interested in what we’re building, let’s talk! You can reach me professionally at jon@thedispatch.ai, and (sigh) on LinkedIn.
Even offshore engineers; those generally cost $35+/hour and up, so a team of three runs $18,000/month without even counting management, QA, etc.
A long but worthwhile quote from one of our advisors:
”At [household name tech company], at the end of every week, the engineering managers wrote up a summary of what your team did and sent it to our VP of engineering. Then he collated them and sent that out to all of us. So we actually had very good visibility across the company. And it was super useful, for me, but also for up the chain. but it kind of sucked to do, it always felt like it took half my Friday. Automating that would have been amazing.
Then at [subsequent company], we tried to institute it and just never happened. But we struggled with that because we just had very little visibility into what was going on across the company. I think that if like a half pager for each team got sent out every Friday, people would eat that up. I think that might be really valuable. I don't know how you quantify the need and the value. But I think people underestimate, especially at the leadership level, how little visibility a lot of people have into what's going on at a medium to large sized organization. Actually having that broad knowledge is absolutely super important to get alignment.”
I will concede however that the fact that this approach seems highly lucrative is a not undesirable side effect.