Intro: "Chains of Thoughts" AI Blog

Newsletter on AI Tech, Consumer Trends, and Business

Image generated by DALL·E-3

All the attention you may or may not need, focused on the next big disruption:

"It's a brave new world" says an enterprise business leader - let's call her Jill Donaghy - one day as she stares out the 48th floor of skyscraper, overlooking the majestic New York City skyline.  The sun is setting, casting a golden glow over the chaos below, but Jill's mind is buzzing with a question every executive seems to be asking these days: is your business leading - or lagging -- with respect, to the upcoming technology-driven supercycle. That thing that’s been described as a generational disruption, that’s been breathlessly mentioned in what seems like every headline across every newspaper and media report these days. Where does Jill - and every enterprise leader like her - stand when it comes to her business and this new “Intern Revolution?”

Not that the idea of businesses hiring interns was even new - businesses had been doing this (at a smaller scale) for decades with little notice. But at some point, within the last few years, a couple key things happened. First, interns got a little smarter (experts speculated - perhaps it had to do with the rise of EduTok and shifts in viewing habits that the younger generation picked up during Covid). And second, the emergence of a noticeable compounding effect (or Power Law it was called) where you could predictably achieve better results with more interns staffed on a project, a discovery that unexpectedly resulted in a breakthrough.

So now, academics and experts all over were saying that businesses could drive efficiency gains in the range of 20-30% (staggering labor productivity gains compared to previous technologies) if they embarked on transformations harnessing the power of interns and embraced this new paradigm.

Capabilities may vary…

But was that really the case, thought Jill? Sure intern-fueled gains drove some task improvements, and even some department's productivity metrics (coding for sure). But as her CFO had asked her in a recent headcount planning session, Jill wasn't ready to commit to hiring 30% fewer people (or sign up for goals 30% higher).

The problem was the expert's research was mainly based on academic benchmarks - the kind where interns scored more highly on the GMAT or the LSAT. Not exactly the measure of the real-world.  When the interns were exposed to actual business tasks, it was a mixed bag. Sometimes they would nail it - "oh wow, look at that beautiful summary of a company's earnings call," with sources referenced and pointing out relevant connections to topics from previous calls that seemed to represent the intuition and pattern recognition of some of the most tenured analysts on staff. But other times, you get back something, involving basic math, that was just completely wrong. When mapping intern capabilities for each of the tasks making up a regular employee's normal workflow, instead of usual performance bands for employees (which varied typically because of experience or tenure), the graph for interns showed much more of a “jagged edge” line chart, with spikes in some areas, and drops in others, and huge inconsistencies - especially in tasks involving lots of different steps.

Source: Ethan Mollick, Centaurs and Cyborgs on the Jagged Frontier (with modifications)

Maybe Trust is all you need… a closer look, broken down into three buckets (Steerability, Accuracy and Transparency):

“Maybe,” thinks Jill, “it’s just about giving them the right instructions.” Enter the new discipline of Intern Instruction Prompting (headlines are saying this is the next “must have” job skill). How it works: you break out every intern task into a series of step-by-step instructions, with each instruction including relevant context and examples, along with some idiosyncratic things (like telling the intern to take on a certain persona while answering or assigning a number from 1-10 in terms of how creative you want their response to be).

The problem is it takes forever. And when you want to change just one little thing, you end up in many cases having to rewrite all those steps to get something that you want. Worse - as the schools moved from regular semesters to rolling graduation dates (something with the post EduTok curriculum), each batch of interns needs these instructions in slightly different ways, meaning one set of instructions can suddenly cease working with the latest batch of (supposedly better) interns. Maintaining just simple consistency and getting the interns to do what you wanted was exhausting.

And then there’s the whole “black box” issue. Sometimes, Jill would ask an intern to explain how they arrived at a conclusion—say, how they figured the profit margin for that new product line—and she’d get a blank stare. Some answers came out right, others came out wrong, but there was no transparency in terms of how the interns would reason to arrive at an answer, just the input and the output which would have to be laboriously checked with no backups to look at.

Between the clunkiness of instruction programming, transparency around outputs, and need to check everything, Jill felt suspicious that interns were the solution they were promised to be.

What about the Experts?

Next, Jill turned to one of the high-powered consultancies, McBain, who usually advised executives like her. Surely they could pinpoint precisely which tasks the interns were best suited to tackle in her business. Initially, McBain seemed promising—they were great at talking in broad terms about the value of the Intern Revolution, presenting well-structured frameworks, and articulating the strategic opportunities for enterprises. But when it came to the core challenge—understanding the current jagged edge of intern performance, which required both a deep knowledge of the rapidly evolving space and hands-on deployment expertise—they fell short.

McBain's advice often remained theoretical, and their knowledge couldn't keep up with the speed of intern capabilities evolving in real time. On the implementation front, Jill needed help with the labor-intensive process of Intern Instruction Prompting, but McBain told her that’s not something they do. Implementation was Jill's greatest challenge— she needed experts who could manage intern programs effectively - and they were in short-supply, even for large companies like the one she represented. But she found the same dearth of experienced talent at places like McBain as well.

Deploying intern programs requires both solid theory and effective practice. McBain's recommendations were heavily theory-based, and as a result, many were impractical. Worse still, they often missed the mark in prioritizing use cases where today's intern capabilities were best suited to tackle.

The disconnect from extracting measurable value through intern deployments, and the stories in the press seemed more pronounced than ever.

Unlocking Value: a better Customer Experience? or a Compliance and Legal nightmare?

Jill’s board however was seeing those headlines. A link in Jill’s inbox one morning had a CNBC clip from the CEO of one competitor neobank, where he spoke about his Intern-powered business transformation. The results were impressive: on customer service alone, he was able to replace 700 full time employees, improve satisfaction and NPS scores, and add millions to his bottom line.

Instead of focusing on internal-focused tasks, why not focus on use cases with customers? Interns could replace things like dreaded phone trees and customers needing to hunt through FAQs online, and move from answering questions to solving billing issues. Plus these were tasks that matched to core strengths of the interns - they were naturally social, and customers found communicating with them was far easier. And as the competitor CEO pointed out, intern-powered solutions could replace real cost centers in ways that could immediately prove out value.

The results of the pilot project were…mixed. Sometimes the interns delivered a brilliant, personalized answer that made the customer feel heard. And other times—well, there were some truly terrible moments. Jill found herself starting at screenshots of conversations with not just wrong answers, but in some instances with customers incorrectly promised discounts, or awarded refunds greater than the original order amounts. There were some examples where Jill was convinced people were just getting the interns to say embarrassing things, which they could later post online.

Here too the proposed solutions were complex. Implementing a code of conduct telling interns, for example, “don’t do that’ was about as effective as commands to “be more accurate” with answers (ie: not very effective). But layered approaches worked, involved things like incentives (goals, using reward functions), and controls (similar to accounting controls for workers - ensuring nobody has unrestricted access to the cash register, authenticating customer identity and transaction information for requests involving higher risk).

But nothing was 100% and the risks for the external use case and use cases involving interactions with systems for doing (billing, payment, order logistics/delivery) were certainly greater. What happens when your interns are saying jibberish to your most important customers or prospects? And for parts of Jill’s business that involved compliance or regulatory requirements (she was part of a regulated bank, some parts of her business fell under regulations like HIPPA relating to healthcare), there were real consequences if things went wrong.

Sure, it was easy for a diversified VC with multiple companies to exhort adoption of an intern-first approach to everything. But for Jill any one of these risks - business risk, PR risk, liability risks, in addition to the regulatory risks mentioned before — could be career ending if she didn’t think this through.

Goal of this Blog

The real world might not be on the cusp of an Intern Revolution, but AI comes close. The technology can be staggeringly advanced, and yet incredibly unpredictable to build on. The potential for value creation is staggering. But understanding how the technology will spread - particularly for businesses doing other things - can seem like a full-time job. Whether that’s keeping up with the pace of innovation, understanding the jagged edge of capabilities, recognizing blockers to adoption and drivers of trust for both consumers and enterprises, or having fluency with technical patterns and architectures that drive success, navigating this Brave New World isn’t easy.

For readers like Jill, the protagonist earlier in this post, the goal of this blog is to make trends in AI accessible to a larger audience, with a focus around how businesses can benefit and draw value. And for readers already in the AI/ML space, the goal is to invite other takes or opinions, since the solutions for many of these problems are still being developed. I love a good debate, and hopefully others find it helpful.

Some future posts will explore topics like the paradigm shift around reasoning, technologies that are enabling the next generation of smaller models, features and improvements in user experience driving growth in the app layer, and emerging AI-native business models.

Thanks for reading, I hope you enjoy.