Changelog

Building the Foundation - Update (03/07/25)

Transcript Generator Project

GitHub Repository

Important Notice - I’m currently testing llama 3.3 :70b after realizing that Meta released this huge update back in December that brings the context window up to 128k tokens (if the terms “context window” and “tokens” make you shy away, don’t worry, just keep reading and it’ll hopefully all make sense).

As I’ve been digging deeper into my transcript generator project, I’ve hit my first major roadblock. I mentioned in my last update that I was exploring local LLMs for formatting, and like I suspected… it turns out that’s a lot trickier than I initially thought.

The Starting Point

I’ve got a raw transcript from the podcast “Under the Radar” with Marco Arment and David Smith. Here’s an excerpt - the text can be scrolled left to right:

Welcome to Under the Radar, a show about independent iOS app development.
I'm Marco Arment. And I'm David Smith. Under the Radar is usually not longer than 30 minutes,
so let's get started. So everyone, welcome to our podcast.
Please stick
around until the very end. But first, we need your permission for something.
We're talking about onboarding today. That was a terrible
joke, I'm sorry.

Not exactly the easiest to read, right? A bit jarring? What I want is something properly formatted with speaker names in bold, well-structured paragraphs, and generally improved readability - while keeping all the original content intact.

The URL analysis and transcript downloading parts of the program were pretty straightforward - I’ve been writing Python for a while now, so that was within my comfort zone. But working with local LLMs like Meta’s Llama 3? This is an entirely new territory for me.

Why Local LLMs?

Why am I bothering with local LLMs when services like ChatGPT and Claude exist? There are a few key reasons:

  1. Cost efficiency : I plan to process a lot of transcripts. Using API-based services would rack up costs quickly (a few cents per transcript adds up when you’re doing dozens or hundreds - especially during testing).

  2. Learning opportunity : A major goal of this project is to deepen my understanding of how LLMs work. Running models locally gives me more direct experience with their capabilities and limitations.

  3. Independence : I want a solution that works reliably without depending on external services that might change their pricing or APIs.

Yes, I could fall back on an existing local application like Vibe if needed, but that defeats the purpose of this project. I’m not just trying to get formatted transcripts - I’m trying to build knowledge and skills that I can apply to future projects and in my career.

My Philosophy on Using AI Assistants

Before I dive into the technical challenges, a quick note on my approach. I’m a firm believer in learning things properly rather than using AI as a crutch. For this project, I’m handling all the coding and decision-making myself.

Where ChatGPT and Claude come in handy is as a quick resource for understanding new concepts or getting unstuck on specific issues. Instead of spending hours searching through documentation or forum posts, I can simply ask specific questions about things like “How do context windows work in LLMs?” or “What’s the difference between tokens and words?”

It’s like having a knowledgeable colleague down the hall that I, at any point, can tap on the shoulder and ask questions (luckily this one doesn’t get annoyed).

Parameters: What They Are and Why They Matter

Before diving into my attempts at solving this, let me explain a bit about what “parameters” actually mean in these models.

Parameters are essentially the weights and biases in the neural network that makes up a Large Language Model. Think of them as the “knowledge” or “skills” the model has acquired during training. More parameters generally means:

  • Better understanding of language nuances

  • More factual knowledge

  • Improved reasoning capabilities

  • Better ability to follow complex instructions

The difference between llama3:8b and llama3:70b is massive - we’re talking about a model with 8 billion parameters versus one with 70 billion parameters.

To put this in perspective:

  • GPT-3, which amazed everyone when it came out in 2020, had 175 billion parameters yet llama3:70b matches, and in some cases, exceeds its capabilities with less than half the parameter count

  • Google’s original BERT model, announced in 2018 and changed the way we think about Natural Language Processing - had only 340 million parameters - llama3:70b is over 200 times larger

Llama3:70b is the kind of model that can write coherent essays, solve complex reasoning problems, and follow intricate instructions. That’s why I was so confident that upgrading from llama3:8b would solve my formatting problems. The jump in capabilities should have been enormous.

First Try: Ollama with llama3:8b

My first attempt was simple:

  1. Feed the transcript to Ollama (which allows you to locally run LLMs) that would be running the llama3:8b model

  2. Ask it to format everything properly

  3. Save whatever comes back

I thought my instructions were pretty clear:

Format speaker names in bold followed by a colon.
Create proper paragraphs by grouping related thoughts.
Start a new paragraph when a speaker changes.
Preserve ALL original content - don't leave anything out.

But instead of a nicely formatted transcript, I got what amounted to a creative writing exercise. The model:

  • Cut out most of the actual conversation

  • Renamed Marco and David to “Host”

  • Made up a completely different structure

  • Kept around 20% of the original content

Not exactly what I was hoping for, but somewhat understandable given that llama3:8b is a smaller model. I figured this was a case where the model just wasn’t powerful enough to handle the “complex instructions” required.

Second Try: Going Bigger with llama3:70b

I thought to myself, “More parameters should fix this”. So I fired up llama3:70b (which is a massive 40GB download vs llama3:8b at a measly 5GB) and appended the following explicit instructions:

IMPORTANT: You MUST preserve ALL original content. Do NOT summarize or create new content.
CRITICAL: Include the ENTIRE transcript - do not truncate or summarize any portion.

This is where things got interesting. Despite having nearly 9x the parameters - which should translate to significantly better instruction-following and comprehension - I still got basically the same result: a heavily summarized, partially fictional version of the transcript.

The results were slightly better in terms of coherence and the summaries were more accurate, but the main problem remained. The model still couldn’t follow the instruction to preserve all content.

Why would a model with 70 billion parameters, which should be way more capable, still fail at what seemed like a straightforward task?

The Real Problem: Context Windows

This is where I needed to understand a crucial concept of LLMs I hadn’t encountered before. After some back and forth with Claude about why this might be happening, I learned about “context windows”.

A context window is basically how much text an LLM can “see” at once - how much it can hold in its working memory. You can think of it as trying to view a 10-page document through a window that only shows 8 pages at once - the remaining 2 pages aren’t just forgotten, they’re completely inaccessible to the model. It can only reason about and reference what’s currently visible in its context window, with no ability to ‘remember’ what was pushed out.

The surprising thing I learned is that both Llama 3 models - despite the huge difference in parameter count - have the exact same context window size:

  • llama3:8b: ~8,000 tokens

  • llama3:70b: also ~8,000 tokens

This was the “blindfold being lifted” moment. Parameters and context windows are completely different things:

  • Parameters determine how smart the model is

  • Context windows determine how much information it can work with at once

Tokens are the basic units of text that AI models process - not just characters or words, but segments of text that may represent partial words, whole words, or punctuation. Different models have different context window sizes meaning that some may only process a few thousand tokens while others can handle tens of thousands, but all have some finite limit that constrains what they can “remember”. Anything beyond that limit is essentially forgotten or inaccessible to the model when generating its next response. This limitation is one reason why hallucinations can occur, the model essentially makes educated guesses based on its training rather than the specific information that was provided but has since been pushed out of the available context.

On average, depending on the tokenization method and content itself, a single token is about 1.3 words. My transcript, being 6,000 words, converts to approximately 7,800 tokens - which is already pushing the 8,000 token limit. Hold on, 7,800 is still less then 8,000 so what’s the issue? Well, the model’s context window isn’t just my transcript. It also includes all the system instructions that tell the model how to behave, previous messages in our conversation, and various formatting markers (essentially how the model internally represents the text). These hidden elements can easily consume over 1,000 tokens, pushing the total well beyond the 8,000 token limit.

So the model wasn’t being stubborn - it has a fundamental structural limitation that prevents it from accessing any information beyond its context window size.

The Challenges Keep Coming

As I’ve been exploring this problem, I’ve realized there are several layers to this issue:

  1. The Memory Problem : Local LLMs have tiny memory compared to something like ChatGPT, which can handle up to 128,000 tokens (this will hopefully no longer be an issue as I discovered llama3.3:70b which does in fact have 128,000 tokens).

  2. Quality Issues : Local models, even when working properly, can exhibit formatting inconsistencies, miss speaker identifications, and struggle with content preservation.

  3. Resource Consumption : Running the bigger models needs serious hardware - we’re talking 64GB+ RAM for optimal performance with the llama3:70b model. I’m using an M4 Max MacBook Pro with 48GB of unified memory, which provides faster data access than traditional RAM, but even with this advantage, compile times still reach around 4 to 5 minutes compared to the under 30 second returns when using cloud services like ChatGPT.

  4. Technical Complexities : Solutions must accurately identify different speakers while maintaining the conversation’s flow and preserving critical context and information.

What I’m Trying Next

I haven’t cracked this yet, but I’ve got a few ideas I’m working on:

  1. Chunking : Breaking the transcript into smaller pieces with some overlap between them.

  2. Verification : Adding checks to make sure no content gets lost.

  3. Post-Processing : Cleaning up formatting inconsistencies after the LLM finishes compiling.

  4. Model Rotation : Exploring other local models that might handle this better.

  5. **Model Quantization: ** Using techniques that reduce the size of LLM models while maintaining most of their performance, allowing them to handle longer texts with the same amount of computer memory.

This project has been a great reminder that AI / LLMs aren’t magic - they have very real limitations that can require more creative workarounds.

What I’ve Learned So Far

Even though I haven’t solved this completely, I’ve picked up some meaningful insights:

  1. Bigger doesn’t mean better : A 70B parameter model doesn’t help if both models have the same memory constraints.

  2. Words vs. tokens : Always estimate how many tokens your text will use, not just word count.

  3. Local AI has tradeoffs : Running things locally gives you control and privacy, but comes with serious technical hurdles.

  4. Learning approach matters : By treating AI as a knowledge resource rather than delegating the entire problem to it, it gives me more confidence in my problem solving skills.

Where This Is Going

I’m still chipping away at this problem. The URL analysis and transcript downloading parts of my script are solid, so I’m not starting from scratch. It’s just this formatting piece that’s proving tricky. I also need to keep in mind that the goal isn’t just to make it work for a single source, but to create a solution that handles transcripts reliably across different media types while also accommodating various lengths and structural patterns.

LLMs can have incredible capabilities even with their limitations - the key is being intentional about how you present problems to them.