Changelog

Building the Foundation - Update (03/07/25)

Making Podcast Transcripts Readable - Update (03/10/25)

Transcript Generator Project

GitHub Repository

As a followup to my previous update regarding working with llama3:70b and the improved llama3.3:70 models, I’m still running into issues with fine tuning them for my specific use case.

Endlessly Tweaking Local LLMs

There are an endless number of local LLM models to download and experiment with, and I could spend countless weeks fiddling around with them to try and get exactly what I need. While it was fun using llama3.3:70b (both the base model and the “instruct” tuned model) and seeing my computer using its maximum resources, it quickly became apparent that this approach wasn’t sustainable.

Tweaking parameters, running the program, tweaking some more, running again… it was just taking up too much time. Depending on the size of the text file I was feeding it, run times could reach up to 10 minutes - not very practical when I was making so many incremental changes to the program. I also realized that since I want my program to be accessible through a web server, it would need to run on a system where GPU availability wouldn’t be guaranteed.

This is where I had to make a practical decision.

After some research, I decided to try mistral:7b-instruct - a model from the company Mistral AI. Compared to the variations of llama3.3, the Mistral model requires much less VRAM and seems to be configured to run much more reasonably on CPU, which would potentially take care of my issue with running the program on a web server.

The results? Mixed at best. While mistral:7b-instruct did run quicker, I was still experiencing issues where the model was modifying words for “clarification” even after being explicitly instructed to preserve every word of the original text. The model was also not properly formatting the text to make it more readable - it was still splitting sentences into new lines halfway through.

Here’s what my instructions looked like - the text can be scrolled left to right:

FORMAT TASK: This is a raw transcript. Your ONLY task is to format it into paragraphs.

CRITICAL INSTRUCTIONS:
- YOU MUST PRESERVE EVERY SINGLE WORD FROM THE ORIGINAL TEXT
- DO NOT remove any words, phrases, or content
- DO NOT add any new words, phrases, or content
- DO NOT paraphrase or summarize
- DO NOT fix grammar, typos, or correct speaking errors
- DO NOT combine sentences in ways that change any words
- ONLY group the existing text into paragraphs
- Keep exact sentence structure as in the original
- Use the exact same words in the exact same order
- Only add paragraph breaks where appropriate

Think these instructions are clear enough? I did too. And yet…

The Gap Between Instructions and Reality

Here’s an example using a transcript of a video from the YouTube channel ‘The Ash Files’ - original output:

Which is how I found myself here, actively pursuing the impossible. A soft,
quiet, ambitious
life. Mornings are slow. They have to be. Not because I'm a wellness influencer
with a five-step
journaling ritual, but because I have the circadian rhythm of a person whose
body clock and actual
clock have never once agreed on anything. There's no morning rush, no
immediately checking emails,
no diving into work before I remember that I'm an actual person with needs.
Instead, there's my dog,
who gets the first part of my day. There is a walk involved, quiet, still,
occasionally interrupted
by my dog's deep need to investigate a single leaf for an unsettling amount of
time before the
notifications start. There's caffeine in some form, whatever requires the
least effort to acquire,
and no matter how urgent everything feels, the world will still be there when
I get to it.
It's a delicate balance, making space for the kind of stillness that keeps you
sane without falling
into full inertia. I've been guilty of both, of doing too much and of doing
absolutely nothing,
of measuring my worth by productivity, then swinging too far the other way,
convincing
myself that watching six hours of prestige reality television was the same as
self-care.
It wasn't, but I did develop a concerning level of emotional attachment to
people I will never meet...

And this is a segment of the output from Mistral (more or less very similar to any local model that I’ve tested):

Which is how I found myself here, actively pursuing the impossible. A soft,
quiet, ambitious life.

   Mornings are slow. They have to be. Not because I'm a wellness influencer
with a five-step journaling ritual, but because I have the circadian rhythm of
a person whose body clock and actual clock have never once agreed on anything.
There's no morning rush, no immediately checking emails, no diving into work
before I remember that I'm an actual person with needs. Instead, there's my
dog, who gets the first part of my day. There is a walk involved, quiet,
still, occasionally interrupted by my dog's deep need to investigate a single
leaf for an unsettling amount of time before the notifications start.

There's caffeine in some form, whatever requires the least effort to acquire,
   and no matter how urgent everything feels, the world will still be there
when I get to it. It's a delicate balance, making space for the kind of
stillness that keeps you sane without falling
   into full inertia. I've been guilty of both, of doing too much and of doing
absolutely nothing,
   of measuring my worth by productivity, then swinging too far the other way,
convincing
   myself that watching six hours of prestige reality television was the same
as self-care. It wasn't, but I did develop a concerning level of emotional
attachment to people I will never meet...

You can see that while the sentences are somewhat grouped into paragraphs, there are still some odd formatting decisions with random line breaks in the middle of sentences and indentation in some spots.

The problem goes even deeper. Take a look at these logs from a podcast transcription, which compare the model’s output with the original text:

Processing chunk 1/25...
Processing chunk 2/25...
Processing chunk 3/25...
Word count mismatch: original=182, formatted=181
LLM changed content. Falling back to rule-based for this chunk.
Processing chunk 4/25...
Processing chunk 5/25...
Processing chunk 6/25...
Processing chunk 7/25...
Word count mismatch: original=184, formatted=179
LLM changed content. Falling back to rule-based for this chunk.
Processing chunk 8/25...
Word count mismatch: original=167, formatted=164
LLM changed content. Falling back to rule-based for this chunk.
Processing chunk 9/25...
Word mismatch at position 70: original='and', formatted='so'
LLM changed content. Falling back to rule-based for this chunk.

The log continues with more examples of the model changing word counts or substituting words (like changing “and” to “so” or “the”). In total, out of 25 chunks of text, the LLM modified content in at least 12 of them despite the most explicit possible instructions not to do so.

Why would these models be so resistant to following seemingly straightforward instructions?

My theory is that these local models are designed to improve text rather than merely format it. Unlike online models like ChatGPT, which can instantly access vast general knowledge about podcast hosts’ speaking styles and conversational patterns, local LLMs have to make decisions based solely on their training data. Without this broader context to understand that verbal quirks and grammatical inconsistencies are natural parts of spoken conversation, my local model was so convinced that these changes were beneficial that it made them despite explicit contrary instructions. It’s like telling someone “just hold this for me, don’t change anything” and they can’t help but straighten it out or adjust it “just a little bit,” because in their limited understanding, they believe they’re being helpful.

Now, improving grammar doesn’t seem like a necessarily bad thing to do, but when you’re working with transcripts, preserving the original text is key. If I want to read a podcast transcript while listening, I personally want to preserve:

  1. Casual speech patterns

  2. Verbal tics and word repetitions that reflect the speaker’s actual words

  3. Grammar inconsistencies that might be part of someone’s unique speaking style

Trying a Rule-Based Approach

Remember those log entries that mentioned “Falling back to rule-based for this chunk”? That was my backup plan.

After running local models over and over with more or less similar results, I decided to try a more deterministic approach - rule-based formatting. This approach formats transcripts using fixed patterns. It looks for specific clues in the text to decide where paragraphs should start and end, while keeping all the original words exactly as they are.

Here’s how it works:

  1. Breaking the text down : First, the system splits the transcript into individual lines.

  2. Cleaning up : It removes extra spaces and normalizes line endings, but doesn’t change any words.

  3. Looking for paragraph signals : As it reads each line, it checks for:

  4. Building paragraphs : When it finds one of these signals, it ends the current paragraph and starts a new one.

  5. Sentence detection : The program knows that “Dr. Smith” doesn’t end a sentence, but “Hello there.” does.

  6. Joining it back together : Finally, it connects the lines within each paragraph with spaces and separates paragraphs with blank lines.

Let’s look at the results. Using the same raw transcript from earlier:

Which is how I found myself here, actively pursuing the impossible. A soft,
quiet, ambitious
life. Mornings are slow. They have to be. Not because I'm a wellness influencer
with a five-step
journaling ritual, but because I have the circadian rhythm of a person whose
body clock and actual
clock have never once agreed on anything. There's no morning rush, no
immediately checking emails,
no diving into work before I remember that I'm an actual person with needs.
Instead, there's my dog,
who gets the first part of my day. There is a walk involved, quiet, still,
occasionally interrupted
by my dog's deep need to investigate a single leaf for an unsettling amount of
time before the
notifications start. There's caffeine in some form, whatever requires the
least effort to acquire,
and no matter how urgent everything feels, the world will still be there when
I get to it.
It's a delicate balance, making space for the kind of stillness that keeps you
sane without falling
into full inertia. I've been guilty of both, of doing too much and of doing
absolutely nothing,
of measuring my worth by productivity, then swinging too far the other way,
convincing
myself that watching six hours of prestige reality television was the same as
self-care.
It wasn't, but I did develop a concerning level of emotional attachment to
people I will never meet...

The result of using the rule-based approach:

Which is how I found myself here, actively pursuing the impossible. A soft,
quiet, ambitious life. Mornings are slow. They have to be. Not because I'm a
wellness influencer with a five-step journaling ritual, but because I have the
circadian rhythm of a person whose body clock and actual clock have never once
agreed on anything. There's no morning rush, no immediately checking emails,
no diving into work before I remember that I'm an actual person with needs.
Instead, there's my dog, who gets the first part of my day. There is a walk
involved, quiet, still, occasionally interrupted by my dog's deep need to
investigate a single leaf for an unsettling amount of time before the
notifications start. There's caffeine in some form, whatever requires the
least effort to acquire, and no matter how urgent everything feels, the world
will still be there when I get to it.

It's a delicate balance, making space for the kind of stillness that keeps you
sane without falling into full inertia. I've been guilty of both, of doing too
much and of doing absolutely nothing, of measuring my worth by productivity,
then swinging too far the other way, convincing myself that watching six hours
of prestige reality television was the same as self-care.

It wasn't, but I did develop a concerning level of emotional attachment to
people I will never meet...

No random indentation. No arbitrary line breaks. Just clean, readable paragraphs that preserve every single word of the original text.

Not Necessarily Giving Up, But Moving Forward

For what seems like a relatively straightforward formatting task, this level of complexity feels excessive. I’m caught between knowing the problem is solvable with advanced AI and the practical realities of local implementation.

I could always try to train a model on a specific transcript format, but if I want to use it for a variety of videos or podcasts, the feasibility becomes questionable. I’m sure there’s a way to achieve what I want using a local LLM - I just haven’t found it yet.

But here’s the thing: I want to continue moving forward with this project. The next major step is figuring out how to connect this program to a front end so that I can use it through a web browser.

There will always be improvements that can be made, and I can always come back and continue to fiddle with the formatting logic later.

Sometimes you have to recognize when “perfect is the enemy of good” and take the next step forward.