Using GenAI to Evaluate TikTok Trends

What happens when AI fact-checks TikTok advice? Using GenAI, I unraveled the truth behind viral tips, separating wisdom from pseudoscience.

Over 40% of Americans are using TikTok as a search engine. With about 10% of Gen Z relying on it more than Google, TikTok isn’t just a platform for dances and memes, it’s where people go to find advice. When TikTok Creator Search Insights revealed a content gap for “Positive Thinking Tips,” I decided to see what was currently out there. I downloaded over 500 TikTok videos with the hashtags “positivemindset”and using some cutting-edge AI tools, I set out to unravel the chaos. What I found was equal parts enlightening, hilarious, and deeply concerning.

The Experiment Begins

The first step? Collect the data and using the app I wrote I was able to convert TikTok videos into knowledgebases. The result: pages and pages of bite-sized wisdom, ranging from genuinely uplifting advice to utterly bizarre claims.

Next, I fed the transcriptions into GPT-4o-mini to categorize the content for me. This saved me hours of manual work, organizing the tips into neat categories like “Affirmations,” “Daily Habits,” and “Mindset Shifts.” To make sense of the dataset, I used GenAI to write a Google Apps Script. The script looped through my spreadsheet and converted the content into a structured Google Doc. It was fascinating, until I noticed something: much of the advice was repetitive. My “organized” content was now 97 pages long, bloated with overlapping ideas. I needed something with a bigger context window so I could consolidate the categories and repetitive advice. Using this prompt, I turned to Google Gemini Advanced which has a 1 million token context window.

I have the following list, but I want to consolidate categories so there are fewer. Every category should have at least three bullets; if not, merge it with another. The total number of bullets should stay the same, and I want the full content for each bullet.

Gemini handled the task seamlessly, but as I combed through the refined categories, something struck me, some of the advice wasn’t just repetitive, it was outright wrong.

The Wild, Wild World of TikTok Advice

Here are just a few of the “tips” I encountered:

• “Being forgetful is actually a sign of high intelligence.”
• “Good liars are also the best at detecting lies.”
• “It takes 60 centimeters of body fat to stop a bullet.”
• “The presence of taste buds in the anus explains the burning sensation after eating spicy food.”
• “Speaking affirmations consistently for 21 days can rewrite your DNA.”

Some of these are laughable. Others, like the claim about body fat and bullets, could be dangerously misleading if taken seriously. It was clear that I needed to evaluate every piece of advice for accuracy and usefulness.

The Power of AI Evaluation

To tackle this, I turned back to Gemini. This time, I asked it to review each piece of advice for accuracy and overall value. The results were illuminating:

• Many of the tips were indeed harmless and even helpful. Gemini affirmed the value of classics like “Believe in yourself,” “Embrace your imperfections,” and “Focus on the positive.” It was reassuring to see that amidst the noise, there were nuggets of genuine wisdom.
• Some advice, while well-intentioned, was overly simplistic or lacked nuance. For example, “Don’t be afraid to be yourself” is generally good advice, but Gemini pointed out that social context and awareness are also important.
• A portion of the advice was based on unfounded claims, pseudoscience or just bias. For example, “When babies are drawn to you, it can signify an innate charm or warmth that is appealing, indicating a positive personal trait.” This is a classic example of confirmation bias. People tend to remember and interpret events in a way that confirms their existing beliefs. If you believe you’re charming, you’re more likely to notice when babies seem drawn to you and attribute it to your personality, while ignoring instances where babies might react neutrally or negatively.

AI didn’t just streamline the evaluation, it highlighted the importance of critical thinking in an era of information overload. In essence, this approach transforms LLMs into more than just tools, they become co-educators, helping us navigate and evaluate the overwhelming amount of information in our digital world. By showing how AI can consolidate, critique, and contextualize TikTok advice, we offer a blueprint for anyone looking to think critically about what they consume online. It’s a skill that’s increasingly vital, not just for TikTok users but for professionals, creators, and even engineers like myself who work on developer tools aimed at boosting productivity and insights.

And for those of you wanting to see the full data set of positive thinking tips, Happy Reading!