🚀 Sign up for Bright Data right now: https://brdta.com/cfe
Automatically find and track topics you care about across Reddit posts. From camping to the latest in AI news, this course will show you how to build a powerful and resilient system in Python.
The goal is of this course is to help you develop the skills you need to build a resilient data extraction platform using only a handful of tools and the latest in LLMs from Google. In addition to the new skills you’ll learn, you’ll also have rich data to help you better learn from what real people are experiencing all around the world.
Topics:
✅ Easily download the latest Reddit conversations around topics you care about
✅ Ai-Powered Google search to extract relevant Reddit Communities (aka SERP)
✅ Build & ingest data through public webhooks (notifications that work software-to-software or app-to-app)
✅ Rapid prototype data scraping/extracting with Python & Jupyter Notebooks
✅ Use Gemini to run your Python functions based on plain english (aka Tool Calling)
✅ Store extracted data through the Django ORM and PostgreSQL
✅ Strict & structured data outputs for LLMs with Pydantic
✅ Fault-tolerant data downloads using background tasks & webhooks
✅ Configure serverless and serverfull worker managers (django-qstash & celery)
✅ and much more
Resourses
– My github: https://cfe.sh/github
– Project Code Repo https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent
– My Bright Data link – https://brdta.com/cfe (means more sign ups, more free courses)
– Django QStash repo & docs https://djangoqstash.com
– Django with Celery & Redit Blog Post: https://www.codingforentrepreneurs.com/blog/celery-redis-django
Stack:
‣ Python
‣ Jupyter (rapid prototyping)
‣ Django (web app & automation coordinator)
‣ Postgres (database)
‣ Redis (caching & queues)
‣ Celery (background tasks)
‣ Django QStash (serverless background tasks)
‣ Bright Data Search Engine AI (SERP)
‣ Bright Data Crawl API (extract Reddit posts)
‣ LangChain (integration to Google Gemini LLM)
‣ LangGraph (easily unlock Tool Calling)
‣ Cloudflare Tunnels (public domain to your project to accept webhooks)
Chapters
00:00:00 Welcome
00:03:46 Demo
00:12:03 Using Search Engine Results
00:14:16 Setup your Python Project
00:20:36 Load API Keys with Dotenv Files
00:24:26 Intro to LangChain
00:26:19 Bright Data Serp API with Python & LangChain
00:38:01 Strip Notebook Outputs for Security with pre-commit
00:42:56 Setup Google Gemini Models with LangChain
00:52:43 LLM with Structured Output
00:59:58 LLM Tool Calling The Hard Way
01:08:19 Tool Calling with LangGraph
01:23:41 Search & Format Reddit Communities via LLM and Bright Data
01:29:38 Scrape Reddit with the Bright Data Crawl API
01:41:58 Get Crawl API Snapshot Progress
01:47:00 Download Data from the Crawl API
01:54:53 Automating Data Pulls for Users
01:58:39 Install & Start the Django Project
02:02:31 Combine Django with Jupyter
02:05:23 Implement Postgres Database with Django
02:15:19 Setup Redis for Django & Caching
02:22:36 Getting Started with Celery & Django
02:33:51 Webhooks & Cloudflare Tunnels
02:36:47 Setup Cloudflare Tunnel with a Custom Domain
02:45:24 Django Qstash for Webhook-based Background Tasks
02:52:55 Bright Data to Django Model Part 1
03:02:16 Bright Data to Django Model Part 2
03:09:38 Store Bright Data Snapshots
03:17:38 Helper Functions for Scraping Events Part 1
03:24:56 Helper Functions for Scraping Events Part 2
03:32:52 Saving Snapshot Scraping Results
03:38:29 Configure Scraping as Background Tasks
03:49:48 Run Background Scraping Tasks
03:53:48 Poll Scrape Status as Background Task
04:02:04 Tracking Scrape Event Finished At Time
04:08:36 A Webhook Handler View in Django
04:16:11 Tracking Scraping Snapshots through Webhooks with Django
04:25:48 Improved Auth Key for Webhooks
04:30:42 Webhook Handler for Reddit Posts
04:38:32 Adjust Data to Scrape
04:50:47 Background Sync Snapshot Reddit Results
05:04:39 Storing Reddit Communities in Django
05:16:53 Reddit AI Agent into Django Project
05:26:04 Topic Extraction Agent
05:32:50 Fuzzy Query to Scraping
05:40:45 Auto Scrape Reddit Communities on Save
05:52:37 Scraping Workflow as a Service Function
06:00:17 Store Queries & Topics
06:09:24 Topics to Reddit Communities
06:16:41 Full Query Automation
06:23:09 Reddit Community Trackablity
06:28:19 Scheduled Background Task to Trigger Reddit Scraping
06:33:27 Django Management Command to Trigger Scraping
06:36:17 Final Query Commands
06:38:22 Thank you and next steps
https://www.youtube.com/watch?v=XI-iP-qk_Vk
#financialfreedomllc #businesstips #business #entrepreneur #businessowner #entrepreneurship #marketing #smallbusiness #businesscoach #digitalmarketing #success #entrepreneurlife #motivation #businessideas #businessgrowth #businesswoman #businessman #businessquotes #businessowners #businessstrategy #startup #businesslife #businessmindset #businessminded #entrepreneurs #businessadvice #entrepreneurmindset #marketingtips #onlinebusiness #branding