Reddit scraper in Python: subreddits, posts, comments and search

You can scrape public Reddit subreddits, posts with their first comments, keyword search and a user's posts with one Python file and no API key: toolzerhub/reddit-scraper on GitHub. It is 350 lines, and all five commands returned live data when we ran them on 2026-10-13. It does not use Reddit's .json URLs, the shortcut many tutorials start with, because those refused all 18 of our requests that day. We make ToolzerHub: for more of a thread's comments, a user's comments or comment search, the API section below shows what one call returns.

The .json trick now returns 403

Adding .json to a Reddit URL used to return the page as JSON. We tried it on 2026-10-13 through three residential proxy exits, once with a descriptive User-Agent and once with Chrome's headers:

From our datacenter server, all six requests returned 403. That is one day and one proxy provider, but it matches what Scrapfly reported on 2026-10-01: that Reddit plans to end unauthenticated .json access. The redd package on PyPI (0.2.1, "uses Reddit's public .json endpoints") and several tutorials ranking for this search still build on these URLs, so if your old scraper stopped returning data, check this first. PRAW goes through the official Data API, whose terms require the OAuth token Reddit issues you (more under check the rules).

Install the script and run the five commands

Clone the repo, install the two dependencies (curl_cffi and beautifulsoup4) and point subreddit at any public community. We ran everything below on Python 3.12.13.

git clone https://github.com/toolzerhub/reddit-scraper.git
cd reddit-scraper
pip install -r requirements.txt
python reddit_scraper.py subreddit python

The other four commands:

python reddit_scraper.py subreddit space --sort top --time week --limit 50 -o space.json
python reddit_scraper.py about nasa dataisbeautiful python space -o about.json
python reddit_scraper.py post https://www.reddit.com/r/Python/comments/1x1t161/python_315_released/ --comments 20
python reddit_scraper.py search "web scraping" --limit 25 -o search.json
python reddit_scraper.py user nasa --limit 10

Every command accepts -o file.json to save the result and --proxy URL to route through a proxy. A subreddit or user can be a name (python, r/python, u/nasa) or a URL. One post from user nasa, as returned on 2026-10-13:

{
  "id": "1wzz0yh",
  "title": "NASA just released 814GB of Artemis II data, including nearly 1,000 new images, in their Artemis II Preliminary Lunar Science Report",
  "url": "https://www.reddit.com/user/nasa/comments/1wzz0yh/nasa_just_released_814gb_of_artemis_ii_data/",
  "subreddit": "u_nasa",
  "author": "nasa",
  "created_at": "2026-10-07T15:09:21+00:00",
  "score": 244,
  "upvote_ratio": 1.0,
  "comment_count": 2,
  "post_type": "link",
  "flair": null,
  "content_url": "https://science.nasa.gov/feature/artemis-ii-preliminary-lunar-science-report/",
  "domain": "science.nasa.gov",
  "text": null,
  "is_pinned": false,
  "is_locked": false,
  "is_spoiler": false
}

Feed and user posts share these fields. search returns a shorter record (id, title, url, subreddit, author, date, score, comment count, NSFW flag), and post adds a comments list with parent_id and depth on each comment.

What each command returned

All five commands returned live data in 3.1 to 12.9 seconds per run. We ran them on 2026-10-13 through a residential proxy and timed each one, including the session setup.

• subreddit python took 3.3 seconds and returned 25 posts from the hot feed, starting with the pinned daily thread.
• subreddit space --sort top --time week --limit 50 took 6.3 seconds for 50 posts over two pages, with scores from 6 to 17,025 (median 259). The first try hit a 45-second proxy timeout; the rerun worked.
• about for 4 subreddits took 3.1 seconds, and for 10 subreddits 12.9 seconds, 10 of 10. Each record has the description, weekly visitors and contributions, creation date and the full rules list. There is no member count: the pages the script reads don't show one, so r/space reports 402,528 weekly visitors instead.
• post on the Python 3.15 release thread took 3.3 seconds and returned the post text plus 20 comments, 14 of them replies. The thread reports 70 comments; with --comments 200 we still got 25, because the script reads only the first batch.
• search "web scraping" --limit 25 took 9.2 seconds and returned 25 posts from 11 subreddits.
• user nasa --limit 10 took 6.6 seconds and returned NASA's 10 latest posts.

How the script gets the data

Reddit's current site loads feeds, comments and search results as small HTML fragments from /svc/shreddit/, and the script asks for the same fragments.

1. It sends requests with curl_cffi, an HTTP client built on curl that makes its requests look like Chrome's.
2. Each fragment holds elements such as <shreddit-post> and <shreddit-comment>, with the data in their attributes: score, comment count, author, timestamps. Beautiful Soup reads those attributes, and the script turns them into the fields above. post also reads the post's own page for the body text and flags.
3. For the next page, Reddit wants an after cursor. The script builds it from the last post's id:

def cursor_for(fullname):
    # Reddit's `after` is base64 of the last item's fullname. The '=' padding matters:
    # without it Reddit answers 200 with an empty feed, which looks like the end.
    return base64.b64encode(fullname.encode()).decode()

cursor_for("t3_1wzz0yh") returns dDNfMXd6ejB5aA==. As the comment in the script warns, a cursor without its = padding gets an empty feed with status 200, so a broken pager looks like the end of the subreddit.

Save the top posts from several subreddits to CSV

Reddit is a class you can import, so a list of subreddits becomes one CSV in a few lines. This file sits next to reddit_scraper.py, reuses one session, waits 2 seconds between subreddits and reports the ones that fail. It is the version we ran.

import csv
import os
import sys
import time

from reddit_scraper import Reddit, ScrapeError

FIELDS = ["subreddit", "title", "score", "comment_count", "created_at", "url"]


def main(path, subreddits):
    reddit = Reddit(os.environ.get("HTTPS_PROXY"))  # one session for the whole list
    rows = failed = 0
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS, extrasaction="ignore")
        writer.writeheader()
        for name in subreddits:
            try:
                posts = reddit.subreddit(name, sort="top", period="week", limit=10)
                writer.writerows(posts)
                rows += len(posts)
            except ScrapeError as exc:
                print(f"{name}: {exc}", file=sys.stderr)
                failed += 1
            time.sleep(2)  # pause between subreddits
    print(f"wrote {rows} posts from {len(subreddits)} subreddits to {path}")
    if failed == len(subreddits):
        sys.exit(1)  # every subreddit failed: don't let a cron job think it worked


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2:])

python subreddits_to_csv.py top-week.csv space nasa python dataisbeautiful zz_no_such_sub_88123

Output on 2026-10-13, after 16.2 seconds (file path and URLs shortened):

zz_no_such_sub_88123: No posts found in r/zz_no_such_sub_88123. It may not exist, or it may be private or banned.
wrote 40 posts from 5 subreddits to top-week.csv

subreddit,title,score,comment_count,created_at,url
space,"Margaret Hamilton, computing pioneer who led software development for the Apollo program, dies at 90",17024,147,2026-10-07T22:34:43+00:00,https://www.reddit.com/r/space/comments/1x0ahqw/...
space,"Moonlight reflecting off the Amazon river, seen from the International Space Station",9924,59,2026-10-12T23:12:45+00:00,https://www.reddit.com/r/space/comments/1x4gwt8/...
space,"Abandoned under the cosmos | Southland, New Zealand",7340,34,2026-10-11T18:17:23+00:00,https://www.reddit.com/r/space/comments/1x3fba0/...

The four real subreddits gave 10 posts each; the made-up name was reported and skipped. When every subreddit fails, as it did from our datacenter server, the script exits with status 1, so a scheduled job sees the failure. Scores and comment counts are live, so they will differ when you run it.

Where the script gets blocked

The repo's README says Reddit blocks most cloud and datacenter IPs, and that matched our test. On 2026-10-13 we ran subreddit python on our server with no proxy and got, after 0.2 seconds:

Error: Reddit blocked the request (HTTP 403). Cloud and datacenter IPs are blocked; try a home connection or --proxy. A residential IP is sometimes blocked once too: run it again.

We ran the tests above through a residential proxy, and the README suggests a home connection. CLIENT_VERSION at the top of the script is a build id from Reddit's own frontend; the README says Reddit retires these over time, so if every call starts failing, refresh it from a browser first. Per the README, the script has no retries and no concurrency, so a timeout surfaces as an error and you rerun the command.

Check Reddit's rules before you scrape

Reddit's User Agreement (effective 2026-07-01) lists, among things you may not do: "Access, search, or collect data from the Services by any means (automated or otherwise) except as permitted in these Terms or in a separate agreement with Reddit (we conditionally grant permission to crawl the Services in accordance with the parameters set forth in our robots.txt file, but scraping the Services without Reddit's prior written consent is prohibited)". The script reads only what any visitor can see, and the terms still apply to it; the script does not read robots.txt. Posts and comments carry usernames and free text, so collect only what you need and delete it when you are done.

The official route is the Data API. Its terms (revised 2026-07-20) require the OAuth token Reddit gives you, and say commercial use, or research beyond the rate limits, needs a separate agreement with Reddit. Our Reddit API pricing breakdown covers what that route allows for free and what is known about its price.

What the script can't do and the API calls that do

The script stops at the first 25 comments on a thread, has no user comments and searches posts only. Our Reddit API covers all three as JSON from any server with one API key. On 2026-10-13 we called these endpoints:

The thread call came back with 47 of 70 comments, not all of them. The response says why: truncated_threads: 2 counts branches Reddit only links to, and unfollowed_branches: 3 counts branches the call did not walk (end_of_tree: false). Comments come back flat in thread order, and depth plus parent_fullname let you rebuild the tree. The response below is trimmed:

curl -H "x-api-key: $TOOLZERHUB_API_KEY" \
  "https://api.toolzerhub.com/v1/reddit/post-comments?subreddit=Python&post_id=1x1t161&count=200"

{
  "data": {
    "total_comments": 70,
    "count": 47,
    "requested_count": 200,
    "end_of_tree": false,
    "truncated_threads": 2,
    "comments": [
      { "id": "pewn5rx", "parent_fullname": null, "depth": 0, "score": 60, "body": "...", "created_at": "2026-10-09T19:04:34.899000+0000" }
    ]
  }
}

The real response carries 17 fields per comment, including the author and permalink. There are nine /v1/reddit endpoints, from subreddit info to user profiles. We called each one three times on 2026-10-13: 27 of 27 returned data, median 3.9 seconds, slowest 17.3 seconds (post details).

What the API costs

A free key gives you 100 credits a month with no card: get one here. Every /v1/reddit call costs 1 credit, and a failed call costs nothing. Paid plans are $29 for 20,000 credits ($1.45 per 1,000), $79 for 60,000 ($1.32), $199 for 165,000 ($1.21) and $495 for 500,000 ($0.99). At 200 posts per call, 1,000 posts is 5 calls: 5 credits, under one cent. Check the pricing page for current rates.

We are not affiliated with Reddit, Inc. For the same kind of script on another platform, see our Instagram scraper in Python.

Sources

• toolzerhub/reddit-scraper, source and README (commit c7d3eaa), https://github.com/toolzerhub/reddit-scraper, run 2026-10-13
• Reddit User Agreement (effective 2026-07-01), https://redditinc.com/policies/user-agreement, read 2026-10-13
• Reddit Data API Terms (revised 2026-07-20), https://redditinc.com/policies/data-api-terms, read 2026-10-13
• ToolzerHub pricing, https://toolzerhub.com/pricing, checked 2026-10-13
• ToolzerHub API OpenAPI spec (credits per endpoint), https://api.toolzerhub.com/openapi.json, read 2026-10-13