This project aims to evaluate the feasibility of an architecture based on serverless technology for a web crawler using scrapy as a framework of choice. If you are interested in this project you can visit my blog post and read about it.
This is a webscraper for a specific website. It is tuned to extract the headlines of that website. With some little adjustments the webscraper is able to extract any part of the website.
crawlersuseragents This Python script can be used to check if there is any differences in responses of an application when the request comes from a se
A small library for extracting rich content from urls. what does it do? micawber supplies a few methods for retrieving rich metadata about a variety o
Matrix Content Scanner A web service for scanning media hosted by a Matrix media repository Installation TODO Development In a virtual environment wit
12306抢票脚本
BiliComment A spider for BiliBili comment. Spider Usage Put config.json into config directory, and then python . ./config/config.json. A example confi
scraper-dblp Get paper names from dblp.org and store them in a .txt file Useful for a related literature :) Install libraries pip3 install -r requirem
DaProfiler allows you to get emails, social medias, adresses, works and more on your target using web scraping and google dorking techniques, based in France Only. The particularity of this program i
AutoReserve A webdriver-based script for reserving badminton courts. 使用说明 下载 chromedriver 选择当前Chrome对应版本 安装 selenium pip install selenium 更改场次、金额信息dat
paper finder Find papers by keywords and venues. Then download it automatically. How to use this? Search CLI python search.py -k "knowledge tracing,kn
Grab Framework Documentation Installation $ pip install -U grab See details about installing Grab on different platforms here http://docs.grablib.
The Onion Article Scraper Scrapes all articles and their headlines from the satirical news website https://www.theonion.com Also see Clickhole Article
Pro Football Reference Game Data Webscraper Code Copyright Yeetzsche This is a simple Pro Football Reference Webscraper that can either collect all ga
onlyfans-scraper A command-line program to download media, like and unlike posts, and more from creators on OnlyFans. Installation You can install thi
Unja Fetch Known Urls What's Unja? Unja is a fast & light tool for fetching known URLs from Wayback Machine, Common Crawl, Virus Total & AlienVault's
Instagram_scrapper This project allow you to scrape the list of followers, following or both from a public Instagram account, and create a csv or exce
Amazon-web-scraping-using-Scrapy-Framework Scrapy Scrapy is an application framework for crawling web sites and extracting structured data which can b
Meme Videos Scrapes memes from reddit using praw and request and then converts t
NASA APOD Discord Bot - Fetches information from NASA APOD site.
Annex Bubt Scraping Script I think this is the first public repository that provides free annex-BUBT, BUBT-Soft, and BUBT website scraping API script
1 Dec 13, 2021
13 Dec 27, 2022
588 Dec 27, 2022
5 Dec 01, 2022
457 Jan 05, 2023
3 Jul 05, 2021
1 Dec 07, 2021
347 Jan 07, 2023
4 Nov 09, 2021
2 Dec 15, 2022
2.3k Jan 04, 2023
0 Nov 17, 2021
6 Dec 21, 2022
185 Jul 23, 2022
10 Aug 07, 2022
5 Oct 17, 2022
1 Jan 25, 2022
12 Oct 28, 2022
4 Apr 23, 2022
4 Dec 03, 2022