Collect web pages and data,It can help you build a website quickly。Besides using a capture device to capture,You can also customize Python web crawlers,However, you need to know how to program。
The crawler has independent software、Various methods, including plugins and cloud-based collection。Standalone software requires a dedicated computer to run continuously,The publishing interface for the corresponding CMS。The plugin version and the site are on the same server,Collection will slow down the website。Publishing to the cloud typically costs a considerable amount of money for one's own website。Not well-off,This section only mentions those that can be used for free.。
Crawler solutions also include filling in rules such as regular expressions,and the WYSIWYG simulated browser approach。
Crawler software
- Train Collector:They all like to be called the 'Locomotive' scraper,This might be the most widely used scraper currently,Years of upgrades have made it very complete,There is a free version,All basic features are included,That's enough。
- A cracked version of LocaLizer 7.6 Enterprise Edition is circulating online、Train Head Collector V9 (High-Speed Rail Hero) Intranet Encryption Dog Cracked Version,If you have sufficient funds,Please support the official version。
- Video tutorial
- Train Browser:Same manufacturer as the one above,Visual operation,The free plan limits a single task to run for 30 minutes。
- ET mining software:It is also an established brand,Fully automated unattended operation。There is a free version。
- VG automation tool:Originally named VG Browser,VG web operation tool,Focuses on web automation (automatic login、Auto data collection、Auto-submit、Auto-registration、 Auto-posting、CAPTCHA auto-detection。)A free version is available, Limit a single task to run for 30 minutes。
- Octopus crawler:Software download required,However, the flagship version can perform data collection in the cloud.,Can be harvested just by seeing it,Whatever the browser can see, it can capture。There is a free version,The free version only supports local collection,Local collection seems to require points for export。
- Houyi Collector:A WYSIWYG browser-based data collection tool,Also supports Windows、Collector for all Mac and Linux operating systems。Free web scraping software,Basic features required for data collection,No restrictions whatsoever,No points required,Free export of collected results。However, if you want to run it in the background for scheduled collection, you need the paid version.。
- Video tutorial | Text Tutorial
- Download the latest version:http://www.houyicaiji.com/?type=download
Crawler plugin
Plugin-type collectors can publish data directly without needing to release an API,But it can only be used with one specific CMS or blog platform at a time.。
WordPress Scraping Plugin
- WP-AutoBlog:Original WP-AutoPost,Supports translation into other languages,WeChat Official Accounts can be collected、Headline accounts and other self-media content,Supports downloading remote images to a local server。There is a free version,However, there are feature limitations。
- A cracked version of WP-AutoPost is circulating online
- Pangshu Collection:Free and open source,Using Html,jQuery Syntax,Automatically collect and publish articles,
- Search in WordPress plugins Pangshu Collection Just install it
- Github Open Source Address
- How to Use
- Long-legged spider(WP-CTSPIDER):Free to use,To use the full version, you need toOfficial websiteActivate authorization for free。It seems like it hasn't been updated for a long time.。Supports pseudo-original content generation,But you need to fill it in yourself。Supports Baidu Translate。
DEDECMS Scraping Plugin
- Dreamweaver Collector:Supports RSS scraping,Perform broad collection based on the keywords set by the user。No free version
Cloud Collection
- Blue-sky Collector:Official Cloud-based Automated Data Collection and Publishing System,Can be integrated with any CMS or blog,But the corresponding publishing plugin is required。It's said to be cloud-based, but it needs to be on the same server as the published website,Unable to publish remotely,It might be more suitable to place this in the plugins category。
- Officially provides most mainstream CMSs、BLOGPost plugin