Software Downloads for "Crawlers"
- ETagFix
- License: Freeware

ETagFix is an add-on for Microsoft Internet Information Server (IIS) or any web server that supports ISAPI filters, that cuts off part of the etag that IIS produces so that is does not change every time your computer is rebooted. Etag is an HTTP header that is produced by IIS to allow web crawlers and user clients to check against to see if a web page has changed. ETagFix fixes the problem where etags for each page change once your machine is rebooted which if not corrected can eat up more bandwidth by web crawlers crawling your web site when your web page has not changed at all.
- Platform: Windows
- Publisher: ISAPILabs
- Date: 23-04-2004
- Size: 713 KB
- obdkit sitemaps generator
- License: Shareware

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export data.
Support URL filtering and page content filtering, you can specify the content of the Web site in the sitemaps.
- Platform: Windows
- Publisher: obdkit Technology
- Date: 10-03-2011
- Size: 1665 KB
- tradesir sitemaps generator
- License: Shareware

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export data.
Support URL filtering and page content filtering, you can specify the content of the Web site in the sitemaps.
- Platform: Windows
- Publisher: tradesirTechnology
- Date: 05-02-2011
- Size: 1665 KB
- IPush.NET
- License: Shareware

IPush.NET is a powerful solution that provides an industrial strength stealth engine that can recognize incoming search engines, spiders and crawlers. Through IPush.NET you can provide search engines with optimized pages tailored for that specific spider. Human visitors will, instead, see your standard web pages. IPush.NET provides protection for your search engine ranking secrets. Because meta-tag are no more requested in your standard pages, you ranking methods are safe from your competitor's prying eyes.
- Platform: CGI and Perl, Scripts
- Publisher: ipush.net
- Date: 02-09-2011
- Spider Log Statistics
- License: Shareware

Addition to the standard PHP-Nuke Statistics Module. Adding the information about robots, spiders, or crawlers and spammers or email harvesters that hit your site..
- Platform: PHP, Scripts
- Publisher: indodigest.com
- Date: 06-03-2011
- CrawlTrack
- License: Freeware

CrawlTrack is a free application (license GNU GPL), which makes it possible to follow the search-engines crawlers and spiders visits on your website. Datas are displayed in the form of tables and of graphics for easy viewing. CrawlTrack is a php script using MySQL database. CrawlTrack exist for the moment in English, Spanish, German and French, the translators for other languages are welcome..
- Platform: PHP, Scripts
- Publisher: Jean-Denis Brun
- Date: 21-02-2011
- Free Large Android Icons
- License: Freeware

Free Large Android Icons picture various space creatures, such as robots, crawlers and androids. Not only does the set present the creatures themselves, it also shows their moods and actions. All the images are created by professional artists and demonstrate bright colors and well-finished edges. This free icon set is a perfect choice for your personal desktop customization. Each icon includes the following sizes: 32x32, 48x48 and 256x256 pixels. 32-bit color scheme is used, making the set compatible with Windows XP and Windows Vista.
- Platform: Android, AS/400, BlackBerry, Handheld, iOS 4.x, Java, Linux, Mac, OS/2, Palm, Pocket PC, Tablets, Unix, Windows
- Publisher: large-icons.com
- Date: 20-09-2012
- Size: 29388 KB
- Java Sitemap Parser
- License: Freeware

The Java Sitemap Parser can parse a website's Sitemap (http://www.sitemaps.org/). This is useful for web crawlers that want to discover URLs from a website that is using the Sitemap Protocol..
- Platform: Linux, Mac, Windows
- Publisher: sitemap-parser.sourceforge.net
- Date: 01-08-2012
- Size: 36 KB
- Noti
- License: Shareware

Noti is a news publishing framework. It consist of several crawlers that fetch news web pages, index their content, and publish it in a centralized web site, providing user customized feeds and daily emails of the lastest news..
- Platform: PHP, Scripts
- Publisher: Pablo Hoffman
- Date: 15-05-2013
- Size: 143 KB
- OpenWebSpider
- License: Freeware

OpenWebSpider ia a free PHP search engine project, OpenWebSpider uses MySQL Server as the backend to store the index.The new released OpenWebSpider v0.1.4 is featured with MP3s and PDFs support! Complete CHANGELOG of OpenWebSpider: - Mysql/NET Connector upgraded to 5.2.5.0 - Enhanced encodings support - New feature: Support to META: 'robots' (NOINDEX, NOFOLLOW) - New feature: New configuration file field: crawler_id - New field 'crawler_id' in table 'hostlist' - New table: crawler_act - New feature: Remote actions over running crawlers [Status, Play, Pause, Kill] - New file support: PDFs [Using PDFBox and IKVM] - New table: pdf - New file support: MP3s [Using UltraID3Lib] - New table: mp3 - New feature: new command-line argument: d-?OCOd-?OCOpdf - New feature: new command-line argument: d-?OCOd-?OCOmp3.
- Platform: PHP, Scripts
- Publisher: Jelsoft Enterprises Ltd
- Date: 09-01-2013
- Size: 16148 KB
- Methanol Web Crawling System
- License: Freeware

A cross-platform, highly configurable, distributed web crawling system with crawlers optimized for speed.
Methanol Web Crawling System License - BSD License.
- Platform: WinOther
- Publisher: Methanol
- Date:
- Music Ontology tools
- License: Freeware

Some tools related to the Music Ontology - including domain-specific Semantic Web crawlers, audio collection management and mapping tools
Music Ontology tools License - BSD License.
- Platform: WinOther
- Publisher: Motools
- Date:
- Size: 50 KB
- academixml
- License: Freeware

AcademiXML is markup standard to make academic (scholar, research) texts (papers, reports) syndicable, parsable, indexable and readable by both humans and bots (web-crawlers) without rendering engine (browser, app, parser, database).
academixml License - Public Domain.
- Platform: WinOther
- Publisher: Academixml
- Date:
- Open Dungeon Crawler 3D
- License: Freeware

The Open Dungeon Crawler 3D (ODC) is a remake of the classic 90s era 3D dungeon crawlers such as "Eye of The Beholder" and "Dungeon Master". It is initially targeted at the GameCube and desktop OSes.
Open Dungeon Crawler 3D License - GNU General Public License (GPL); Other License.
- Platform: WinOther
- Publisher: Opendc3d
- Date:
- hylafax integration
- License: Freeware

integration of hylafax and web crawlers
hylafax integration License - GNU General Public License (GPL).
- Platform: WinOther
- Publisher: Hylafaxintegrat
- Date:
- Web Scraper Plus+: Web Spider Edition
- License: Shareware

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own web grabber that can screen scrape the data to a database or Excel has never been easier. The website spider / crawlers can even sort and filter the webpages while spidering. Web Scraper Plus+: Web Spider Edition is the leading web data harvesting application.
- Platform: Windows
- Publisher: velocityscape.com
- Date:
- Size: 52520 KB
- SoulCraft - Action RPG (free)
- License: Freeware

SoulCraft is the best free2play Action RPG game for Android. If you have been a hunter in enough dungeon crawlers and feel like you have been a warrior for all eternity already, answer the call, fight for the torchlight of hope, do your duty and choose to be a legendary hero in the action role playing game SoulCraft :)
*** SoulCraft featured by Google for Google I/O 2012 ***
*** Handpicked game for Google booth at Mobile World Congress 2012 ***
*** our favourite user comment *This game is better than Elvis* :) ***
*** please see SoulCraftGame.
- Platform: Android, Windows
- Publisher: MobileBits GmbH
- Date:
- Size: 237568 KB
- Live Deals
- License: Shareware

Live Deals is an application that searches for deals, discounts, coupons, bargains, etc across the web. The site is updated throughout the day by the staff and the automated crawlers. We are dedicated to providing you with the tools needed to find the best deals online..
- Platform: Windows
- Publisher: Athena
- Date:
- Size: 1024 KB
- SoulCraft THD (free)
- License: Freeware

Optimized version for NVIDIA Tegra Devices
SoulCraft is the best free2play Action RPG game for Android. If you have been a hunter in enough dungeon crawlers and feel like you have been a warrior for all eternity already, answer the call, fight for the torchlight of hope, do your duty and choose to be a legendary hero in the action role playing game SoulCraft :)
*** SoulCraft featured by Google for Google I/O 2012 ***
*** Handpicked game for Google booth at Mobile World Congress 2012 ***
*** our favourite user comment “This game is better than Elvis” :) ***
*** please see SoulCraftGame.
- Platform: Android 2.x, Android 3.x, Android 4.4, Android 4.x
- Publisher: MobileBits GmbH
- Date: 08-03-2015
- Size: 165888 KB
- Octoparse
- License: Shareware

Octoparse is the perfect tool for people who want to scrape websites without learning to code. It features a point and click screen scraper, allowing users to scrape behind login forms, fill in forms, input search terms, scroll through infinite scroll, render javascript, and more. It also includes a site parser and a hosted solution for users who want to run their scrapers in the cloud. Best of all, it comes with a generous free tier allowing users to build up to 10 crawlers for free. For enterprise level customers, they also offer fully customized crawlers and managed solutions where they take care of running everything for you and just deliver the data to you directly.
- Platform: https://www.octoparse.es/
- Publisher: Melisa
- Date: 14-07-2020
- Size: 1300 KB






