Software Downloads for "Crawlers"

ETagFix is an add-on for Microsoft Internet Information Server (IIS) or any web server that supports ISAPI filters, that cuts off part of the etag that IIS produces so that is does not change every time your computer is rebooted. Etag is an HTTP header that is produced by IIS to allow web crawlers and user clients to check against to see if a web page has changed. ETagFix fixes the problem where etags for each page change once your machine is rebooted which if not corrected can eat up more bandwidth by web crawlers crawling your web site when your web page has not changed at all.

  • Platform: Windows
  • Publisher: ISAPILabs
  • Date: 23-04-2004
  • Size: 713 KB

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export data.
Support URL filtering and page content filtering, you can specify the content of the Web site in the sitemaps.

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export data.
Support URL filtering and page content filtering, you can specify the content of the Web site in the sitemaps.

IPush.NET is a powerful solution that provides an industrial strength stealth engine that can recognize incoming search engines, spiders and crawlers. Through IPush.NET you can provide search engines with optimized pages tailored for that specific spider. Human visitors will, instead, see your standard web pages. IPush.NET provides protection for your search engine ranking secrets. Because meta-tag are no more requested in your standard pages, you ranking methods are safe from your competitor's prying eyes.

  • Platform: CGI and Perl, Scripts
  • Publisher: ipush.net
  • Date: 02-09-2011

Addition to the standard PHP-Nuke Statistics Module. Adding the information about robots, spiders, or crawlers and spammers or email harvesters that hit your site..

CrawlTrack is a free application (license GNU GPL), which makes it possible to follow the search-engines crawlers and spiders visits on your website. Datas are displayed in the form of tables and of graphics for easy viewing. CrawlTrack is a php script using MySQL database. CrawlTrack exist for the moment in English, Spanish, German and French, the translators for other languages are welcome..

Free Large Android Icons picture various space creatures, such as robots, crawlers and androids. Not only does the set present the creatures themselves, it also shows their moods and actions. All the images are created by professional artists and demonstrate bright colors and well-finished edges. This free icon set is a perfect choice for your personal desktop customization. Each icon includes the following sizes: 32x32, 48x48 and 256x256 pixels. 32-bit color scheme is used, making the set compatible with Windows XP and Windows Vista.

  • Platform: Android, AS/400, BlackBerry, Handheld, iOS 4.x, Java, Linux, Mac, OS/2, Palm, Pocket PC, Tablets, Unix, Windows
  • Publisher: large-icons.com
  • Date: 20-09-2012
  • Size: 29388 KB

The Java Sitemap Parser can parse a website's Sitemap (http://www.sitemaps.org/). This is useful for web crawlers that want to discover URLs from a website that is using the Sitemap Protocol..

  • Noti
  • License: Shareware
  • Rating

Noti is a news publishing framework. It consist of several crawlers that fetch news web pages, index their content, and publish it in a centralized web site, providing user customized feeds and daily emails of the lastest news..

  • Platform: PHP, Scripts
  • Publisher: Pablo Hoffman
  • Date: 15-05-2013
  • Size: 143 KB

OpenWebSpider ia a free PHP search engine project, OpenWebSpider uses MySQL Server as the backend to store the index.The new released OpenWebSpider v0.1.4 is featured with MP3s and PDFs support! Complete CHANGELOG of OpenWebSpider: - Mysql/NET Connector upgraded to 5.2.5.0 - Enhanced encodings support - New feature: Support to META: 'robots' (NOINDEX, NOFOLLOW) - New feature: New configuration file field: crawler_id - New field 'crawler_id' in table 'hostlist' - New table: crawler_act - New feature: Remote actions over running crawlers [Status, Play, Pause, Kill] - New file support: PDFs [Using PDFBox and IKVM] - New table: pdf - New file support: MP3s [Using UltraID3Lib] - New table: mp3 - New feature: new command-line argument: d-?OCOd-?OCOpdf - New feature: new command-line argument: d-?OCOd-?OCOmp3.

A cross-platform, highly configurable, distributed web crawling system with crawlers optimized for speed.
Methanol Web Crawling System License - BSD License.

  • Platform: WinOther
  • Publisher: Methanol
  • Date:

Some tools related to the Music Ontology - including domain-specific Semantic Web crawlers, audio collection management and mapping tools
Music Ontology tools License - BSD License.

  • Platform: WinOther
  • Publisher: Motools
  • Date:
  • Size: 50 KB

AcademiXML is markup standard to make academic (scholar, research) texts (papers, reports) syndicable, parsable, indexable and readable by both humans and bots (web-crawlers) without rendering engine (browser, app, parser, database).
academixml License - Public Domain.

The Open Dungeon Crawler 3D (ODC) is a remake of the classic 90s era 3D dungeon crawlers such as "Eye of The Beholder" and "Dungeon Master". It is initially targeted at the GameCube and desktop OSes.
Open Dungeon Crawler 3D License - GNU General Public License (GPL); Other License.

  • Platform: WinOther
  • Publisher: Opendc3d
  • Date:

integration of hylafax and web crawlers
hylafax integration License - GNU General Public License (GPL).

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own web grabber that can screen scrape the data to a database or Excel has never been easier. The website spider / crawlers can even sort and filter the webpages while spidering. Web Scraper Plus+: Web Spider Edition is the leading web data harvesting application.

SoulCraft is the best free2play Action RPG game for Android. If you have been a hunter in enough dungeon crawlers and feel like you have been a warrior for all eternity already, answer the call, fight for the torchlight of hope, do your duty and choose to be a legendary hero in the action role playing game SoulCraft :)


*** SoulCraft featured by Google for Google I/O 2012 ***
*** Handpicked game for Google booth at Mobile World Congress 2012 ***
*** our favourite user comment *This game is better than Elvis* :) ***
*** please see SoulCraftGame.

  • Platform: Android, Windows
  • Publisher: MobileBits GmbH
  • Date:
  • Size: 237568 KB

Live Deals is an application that searches for deals, discounts, coupons, bargains, etc across the web. The site is updated throughout the day by the staff and the automated crawlers. We are dedicated to providing you with the tools needed to find the best deals online..

  • Platform: Windows
  • Publisher: Athena
  • Date:
  • Size: 1024 KB

Optimized version for NVIDIA Tegra Devices

SoulCraft is the best free2play Action RPG game for Android. If you have been a hunter in enough dungeon crawlers and feel like you have been a warrior for all eternity already, answer the call, fight for the torchlight of hope, do your duty and choose to be a legendary hero in the action role playing game SoulCraft :)


*** SoulCraft featured by Google for Google I/O 2012 ***
*** Handpicked game for Google booth at Mobile World Congress 2012 ***
*** our favourite user comment “This game is better than Elvis” :) ***
*** please see SoulCraftGame.

  • Platform: Android 2.x, Android 3.x, Android 4.4, Android 4.x
  • Publisher: MobileBits GmbH
  • Date: 08-03-2015
  • Size: 165888 KB

Octoparse is the perfect tool for people who want to scrape websites without learning to code. It features a point and click screen scraper, allowing users to scrape behind login forms, fill in forms, input search terms, scroll through infinite scroll, render javascript, and more. It also includes a site parser and a hosted solution for users who want to run their scrapers in the cloud. Best of all, it comes with a generous free tier allowing users to build up to 10 crawlers for free. For enterprise level customers, they also offer fully customized crawlers and managed solutions where they take care of running everything for you and just deliver the data to you directly.

  • Platform: https://www.octoparse.es/
  • Publisher: Melisa
  • Date: 14-07-2020
  • Size: 1300 KB
New Reviews


Print Inspector

Whether you are using a printer connected to your local desktop computer or a print server sometimes you need to see whats going on in your printer by having a close look at the print jobs sent by the users. Print Inspector is such type software ...


Disk Genius

Disk Genius was formerly known as Partition Guru which is an all-in-one solution for disk partition management, data recovery and disk repairing. It can recover lost files from partition, backup and restore partition table, clone partition or clone ...


Safe In Cloud

It is quite difficult to find any computer users nowadays who do not need to use the web, for any purpose. Whether it is for education or entertainment, the web is your best resource. You also need to use various online services for banking, ...


PictBear

Nowadays, a majority of PC users resort to digital image editing once in a while. It may be required for retouching image captured in smart phones or for creating an album in Facebook. For some people, tinkering with digital images is a pastime too. ...


Panda Cloud Cleaner

PC security has become a tricky and complicated subject both for home users and network supervisors, with complexity and risk factors multiplying over the years. Using a single antivirus app may not be enough anymore. That explains why many users ...


SnapCrab

Nearly every PC users need to take screenshots from time to time, whether it is for personal or professional needs. While using the basic Windows screenshot capture method is available, it is not adequate for everyone. When you want to capture ...


BitKiller

Dealing with various types of data efficiently holds the key to success in any computing task. Apart from generating new data and modifying existing files, it becomes necessary to delete existing files at times. While most users resort to using ...


SeaMonkey, Portable Edition

Using the Internet has become a necessity for a majority of people, including those who do not use a PC regularly. To browse web, stream video or download online content, people often resort to using various third party apps. For example, using ...


TuckAway Intelligent Email Organizer Pro

Using an email software allow you to do a number of things including managing your address lists, emails and even ending memos, documents and even virtual letters. You can also use it to manage your tasks easily. The email software comes in ...


WebAcappella

Web design sector has witnessed several trends and technological innovations affect website development in big ways. With time, designing websites has become easier. Nowadays, you can find web design software that nearly eliminates the need to ...


New Downloads

Spool Queue Viewer++

Spool Queue Viewer++ is a
feature rich spool queue
viewer that windows should
have by default, we have taken
it to the ...

GlassWire Firewall

GlassWire\'s free firewall
software helps protect your
computer, privacy, and data by
monitoring your network
activity. ...

Easy Card Creator Express

The ideal choice for small to
medium sized organizations and
individuals looking for an
affordable photo ID card ...

Express Zip Plus for Mac

Express Zip Mac Software for
File Compression and
Archiving. Easily create,
edit, manage and extract
compressed files and ...

Devart ODBC Driver for
Oracle

Devart ODBC Driver for Oracle
is a high-performance
connectivity solution with
enterprise-level features for
accessing ...

Lodgit Desk Hotel Software
for Windows

Booking software for all kinds
of lodging establishments
incl. reservation schedule,
booking and guest list,
cleaning ...

AllMyNotes Organizer
Portable

This is simply the most
advanced software to keep all
private info securely
encrypted on USB drive, worry
free. Simply ...

Pixillion Image Converter
Software Free

Pixillion Image Converter
Software Free is perfect for
converting your image files
for posting on the web,
emailing to ...

Windows 10 Manager

An all-in-one utility for
windows 10 specially, it
includes over forty utilities
to optimize, tweak, clean up
and fix ...

Aml Maple

Aml Maple has a new kind of
view on keyboard layout
indicators: simple, modern,
flexible. In addition, you can
change the ...