Software Downloads for "Crawlers"

ETagFix is an add-on for Microsoft Internet Information Server (IIS) or any web server that supports ISAPI filters, that cuts off part of the etag that IIS produces so that is does not change every time your computer is rebooted. Etag is an HTTP header that is produced by IIS to allow web crawlers and user clients to check against to see if a web page has changed. ETagFix fixes the problem where etags for each page change once your machine is rebooted which if not corrected can eat up more bandwidth by web crawlers crawling your web site when your web page has not changed at all.

  • Platform: Windows
  • Publisher: ISAPILabs
  • Date: 23-04-2004
  • Size: 713 KB

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export data.
Support URL filtering and page content filtering, you can specify the content of the Web site in the sitemaps.

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export data.
Support URL filtering and page content filtering, you can specify the content of the Web site in the sitemaps.

IPush.NET is a powerful solution that provides an industrial strength stealth engine that can recognize incoming search engines, spiders and crawlers. Through IPush.NET you can provide search engines with optimized pages tailored for that specific spider. Human visitors will, instead, see your standard web pages. IPush.NET provides protection for your search engine ranking secrets. Because meta-tag are no more requested in your standard pages, you ranking methods are safe from your competitor's prying eyes.

  • Platform: CGI and Perl, Scripts
  • Publisher: ipush.net
  • Date: 02-09-2011

Addition to the standard PHP-Nuke Statistics Module. Adding the information about robots, spiders, or crawlers and spammers or email harvesters that hit your site..

CrawlTrack is a free application (license GNU GPL), which makes it possible to follow the search-engines crawlers and spiders visits on your website. Datas are displayed in the form of tables and of graphics for easy viewing. CrawlTrack is a php script using MySQL database. CrawlTrack exist for the moment in English, Spanish, German and French, the translators for other languages are welcome..

Free Large Android Icons picture various space creatures, such as robots, crawlers and androids. Not only does the set present the creatures themselves, it also shows their moods and actions. All the images are created by professional artists and demonstrate bright colors and well-finished edges. This free icon set is a perfect choice for your personal desktop customization. Each icon includes the following sizes: 32x32, 48x48 and 256x256 pixels. 32-bit color scheme is used, making the set compatible with Windows XP and Windows Vista.

  • Platform: Android, AS/400, BlackBerry, Handheld, iOS 4.x, Java, Linux, Mac, OS/2, Palm, Pocket PC, Tablets, Unix, Windows
  • Publisher: large-icons.com
  • Date: 20-09-2012
  • Size: 29388 KB

The Java Sitemap Parser can parse a website's Sitemap (http://www.sitemaps.org/). This is useful for web crawlers that want to discover URLs from a website that is using the Sitemap Protocol..

  • Noti
  • License: Shareware
  • Rating

Noti is a news publishing framework. It consist of several crawlers that fetch news web pages, index their content, and publish it in a centralized web site, providing user customized feeds and daily emails of the lastest news..

  • Platform: PHP, Scripts
  • Publisher: Pablo Hoffman
  • Date: 15-05-2013
  • Size: 143 KB

OpenWebSpider ia a free PHP search engine project, OpenWebSpider uses MySQL Server as the backend to store the index.The new released OpenWebSpider v0.1.4 is featured with MP3s and PDFs support! Complete CHANGELOG of OpenWebSpider: - Mysql/NET Connector upgraded to 5.2.5.0 - Enhanced encodings support - New feature: Support to META: 'robots' (NOINDEX, NOFOLLOW) - New feature: New configuration file field: crawler_id - New field 'crawler_id' in table 'hostlist' - New table: crawler_act - New feature: Remote actions over running crawlers [Status, Play, Pause, Kill] - New file support: PDFs [Using PDFBox and IKVM] - New table: pdf - New file support: MP3s [Using UltraID3Lib] - New table: mp3 - New feature: new command-line argument: d-?OCOd-?OCOpdf - New feature: new command-line argument: d-?OCOd-?OCOmp3.

A cross-platform, highly configurable, distributed web crawling system with crawlers optimized for speed.
Methanol Web Crawling System License - BSD License.

  • Platform: WinOther
  • Publisher: Methanol
  • Date:

Some tools related to the Music Ontology - including domain-specific Semantic Web crawlers, audio collection management and mapping tools
Music Ontology tools License - BSD License.

  • Platform: WinOther
  • Publisher: Motools
  • Date:
  • Size: 50 KB

AcademiXML is markup standard to make academic (scholar, research) texts (papers, reports) syndicable, parsable, indexable and readable by both humans and bots (web-crawlers) without rendering engine (browser, app, parser, database).
academixml License - Public Domain.

The Open Dungeon Crawler 3D (ODC) is a remake of the classic 90s era 3D dungeon crawlers such as "Eye of The Beholder" and "Dungeon Master". It is initially targeted at the GameCube and desktop OSes.
Open Dungeon Crawler 3D License - GNU General Public License (GPL); Other License.

  • Platform: WinOther
  • Publisher: Opendc3d
  • Date:

integration of hylafax and web crawlers
hylafax integration License - GNU General Public License (GPL).

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own web grabber that can screen scrape the data to a database or Excel has never been easier. The website spider / crawlers can even sort and filter the webpages while spidering. Web Scraper Plus+: Web Spider Edition is the leading web data harvesting application.

SoulCraft is the best free2play Action RPG game for Android. If you have been a hunter in enough dungeon crawlers and feel like you have been a warrior for all eternity already, answer the call, fight for the torchlight of hope, do your duty and choose to be a legendary hero in the action role playing game SoulCraft :)


*** SoulCraft featured by Google for Google I/O 2012 ***
*** Handpicked game for Google booth at Mobile World Congress 2012 ***
*** our favourite user comment *This game is better than Elvis* :) ***
*** please see SoulCraftGame.

  • Platform: Android, Windows
  • Publisher: MobileBits GmbH
  • Date:
  • Size: 237568 KB

Live Deals is an application that searches for deals, discounts, coupons, bargains, etc across the web. The site is updated throughout the day by the staff and the automated crawlers. We are dedicated to providing you with the tools needed to find the best deals online..

  • Platform: Windows
  • Publisher: Athena
  • Date:
  • Size: 1024 KB

Optimized version for NVIDIA Tegra Devices

SoulCraft is the best free2play Action RPG game for Android. If you have been a hunter in enough dungeon crawlers and feel like you have been a warrior for all eternity already, answer the call, fight for the torchlight of hope, do your duty and choose to be a legendary hero in the action role playing game SoulCraft :)


*** SoulCraft featured by Google for Google I/O 2012 ***
*** Handpicked game for Google booth at Mobile World Congress 2012 ***
*** our favourite user comment “This game is better than Elvis” :) ***
*** please see SoulCraftGame.

  • Platform: Android 2.x, Android 3.x, Android 4.4, Android 4.x
  • Publisher: MobileBits GmbH
  • Date: 08-03-2015
  • Size: 165888 KB

Email address code scrambler. Fight spam. Your spam problem comes from spam crawler robots. They crawl your site to add your email addresses their spam lists. Make your email addresses invisible to spam crawlers. Hide your mail addresses from robots in plain sight. Visitors to your web pages can read and click on your email addresses the same way they do now. Robots will not find them. Stay off spam lists. Search engine friendly. Easy to use..

  • Platform: Linux, Mac, OS/2, Palm, Unix, Windows
  • Publisher: InnerPeace.Org
  • Date: 04-10-2002
  • Size: 9 KB
New Reviews


Safe In Cloud

It is quite difficult to find any computer users nowadays who do not need to use the web, for any purpose. Whether it is for education or entertainment, the web is your best resource. You also need to use various online services for banking, ...


PictBear

Nowadays, a majority of PC users resort to digital image editing once in a while. It may be required for retouching image captured in smart phones or for creating an album in Facebook. For some people, tinkering with digital images is a pastime too. ...


Panda Cloud Cleaner

PC security has become a tricky and complicated subject both for home users and network supervisors, with complexity and risk factors multiplying over the years. Using a single antivirus app may not be enough anymore. That explains why many users ...


SnapCrab

Nearly every PC users need to take screenshots from time to time, whether it is for personal or professional needs. While using the basic Windows screenshot capture method is available, it is not adequate for everyone. When you want to capture ...


BitKiller

Dealing with various types of data efficiently holds the key to success in any computing task. Apart from generating new data and modifying existing files, it becomes necessary to delete existing files at times. While most users resort to using ...


SeaMonkey, Portable Edition

Using the Internet has become a necessity for a majority of people, including those who do not use a PC regularly. To browse web, stream video or download online content, people often resort to using various third party apps. For example, using ...


TuckAway Intelligent Email Organizer Pro

Using an email software allow you to do a number of things including managing your address lists, emails and even ending memos, documents and even virtual letters. You can also use it to manage your tasks easily. The email software comes in ...


WebAcappella

Web design sector has witnessed several trends and technological innovations affect website development in big ways. With time, designing websites has become easier. Nowadays, you can find web design software that nearly eliminates the need to ...


Zoolz

No serious PC user needs introduction about the need to backup data, both at home and workplace. There was a time when majority of people used external HDD or storage media to backup important data. However, local data backup has its limitations and ...


Driver Fusion

To keep your laptop and desktop running smoothly and get the best performance out of it, it is important to pay attention to some aspects. It is quite important to ensure your PC device drivers are updated. Components like motherboards, GPU and ...


New Downloads

Terra Excess

Terra Excess is land surveying
and civil engineering CAD
software developed to be
extremely intuitive. It can be
used for ...

PC Smart Cleanup

PC Smart Cleanup is a strong
and efficient PC Optimization
and Maintenance software
solution that clean up
accumulated ...

Proxy Log Explorer

Proxy Log Explorer is the
fastest and most powerful
analysis application for
monitoring the efficiency
corporate Internet ...

Altova MissionKit
Professional Edition

Altova MissionKit 2018
Professional Edition is an
award-winning software
development suite of XML, SQL,
and UML tools, ...

Android Sync Mac

Android Sync Mac will sync
data between your Mac and
Android devices. With Android
Sync Mac you can sync contacts
and ...

FXCartoonizer

Convert photos into cartoons
with just few clicks of a
mouse with our FXCartoonizer
for windows PC. Now, you can
convert ...

Thesaurus Synonym Database

Thesaurus, Synonym, Database
for English, French, German,
Portuguese, Spanish, Italian,
Dutch, Swedish in SQL, Excel
or ...

SoftPerfect Bandwidth
Manager

SoftPerfect Bandwidth Manager
is a full-featured traffic
management solution for offers
cost-effective bandwidth
control ...

SoftPerfect RAM Disk

A RAM Disk application that
lets you have a drive on your
PC entirely stored in the
memory. As RAM is much faster
than ...

Citrus Contract Invoicer

Create upto 20 Work
Assignments and unlimited
Services.Schedule Service
Weekly, Bi-Monthly, Monthly,
SemiAnnually or ...