Sphider is a program designed to visit a web site in an ordered fashion to find the information necessary to create an index for a search engine. This, in turn, allows the site to be searched for pages containing certain keywords or phrases. Spidering programs are also called web crawlers or bots. They operate by following the hyperlinks on each page.
The crawlers which build major internet search sites (Google, Yahoo, Bing, etc.) are quite sophisticated and can find not only keywords and phrases, but images and other content as well. The ranking system of these crawlers is equally sophisticated. Not only are keywords, considered, but so is keyword location and density, relevancy, traffic patterns, tld names, page design, and domain registration length. In fact, Google has a list of over 200 page ranking factors.
Sphider is much simpler. Pages are ranked solely on keyword weighting. Keyword weighting is calculated by word position and frquency and the user has a level of control over the weighting process. Images are not indexed and relevancy is not a factor (although better word position and greater frequency DO indicate higher relevance). While Sphider can index practically any website, the main purpose of the application is for the user to index his or her won website so that an internal search can be made available to site visitors.
There are a number of Sphider flavors. The original Sphider (version 1.3.6) can be found at http://www.sphider.eu. It is free, but has the disadvantages of being insecure and badly outdated. It is no longer maintained and will start throwing errors on any system running PHP 6.6 or greater. It will not function at all on PHP 7.
Sphider-Plus (http://www.sphider-plus.eu) and Sphider-Pro (http://www.sphiderpro.eu) are both paid versions of the original and do have added features. I cannot speak as to security or support. Sphider-Pro is at version 3.3, which has a date of 2013, so that may not speak well as to its status. For a small website, many of the enhancements provided by these variations may be overkill.
Then there is the Sphider located here on our Downloads page, It, too, is based upon the original, but has been updated. It functions without error with PHP 5.5 or greater, even with PHP 7. It is much more secure. All SQL queries are made using prepared statements to avoid the risk of SQL injection. Other security measures have also been taken. We even have a variation (PDO) which can not only operate in environments lacking MySQLnd support, but can be used with databases other than MySQL (with some tweaking). It can work with SQLite, PostgreSQL (port kits available for both), ODBC, Microsoft SQL Server, and others. Both the normal and PDO variations are supported. And best of all, they are still free!
