What to expect in Sphider 2.4.0

Sphider 2.4.0 is on track for an April 10th release. For the user, the changes are focused on cosmetics. Up until this point, search results ALWAYS had a result number and, after the description, a text url to the page containing the search result. In 2.4.0, you will have the option to either display or not to display those items. Also, the option to display the page’s indexing date has been added.

As to search templates, what were probably seven of the crappiest, lamest templates to have ever seen the light of day have been scrapped. Seven NEW templates are being introduced. Depending on your tastes, you might consider some of them crappy, too, but at least they have a bit of style to them. The “newspaper” template was introduced in an earlier post. Here are the other six:

“black” template
“green” template
“grey” template
“simple” template
“terminal” template
“yellow” template

The “green” style is, well, VERY GREEN! The purpose isn’t so much for actual use as to demonstrate the ability and flexibility of CSS in creating your own templates, even using an image as a border.

The “yellow” template features a bit of simple artwork in the upper left corner. This artwork is “logo.png”, located in the templates/yellow directory. The size is 150×150 and has a transparent background. By creating your own similarly sized logo/picture/artwork, and replacing “logo.png”, this template can be customized for your website.

Since everyone has different tastes, different needs, and every website is somewhat unique, these templates can serve as guides in customizing your own templates. With all the above, the ONLY thing different is the CSS.  Start with a copy of the “standard” template and start tweaking away! The basic Sphider modules remain the same.

Additionally in Sphider 2.4.0, the ‘settings’ table has been completely reworked. While this change is transparent to the user, it will make life much easier on the developer as Sphider moves forward.

Besides some minor fixes and tweaks, the only other big change is in the word stemming process. While the majority of Sphider users probably never use word stemming, those who do will be pleased to learn that the algorithm (for English) has been updated to Porter2. Completely new is the ability to use stemming for ten other languages!

The next Sphider is in the pipeline

Sphider 2.3.1 is brand new, but work has already begun on 2.4.0.

Among the features already being implemented are the ability to hide the result number when displaying search results. Also, for the regular text search, the option to display the index date is being added. (This will not be available for the image or RSS searches.) The RSS and image searches will have the option to turn off the advanced search features.

A new template is being added. Unlike nearly all the current templates, this one has some class. Here is a screen shot:

The Newspaper template

In the sample above, in “settings” the result number is turned off, the index date is turned on, and the description length has been increased to 1000.

Probably the biggest change will be transparent to the user. The “settings” table is being reworked. As Sphider has changed, so has the table, with new columns being appended on a regular basis. Now, while the position of columns within a table is totally immaterial to functionality, after awhile it can be really confusing for the developer having to bounce all over the place to gather data.  This change will organize the data in a regular flow which will be much easier to maintain going forward.

Other improvements are also being considered, but whether or not they are implemented at this time is yet to be determined. No release date has been set.

When 2.4.0 is released, whenever that may be, the downloads for the SQLite and PostgreSQL versions will likely be removed due to lack of demand.

Also, earlier thoughts of adding audio (mp3, wav, ogg) indexing support to Sphider have been dropped, also due to lack of demand. The actual indexing algorithm has been proven and sketched out, but there is no rationale for implementing it other than “Gee, that’s a neat feature.”

Sphider 2.3.1 Released

Sphider 2.3.0 principally addressed security concerns, but it also was intended to bring Sphider into PHP 7.2 compliance by removing any use of the deprecated each() function. The function was used extensively, and the majority of the code replacement was very run-of-the-mill straightforward. There were four times the usage was atypical. Substitute code was put in place and tested. It seemed all worked well as many sites were indexed and searches performed as expected.

Well! It seems indexing and searching was being done properly — but only for words composed of Western characters. Words utilizing non-Western characters were not being indexed! And any searches for those words not only returned as “not found” (expected since they weren’t indexed), those searches also complained of gibberish characters/words being either too short or too common.

Investigation of the issue led to three of the four code segments replacing the non-standard usage of the deprecated each() function. The code replacements themselves have been replaced in 2.3.1. Testing on the problem sites now shows that all words are being indexed, those containing Western characters as well as those containing non-Western characters. The search anomalies are gone and searches for non-Western foreign languages is yielding expected results.  If a search word really IS too short or too common, it is reported as such, and not as gibberish. Sphider is now truly PHP 7.2 compliant.

Sphider 2.3.1, both legacy and PDO, are available for download on this blog’s download page, or from the Sphider Home page.

Sphider – PDO vs MySQLi

There are TWO editions of Sphider… the classic edition using MySQLi and the PDO edition.

Why are there two versions? The classic edition uses MySQLi and prepared statements. While MySQLi, by itself, does support prepared statements, there are a couple functions used in Sphider that require MySQLnd (the “nd” stands for “native driver”). These functions are used because they are the most efficient way of doing things.

MySQLnd has been the default driver since PHP 5.4. If you install a modern version of PHP and want MySQLi, you are going to get MySQLnd. Yet SOME hosting companies DISABLE MySQLnd for those using shared hosting. (I suppose they want people to shell out a few more bucks to get VPS or Dedicated hosting.) In those situations, the classic edition just ain’t gonna work! So, there is the PDO edition.

There are those who will tell you that PDO is what you should be using anyway. They will tout how versatile PDO is, how it can do anything MySQLi can do, only better. It is true the PDO IS versatile. It can work with many different databases, not just MySQL. But there ARE some things PDO just can’t do, at least not efficiently. And there is overhead. And memory requirements.

With PDO:              PHP <==> PDO <==> Your data
With MySQLi:       PHP <==> Your data

The classic version of Sphider is the better, more capable edition! The PDO edition is capable enough PROVIDED you aren’t trying to build your personal version of an internet search engine. It IS possible to tax the PDO edition to the point it chokes. (It is probably possible to choke the classic edition as well, but it takes more effort.)

Remember, the intent of Sphider was/is to index a web site for the benefit of that site’s visitors. In can be used to index a number of related sites for the same purpose. An individual may stretch Sphider for personal use to index MANY sites… but it is STILL just a small indexing tool and not a Google replacement!

NOW… the final point. If you REALLY need Sphider to stretch its capabilities to the absolute limit, maybe you should be using the classic edition and not PDO. If that is the case, shell out a couple extra bucks to your host so you can get access to MySQLnd. Don’t try to pull a 20′ travel trailer with a Honda Civic.

Securing your Sphider

You may have read before that I have suggested securing the Sphider admin directory with password protection. After all, you don’t want to take a chance of someone else gaining access and changing all of your settings… or worse!

But Sphider has a number of other directories within it. What about those? Well, MOST of them can also be password protected. The rule is, if files in the directory only need to be accessed by PHP, it can be password protected. If any file in a directory needs to be accessed by a web browser, it shouldn’t be password protected. For example, the templates directory should not be password protected. It contains css files, which the browser needs to display search forms and results properly. Js_suggest should not be password protected, It contains javascript which the browser needs to access. The tmp directory (not the one in admin) should not be password protected as the browser needs to be able to read and write there. Other than that, go ahead and add password protection.

Oh! One more thing. Add SSL. That is something that at one time was expensive and primarily used by sites like businesses with checkout pages and such. Today, SSL in some cases can be free. DreamHost offers “Let’s Encrypt SSL” for free. Then there are self-signed certificates, also free (but not as trustworthy). The advantage of SSL is that when you do need to enter user name/password to one of your password protected directories, it can’t be intercepted.

Typo in Sphider 2.2 install script

A typo has been discovered in ‘install.php’ for Sphider 2.2.0 (classic). This will affect those who have 1) already downloaded ‘sphider-2.2.0.zip’, and 2) are trying to do a clean install. Any downloads after right now already have the file corrected.

This does not warrant a new release, but simply instructions on how to fix the problem if you have it.

Line 23 currently has:


Change this to:

$settings_dir = “../settings”;

This does NOT affect the PDO edition.

Sphider 2.2.0 released

Sphider 2.2.0 was released today in both the Classic and PDO editions.

There are currently no plans to release this update in PostgreSQL or SQLite editions. Those remain at 2.1.0. There seems to be little demand for these.

Sphider 2.2.0 has the search interface redesigned to be more mobile friendly. The code was rewritten to be more responsive to CSS instruction, made to detect the use of mobile browsers, and present a mobile friendly version of the standard template.

A minor security problem was also addressed, as well as the ability of the administrator to easily switch between the tabbed search (traditional, RSS, and image) and just the basic traditional text search.


There are plans for the next release (February 2019 ???) to expand this ability so that the administrator can pick and choose which search interfaces to present. If they want both traditional and image, but not RSS, they will be able to do so.

Also, the templates will be reworked so mobile browsers can experience friendly versions of ALL the templates.

Sphider 2.2 is in the works

The subject(s) has come up recently about the use of html tables, css, and mobile browsers.

Looking at the Sphider search pages, there is room for improvement. So work has begun to alter the code to do away with html tables and expand the use of css. The result is that the appearance of the search page layouts using css is nearly identical to look using html tables.

The kick is, it makes it easier to customize the look from template to template. Even a new template has come about. Named “mobile”, it is much easier to see/use on a mobile device. The search page will also detect the use of a mobile and force the mobile template even when it is not the template set in the configuration and seen by anyone NOT on a mobile.

The changes have already been made and will now undergo a period of testing. Most likely, Sphider classic will see the new version first, with PDO a short time later. If this will also carry over to the PostgreSQL and SQLite variation has not yet been decided.

Core functionality will have no change. This new release will be for benefit of the user’s clients, the people who actually use the search feature.


To get an idea how the mobile template will work, first visit
with a desktop or laptop. Then visit the same link with a mobile device.

Feedback appreciated.

Sphider 2.1 released

It was found that all variations of Sphider 2.0.* had a flaw in the search capability. While earlier versions allowed wild card searches (*), or not contained searches (-), this ability was inadvertently blocked in 2.0! It seems the work to improve security and block undesirable searches went too far!

This is resolved in all variations of 2.1 – Classic, PDO, PostgreSQL, and SQLite.