Category: Tech

The original version of Sphider had very erratic support for indexing HTTPS pages, and wouldn’t even look at the robots.txt file on a HTTPS site. That failing has never been addressed, and even the latest version, 1.5.2, has the same failings when it comes to HTTPS. This has never really been an issue for me before, and even now it is more annoyance than issue as I can work around it.

Still, the “problem” does seem intriguing. After a bit of experimenting, a fix may not be all that difficult. (Famous last words, right?)

I am debating now whether or not to continue investigating alternatives and make more code changes which would improve HTTPS support in Sphider, not only to ensure more reliable connectivity but to enable the robots.txt to be utilized as well. I don’t know that there is that big of a need. We’ve never received any complaints or comments on the issue…

Anyway, at this point there is a POSSIBILITY, but no definite plans one way or the other.

*******************************

UPDATE (Apr 6): I was able to get the robots.txt file read from a https site. First problem, regardless of http or https, the parsing of allowed or disallowed user agents and disallowed files/directories was iffy. If the robots.txt file had lines like “user-agent” or “disallow”, it was parsed, but “User-agent” or “Disallow” was not. It was a case issue. That is now fixed (on my side, not published yet). Second problem, now that I know the file IS being read and parsed, Sphider will STILL index some files in disallowed directories!

If you have any files or directories listed as “url_not_inc” in your settings, that will work, but not the robots.txt disallows, even though that SHOULD be the case. Well, this situation certainly has gotten my interest!

*******************************

UPDATE (Apr 7): I have begun the process of troubleshooting the code to see what is going awry and where. Working alone and having other things to do in life, this can be both time consuming and frustrating. So far, I do know the robots.txt is read and parsed properly. Just where and why the instructions are not acted upon is another matter. At least the question of whether or not I will be attempting another modification has been answered!

*******************************

UPDATE (Apr 8): GOT IT! Preliminary tests show robots.txt is now being followed in both http and https. More testing to follow (found a couple other misc issues and fixed them). Once everything is validated, there will be a 1.5.3. Stay tuned.

The newest version(s) of the Sphider search tool have been released and are available from the Downloads tab above. While there isn’t really anything NEW in these releases, they do address a couple of problems encountered. Of most importance, the problem of having Sphider exit during indexing due to web page coding errors on the site being indexed has been addressed. Instead of issuing a fatal error and stopping, only warnings are generated and indexing continues on its merry way. A potential database error when updating the settings has also be thwarted.

Also, the previous PDO version had a bug in which descriptions could disappear from search results listings. This has been fixed.
If you had the previous PDO version (1.5.1.1) and have lost the descriptions, after upgrading to 1.5.2.1, you will need to restore the descriptions by going into the settings tab, go down to the “Search settings” section where it says “Maximum length of page summary displayed in search results”, change the selection to 250 and “Save settings”. (Updating the settings before would change this from the default 250 to either 0 or 1!)

Happy Holidays and Happy indexing!

The next version of the Sphider search tool is now in testing. Sphider 1.5.2 (and its companion PDO version, 1.5.2.1) is not very different from the previous version, except for a couple minor fixes on the Settings tab and the fact that the indexing portion has been toned down to issue warnings only when an improperly coded web page is encountered. Sphider 1.5.1 exits with a fatal error instead of continuing to index the site. While improper coding in a web page (commonly having to do with some off beat special character the database has no idea how to interpret) is rare, it sure was a monkey wrench when it came to indexing a web site. A couple other page conditions which could have produced a fatal exit now simply issue warnings (like the url exceeding the length the database could store).

At any rate, both the PDO and non-PDO varieties are now being tested to make sure the intended fixes work properly, and that we haven’t introduced any new problems. Expected arrival at this time is early December.

Sphider 1.5.1 has proven to be a good, stable version of Sphider. HOWEVER, it seems some people can’t use it because their host chooses not to support MySQLnd, typically for shared hosting. It isn’t because it can’t be done, but because they don’t want to do it. In those instances, if you want MySQLnd, you to have to upgrade to VPS, at an additional charge of course. Sphider users in that scenario now have an option.

We have taken Sphider 1.5.1 and converted the sql to PDO (PHP Data Objects). PDO support is virtually guaranteed. The PDO version is referred to as Sphider 1.5.1.1. PDO has some advantages over MySQLi/MySQLnd, but there are also disadvantages.

MySQLi/MySQLnd is SPECIFIC to a MySQL database, where PDO is a generic supporting a variety of databases, one of which is MySQL. There is an overhead involved. For Sphider, we STILL consider the MySQLnd prepared statement methodology over PDO prepared statements. Reality dictates a PDO version be made available. Our recommendation is that you install the PDO version only if the standard MySQLi/MySQLnd option is not available. If you already have a working Sphider 1.5.1, DO NOT install 1.5.1.1.

One issue encountered was that PDO has no need to use the real_escape_string function…. EXCEPT WHERE IT IS NEEDED!!! The backup and restore functions failed without it. All research indicated “You don’t need real_escape_string, just use PDO prepared statements!” Dogmatic statements like that can come back to bite you. Well, our scenario wasn’t executing sql, it was CREATING sql, specifically, an sql string. Real_escape_string was necessary to create a valid string, and a prepared statement was not possible. We had ALREADY run a query, now we were manipulating the queried data to create a string for LATER use in a different kind of query. So we had to create an emulation for real_escape_string, which was a bit of trial and error. So much for “PDO NEVER needs real_escape_string”.