Sphider database

The Sphider database, like most databases, has relationships between various tables. Unlike most databases, however, these relationships have never been defined within the database! Sure, tables have had their keys, but there were no foreign key constraints. All relationships between tables and the consequences of record deletions or modifications have been handled strictly by the Sphider code. MySQL can handle this more efficiently than any Sphider code ever could.

At this point, adding foreign key constraints is really a straight forward task, since for the typical user, the tables all have data. It would be easy for new installations, but the presence of data complicates things. What has been done is to develop a method to back up the database, destroy and recreate all the tables (with foreign key constraints), and then repopulate the tables with the backed up data. The existing backup procedure was not an option because when a restore is run from that, the database would revert to one without the constraints. Then, Sphider code can be REDUCED in size by a couple hundred lines to eliminate code that handles relationship changes that MySQL can do more efficiently.

The process of building a process to back up the data, recreate the database with constraints, and restore the data has been completed. Now it has to be thoroughly tested. Early tests are positive, but we want to be sure. Once we have a high level of confidence in the process, Sphider 3.3.0-MB will be released. The intent is that the ONLY changes in 3.3.0 will be the database structure and associated code related to structure.

Look for Sphider 3.3.0-MB towards the end of July.

Sphider 3.2.0-MB will have a couple new features

The next release of Sphider, 3.2.0-MB, will feature two new enhancements. The first will be the ability to show the query score using a 0-5 star system, using half-stars. The options to either show no score, or to show a score as a percentage (100% being the highest) will remain.

Query results with relevancy shown as stars

The second enhancement is the addition of the ability to limit the number of query returns by percentage. Currently, a query will find every possible result. Scores range from 100% to 0% relevancy. In s.2.0-MB, a minimum relevancy can be set. Users can choose from 0, 20, 40, 60, and 80% relevancy as the minimum. The higher the number, the fewer results will be returned. In the example shown above, the query produced 74 results. If the floor had been set at 20%, the number of results would be reduced to 12. A floor of 40% further reduced the results returned to 7,

Along with a couple minor bug fixes, Sphider 3.2.0-MB will be coming in perhaps early June.

Sphider 3.1.1-MB corrects an indexing issue

The original Sphider was built with the English language in mind. While preparing to store indexed words, a check is performed to ensure the word is composed of alphabetic characters. Knowing that sites other than all English sites might be indexed, this check was modified to accommodate non-English but still Western characters. The function used for this was what eventually was named removeAccents(). It actually worked quite well. Sphider advanced and took on still more languages, particularly those with non-Western alphabets. Surprisingly, Sphider somehow kept chugging along, although just how is a bit of a mystery.

Well, recognizing some of the short comings of removeAccents(),  it was “improved” in Sphider 3.1.0. And it WAS improved… in some instances. Unfortunately, it was discovered (after it had been released!!  🙁  ), that Arabic words would no longer index. Other languages were probably similarly impacted, but just one is enough to raise red flags.

Rather than fix a patch that was originally intended to cover a minor flaw, it seemed logical that the initial check on the word should be updated to conform to the realities of unicode. The check to be sure the word is alphabetic now truly covers the many alphabets rather try to patch the check for an ASCII alphabet.

Sphider 3.1.0-MB and Sphider 2.4.2-PDO released

Sphider 3.1.0-MB is multibyte capable, like 3.0.0-MB. However, 3.1.0-MB does NOT require the PHP mbstring extension. Mbstring is recommended, but not required. If it is available, it will be used. If not, Sphider will emulate the mulitbyte character string functions. Also, 3.1.0-MB continues the improvements always being made to the original fork. Since there is no longer any special requirements other than the typical MySQLi/MySQLnd extensions, there is no longer a need for the 2.4.x line.

Sphider 2.4.2-PDO provides a fix for a problem with 2.4.1-PDO which could cause some UTF-8 characters to be mistaken for ISO-8859-1 characters. The resulting “conversion” produced rubbish. The PDO fork will continue to be available and supported, but no further product enhancements are anticipated.

You can run me, you can starve me, you can beat me, you can kill me and you can blackmail me. Just don’t bore me.

I get something like this in my (spam) email folder ever week or so. In fact, I’ve been getting them off and on for the last year or so… haven’t really kept track. I’ve gotten two in the last 24 hours… different bitcoin address each time.

Well, I never have paid a single cent. Really don’t have much use for bitcoin. I am STILL waiting for whoever is sending me these to just hurry up and expose me! I’m really getting bored sitting here waiting for that other shoe to drop! I REALLY want all my friends and contacts watch those same cat videos I find so funny! But videos of me…? MAN, ARE YOU GOOD! You got videos of me and my desktop doesn’t even have a camera!

Oh… all those passcodes you have… could you send them to me? Seems I’ve forgotten most of them…

… and would you PLEASE clean up your grammar? How do you expect to be taken seriously when your sentence structure is so poor?

What’s next for Sphider?

Sphider 2.4.0 is barely out the door, and thoughts are already turning to — “What next?”

There actually are some plans well in the works. Sphider 2.4.1 will be pretty low impact. The “major” change will be in sql error reporting when a statement preparation fails. At this point, an sql statement should never fail, but in the off chance one ever does, better to have a meaningful error message! A second very minor change will improve utf8 text handling.

Thought has been given to the status of the PDO edition. The fact that may people, particularly those on shared hosting, are “forced” into using PDO dictates that the edition should continue to be available. At the same time, PDO users tend to be smaller in scope and less demanding in requirements that others (who tend to be either fully hosted or self hosted). With these thoughts in mind, the PDO edition will continue to be supported and there may even be minor updates from time to time, but major updates in functionality will be discontinued.

Now… the regular/classic/legacy edition will continue on. There is a NEW fork in the works, also. Sphider was been constantly improving with the use of unicode (utf8 variety), but there is still one stumbling block. Unicode has multi-byte characters and character strings, Standard PHP string functions aren’t equipped to handle multi-byte characters/strings. The mbstring module for PHP is equipped… that’s what the “mb” part of the name means — “multi-byte”. The problem is, not every installation of PHP comes with mbstring.

For the time being, the “normal” Sphider will use standard PHP string handling functions, with the drawback that indexing and searching of multi-byte strings may be unpredictable. Sphider 3-MB has replaced all standard string handling with multi-byte string handling, with the drawback that it won’t work for all clients.

Once again, there will be two editions of Sphider — standard string handling, and multi-byte string handling. The eventual goal will be to merge the two so that if mbstring is available, it will be used. If mbstring isn’t available, some custom functions will try to achieve the same result.

Sphider 3-MB will require a MINIMUM of MySQL server 5.5.3. Recommended MySQL server is 5.6 or better. Utf8mb4 is NOT supported in MySQL server versions earlier tha 5.5.3. Sphider 3-MB will also require that PHP have both MySQLnd and mbstring installed and available. Sphider 3-MB will be available by the end of April or early May. A test script will be provided so that support can be determined before installation.

Speaking of MySQL server and utf8mb4, Sphider 2.3+, both standard and PDO, use utf8mb4, so they too require MySQL server 5.5.3+. IF you happen to have a lower version of MySQL server, and are unable to upgrade, we can provide an earlier version (2.2.0) of Sphider upon request.  (Specify standard or PDO.)

Emojis revisited

Not very long ago, I wondered whether or not Sphider still needed to scan for, and remove, emojis. This came about because of a change in the database from 3-byte utf to 4-byte. Upon testing, the scan and removal of emojis will continue. Sphider, and more particularly, MySQL, just doesn’t like emojis. When trying to store any full text containing an emoji, an SQL exception is thrown and the page is not stored.

The earlier issue with the function that was reported is due to the removeEmoji() function operating on an utf-8 level, and the probability of the input NOT being utf-8. For future releases of Sphider, this function will be executing AFTER it is (nearly) guaranteed that the input will be utf-8. (I say “nearly” because there are no guarantees in this world where code is involved.)

It was also noted that the function, as currently implemented, is somewhat outdated.  While updated it is possible, the function would become a bit  unwieldy.  Leaving it alone is practical, however. This is because pages containing emojis are, while not rare, relatively uncommon. And withing the pages that DO contain an emoji, the odds are that emoji is of the simpler, more common type. The kind an expanded filter would catch ARE rare in web pages, being more likely to occur in messaging applications used in smart phones and tablets. In other words, why add the complexity to Sphider to catch something that the vast majority of users are never going to encounter?

Maybe someday I will once again update the database collation to use utf8mb4_unicode_ci as opposed to the current utf8mb4_general_ci, which should allow these emojis, but even if I do, there will probably be a setting to exclude them anyway.


Content-Type meta tags and HTTP response headers

How many of us have used a meta tag to define content type and default character sets? The tag may appear something like this:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

But do we REALLY understand what is going on? This tag is important when a webpage is being opened locally. It instructs the browser as to what character encoding to use to display the page. This may override the platform default.

But what about when a page is being viewed by HTTP? Well, the tag is important if the HTTP response header(s) being sent fail to designate a default character encoding. What if the response header(s) DO include a default character set? AHHH! The the meta tag is (are you ready for this?)… IGNORED!

Let’s say you designate a page, via meta tag, to have a character set of UTF-8, but your web server is sending a response header setting the default as Windows-1252. Your page is going to display in Windows-1252!

And guess what? Your page, viewed over HTTP,  just may still appear correctly giving you the impression that the meta tag is working! Then you force your browser to actually display in UTF-8 and that beautiful page suddenly becomes what is referred to as “mojibake!”

There are at least a couple ways to get this all sorted out. If you are coding in PHP, one way would be to set the response header in the code for each page. Here is an example PHP header:
header('Content-Type: text/html; charset=utf-8');
This needs to appear in the PHP BEFORE a single bit of HTML is displayed.

Another way is if you have access to your server settings, you can specify a default character set.

Still another way, with Apache servers, is to specify a default character set in you .htaccess file.
AddDefaultCharset UTF-8

So…. knowing all this, just HOW do you go about confirming that the character set you want is the character set actually being set? With Firefox/Waterfox/SeaMonkey, bring up the page in question. Up in the url display area, to the left of the url, click on the little circle with the upside-down “!”. There will be information on whether or not the connection is secure, then a “>”. Click on that. Click on “More information”, the the “General” tab.  This will display the text-encoding AND the meta tags. If they don’t agree, the response header being sent isn’t what you want it to be. This applies to Waterfox in Linux, also.

Google Chrome USED to allow the option to see what the default character set REALLY is, but they removed it. Fortunately, there is an extension that does it for you. The extension is simply named “Charset”, and allows you to not only see what the actual character set is for a given page, allows you to change it. The results may be an eye opener. BTW, this applies to Linux Chromium as well.

What about IE/Edge? You’re on your own! I won’t touch those monstrosities! LOL!!

The future of the PDO edition of Sphider…

Sphider comes in two editions, the legacy version and a PDO version. The legacy version is definitely the more stable, faster, easier to maintain version. The PDO version exists primarily for those who are restricted by their shared hosting providers.

Shared hosting has its advantages in that it is very cost effective (cheap) and very simple to use. It is great for personal use or for small businesses or organizations just getting started on the web.

But shared hosting has its downsides, too. It isn’t nearly as efficient, isn’t as secure, suffers from limited resources, and has limited functionality. One of the features commonly lacking in shared hosting is MySQLnd. Thus the need for PDO.

The are quite a few users of the PDO edition, and to simply drop PDO would be a great disservice. On the other hand, trying to keep the PDO edition in sync with the legacy edition is getting harder and requiring much time and effort.

The PDO version, as it stands, is quite usable. It is PHP 7.3 compliant, so it should be reasonably set for awhile, as the majority of shared hosting plans are still at least a few versions behind 7.3!

The thought is that the time for legacy and PDO to part paths, with most future effort going into the legacy edition. Because of the user base, PDO version 2.4.0 would remain and receive hot fixes as needed.

No decision has been made and feedback will be given consideration.