Getting Started With MySQL's Full-Text Search Capabilities

Getting Started With MySQL's Full-Text
Search Capabilities

(Page 1 of 6 )

Need a
rock-solid, powerful search solution for your PHP/MySQL driven web site?
In this article Mitchell introduces us to MySQL's full-text and Boolean
search capabilities.In February 2002 I wrote an article called "Developing A Site Search Engine With PHP And MySQL
".
In this article I showed you how to build a "search engine


"
using a keyword based approach. Notice how the words search and engine
are in quotes in my last sentence? That's because it wasn't actually a
search engine in the right meaning of the word. It was based on the
concept of keyword searching.

Today I'm going to show you how to
use MySQL's extremely powerful full-text search capabilities, which you
can use to build your own simple search engine. In this article I will
assume that you have an intermediate knowledge of MySQL.

I will
also assume that you have at least MySQL 3.23.23 installed (which is
when support for full-text searching was added.). You will need MySQL
version 4.0 or above to test the Boolean search methods described later
in this article.

Getting Started With MySQL's Full-Text
Search Capabilities - What is Full-Text Searching?

(Page
2 of 6 )

Imagine having a database that contained 10,000
tables. In each of these tables there are 1,000 rows with 100 fields.
How would you effectively search this sort of information structure
without killing your web server? The answer is MySQL's full-text search
capabilities.

A full-text search makes use of indexes, which you
can define against a table either when it is created or by using
MySQL's ALTER TABLE command. These indexes are setup on specific fields
of a table and change the way that MySQL stores the records for that
particular table.

For example, let's say that you have a table
to store the title, price, description, availability and picture of some
computer products. When users perform a search, they will most commonly
enter part of the products title or description.

If we setup an
index on the title and description fields in our MySQL database, then
we can make MySQL store the records for the products table in an indexed
format, meaning that when a user performs a search, MySQL can retrieve
related records quicker, because it has already indexed (or ordered)
them.

Full-text searches are faster than other search methods
such as wildcard or character based searches, which are commonly
performed using MySQL's LIKE command.

Besides being
significantly quicker than normal character based searches, why would
you want to use full-text searching? Here's a quick list of points to
wet your appetite:

Full-text searching is ideal for extremely
large databases that contain thousands or even millions of rows.
Computations are performed faster and rows can be ranked based on search
relevance, which is returned as a decimal number by MySQL.

Noise
words and any words that are 3 characters or less in length such as
the, and, etc are removed from the search query. This means that more
accurate results are returned. If you searched for "as the people", then
the noise words "the" and "as" will automatically be removed from your
query.

In addition to simple searches, full-text searches can
also be performed in Boolean mode. Boolean mode allows searches based on
and/or criteria, such as "+person +Mitchell", which would only return
all records that contained the words person AND Mitchell. We will look
at Boolean searches later in this article.

The query is
case-insensitive, meaning that "cat" is ranked the same as "Cat", "CAT"
and "cAT".

As you can see, full-text searching is fast,
powerful and smart. It eliminates the need for us to write complicated
search and Boolean algorithms, and you can be up and running with
MySQL's full-text searching and indexing capabilities in under 5
minutes.

Your First Full-Text Search

Let's jump right
in and see full-text searching in action. First off we need to create
our database, so fire up the MySQL console window and create a new
database called testDB, like this:

create
database testDB:

use testDB;

Next, we need to create
a table using MySQL's FULLTEXT command, specifying which fields we want
to index for searching:

create table
testTable

(

pk_tId int auto_increment not null,

firstName
varchar(20),

lastName varchar(20),

age int,

details text,

primary
key(pk_tId),

unique id(pk_tId),

fulltext(firstName,
lastName, details)

);

What we've done here is
create a basic table that will store the details of some people. The
first field is a uniquely identified primary key and the rest are simple
varchar, int and text fields.

If you already have a table
setup and want to index existing fields, use the ALTER TABLE command
like this: ALTER TABLE myTable ADD
FULLTEXT(field1, field2);

Take a look at the highlighted
line in our CREATE TABLE command, shown above:

fulltext(firstName, lastName, details)

This
line tells MySQL to set an index on the firstName, lastName and details
fields of our table. Indexes can only be created on fields of type
VARCHAR and TEXT. Because these fields contain indexes, we can now use
the power of a MySQL full-text search to find records in this table
based on the values in these 3 fields.

Before we can perform a
full-text search however, we need to add some records to our table with
the following MySQL commands:

insert into
testTable values(0, 'Mitchell', 'Harper', 20, 'Mitchell is the founder
and manager of devArticles and various other sites across the SiteCubed
network');

insert into testTable values(0, 'Ben', 'Rowe', 19,
'Ben is our Flash guru. He is currently writing a series of Flash
articles that show beginners how to use Flash to create dynamic content
with PHP');

insert into testTable values(0, 'Havard', 'Lindset',
19, 'Havard is a PHP and MySQL power user');

insert into
testTable values(0, 'Michael', 'Manatissian', 25, 'Michael is the senior
devArticles editor and is a capable Linux/Apache administrator');

insert
into testTable values(0, 'Sandra', 'Lee', 23, 'Sandra is our finance
and accounting wizard, having 4 years of experience under her belt');

Because full-text searching was designed for larger databases,
it is possible for MySQL to return incorrect results when it's used on
tables containing smaller amount records, that's why I've created 5
"chunky" records for our test table.

Now that we have records in
our table, how do we search them? Simple -– we create a query to invoke
MySQL's full-text search feature, like this:

select firstName from testTable where match(firstName,
lastName, details) against('guru');

Getting Started With MySQL's Full-Text
Search Capabilities - What is Full-Text Searching (contd.)

(Page 3 of 6 )

This query returns the following result:



Let's
analyze our query. First up, we have the SELECT and FROM parts of our
query:

select firstName from testTable

There's nothing unusual about this part of the query: it simply
tells MySQL to retrieve the firstName field from the table called
testTable. Next up, we have the WHERE clause:

where match(firstName, lastName, details)
against('guru');

This is where the power of full-text
searching starts. In the first part of the query we call the MATCH
command, which tells MySQL to match against the values of the firstName,
lastName and details fields when it performs a full-text, natural
language search.

When the MATCH command is used as part of the
SELECT clause it returns a relevance ranking, which is a positive
decimal number. The closer to 0 this number is, the less relevant the
record is. This relevance value is determined based on the search
expression, the number of words in the indexed fields, as well as the
total number of records being searched.

Lastly, we have the
AGAINST command. AGAINST is simple enough and accepts just one
parameter, which is the string that we're searching for. Later in this
article we will see how to perform boolean searches using the AGAINST
command in combination with the IN BOOLEAN MODE keywords.

So far
so good, right? But how does a full-text search differ from using the
LIKE command in a query like this:

select
firstName from testTable where details like '%guru%';

...
they both return the same result don’t they? Well yes and no. Let's now
see how we can determine a relevance ranking for each of the records
returned from a MySQL full-text search.

Determining A
Relevance Ranking

Still working with our test table, take the
following query into consideration:

select
concat(firstName, ' ', lastName) as name, match(firstName, lastName,
details) against('devArticles') as relevance from testTable where
match(firstName, lastName, details) against('devArticles');

This
query produces the following results:



In this query I've included the MATCH command in
the SELECT clause to return a relevance ranking for each record. The
query performs a full-text search against the firstName, lastName and
details fields for the string "DevArticles":

where match(firstName, lastName, details)
against('devArticles');

The query has returned two
records, which contained the string "devArticles" in either their
firstName, lastName or details fields. Looking back at our records, we
can see that the records for Mitchell and Michael contained the string
"DevArticles" in their details fields:

'Mitchell is the founder
and manager of devArticles
and various other sites across the
SiteCubed network'

'Michael is the senior devArticles

editor and is a capable Linux/Apache administrator'

The
relevance field returned from our query was generated with the following
expression:

match(firstName, lastName,
details) against('devArticles') as relevance

Looking
carefully at the query, we can see that this expression has been used
twice: once in the SELECT clause and once in the WHERE clause. MySQL
picks up on this and only performs one full-text search on the table and
not two, meaning that there is no additional overhead produced for a
query like this –- a huge bonus and time saver if a similar query was
performed on a table with 5 million rows.

When the MATCH command
is used in the WHERE clause, MySQL automatically sorts the rows from
highest to lowest relevance. In our previous example query we only
returned records that actually had a match against the string
"devArticles" when a full-text search was conducted. Here's a query that
returns the relevance ranking for every record in our table:

select concat(firstName, ' ', lastName) as name,
match(firstName, lastName, details) against('devArticles') as relevance
from testTable;

We've left out the WHERE clause, so the
resultant records are unordered, as we can see below:

Getting Started With MySQL's Full-Text
Search Capabilities - Full-Text Rules And The MATCH Command

(Page 4 of 6 )

Remember earlier when I listed a number of
bullet points, one of which stated that MySQL removes noise words and
those of less than 3 characters. Let's test that theory will 2 basic
full-text search queries:

select
firstName, match(firstName, lastName, details) against('devArticles is
on the www') as relevance from testTable;

This query
returns the following records:



Notice how our last query only had one word that
was longer than 3 characters in lengh, "devArticles". If we remove all
words of 3 characters or less from the search string then the relevance
ranking will remain the same:

select
firstName, match(firstName, lastName, details) against('devArticles') as
relevance from testTable;

Here is the list of records
that matches the search:



As
we can clearly see, the relevance ranking remains the same -– MySQL
does indeed remove noise words and those words with 3 characters or
less.

MySQL's full-text search ranks words based on their
semantic values -- common words rank lower than uncommon words. This
makes sense, as a word that exists in many records will have a lesser
relevance to a word that only appears in 1 or 2 records. Semantic word
rankings are used in most popular full-text searching algorithms.
Popular search engines and directories also employ this method.

The
50% Threshold

MySQL removes noise words and short words, but if
a word is present in more than 50% of the records being searched, then
those records will not be returned. MySQL calls this the "50%
threshold". In a way this makes sense, as it filters out records that
have a low relevance.

Here's one of the comments from a MySQL
user on their site:

"... you should add at least 3 rows to
the table before you try to match anything, and what you're searching
for should only be contained in one of the three rows. This is because
of the 50% threshold. If you insert only one row, then now matter what
you search for, it is in 50% or more of the rows in the table, and
therefore disregarded.
"

Getting Started With MySQL's Full-Text
Search Capabilities - Performing A Boolean Search

(Page
5 of 6 )

MySQL version 4.0.1 and above can perform complex
full-text Boolean searches. I am using MySQL version 4.0.1, which is
still in the Alpha stage. If you want to perform Boolean searches then click here
to download MySQL version 4.0.

So
what exactly is a Boolean search? Put simply, it is a powerful way to
include or extract words and phrases from your search criteria. If
you've ever searched for something like "+download +games", then you've
used a Boolean search engine.

By combining various operators
within your search string, you can filter in/out other words, change a
words contribution to the relevance value and more. Here's the list of
Boolean operators, as listed at MySQL.com:

+ A leading plus sign
indicates that this word must be present in every row returned.

-
A leading minus sign indicates that this word must not be present in
any row returned.

By default (when neither plus nor minus is
specified) the word is optional, but the rows that contain it will be
rated higher. This mimics the behaviour of MATCH() ... AGAINST() without
the IN BOOLEAN MODE modifier.

< > These two operators
are used to change a word's contribution to the relevance value that is
assigned to a row. The < operator decreases the contribution and the
> operator increases it.

( ) Parentheses are used to group
words into sub expressions.

~ A leading tilde acts as a
negation operator, causing the word's contribution to the row relevance
to be negative. It's useful for marking noise words. A row that contains
such a word will be rated lower than others, but will not be excluded
altogether, as it would be with the - operator.

* An asterisk
is the truncation operator. Unlike the other operators, it should be
appended to the word, not prepended.

" The phrase, that is
enclosed in double quotes ", matches only rows that contain this phrase
literally, as it was typed.

A Boolean search is performed in
much the same way as a normal full-text search, however it includes the
IN BOOLEAN MODE keywords, as shown in the example below:

select * from testTable where match(firstName,
lastName, details) against ('+DevArticles -flash' in boolean mode);

In the example query above I'm specifying that every row
returned must contain the word "DevArticles" but must NOT contain the
word "flash". Still working with our test table, which record do you
think will have the highest relevance ranking if we executed this
query?:

select firstName,
match(firstName, lastName, details) against('+mysql -devArticles
>flash' in boolean mode) as relevance from testTable where
match(firstName, lastName, details) against('+mysql -devArticles
>flash' in boolean mode);

Looking at the bulleted list
above, the results of this query will be ranked higher if:

It
contains the word "mysql"

It doesn’t contain the word
"devArticles"

It contains one or more instances of the word
"flash"

Here is the result from the query:



How
about this query:

select firstName,
match(firstName, lastName, details) against('+devArticles -sitecubed' in
boolean mode) as relevance from testTable where match(firstName,
lastName, details) against('+devArticles -sitecubed' in boolean mode);

In this query, records that contain the word DevArticles and
NOT the word siteCubed will rank. The results are shown below:



All
of the other records returned a 0 relevance ranking, and are therefore
not shown in the results. The best way to become familiar with Boolean
full-text searches is to experiment with the various operators,
including where they are positioned in the search string (including the
use of brackets) and how often they are used.

Getting Started With MySQL's Full-Text
Search Capabilities - Conclusion

(Page 6 of 6 )

In this article we've seen how to setup and use MySQL


full-text and Boolean searches. Just to reiterate, here are the steps
you need to take to get full-text working on your database:

1.
Create/alter a table, making sure a call to the FULLTEXT command is
issued when creating/altering the table.

2. To perform a
full-text normal search, use the MATCH and AGAINST commands in your
query for fields that have been marked as full text, like this:

SELECT *, MATCH(field1, field3) AGAINST('my_query') as
relevance FROM myTable WHERE MATCH(field1, field3) AGAINST('my_query')

4. To perform a full-text Boolean search, use the MATCH and
AGAINST commands in combination with the IN BOOLEAN MODE keywords in
your query for fields that have been marked as full text, like this:

SELECT *, MATCH(field1, field3) AGAINST('my_query' IN
BOOLEAN MODE) as relevance FROM myTable WHERE MATCH(field1, field3)
AGAINST('my_query IN BOOLEAN MODE')

Using the examples
and information provided in this article you should now be able to
create a basic full-text search against your database. Simply use a
scripting language such as PHP to gather a search query from your users.
Next, pass this string to your MySQL query as the parameter to the
AGAINST command. Lastly, iterate through the results, displaying the
relevance ranking if desired.

原文地址：http://www.devarticles.com/c/a/MySQL/Getting-Started-With-MySQLs-Full-Text-Search-Capabilities/