MySQL千万级别表数据中提高RAND随机查询的实验

1,准备测试数据

 

1.1 建库建表

mysql> create database hwdb;

Query OK, 1 row affected (0.34 sec)

 

mysql>

mysql> show create table t_huawei;

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Table   | Create Table                                                                                                                                                                                                                                       |

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| t_huawei | CREATE TABLE `t_huawei` (

 `hwid` int(11) DEFAULT NULL,

 `content` varchar(4000) DEFAULT NULL,

 `TYPE` char(2) DEFAULT NULL,

 `creator_id` varchar(30) DEFAULT NULL,

 `create_time` datetime DEFAULT NULL

) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1 row in set (0.00 sec)

 

mysql>

 

1.2 准备存储过程

DELIMITER $$

 

USE `hwdb`$$

 

DROP PROCEDURE IF EXISTS `pro_insert_hw`$$

 

CREATE DEFINER=`root`@`localhost` PROCEDURE`pro_insert_hw`(num INT)

BEGIN

    DECLARE var1 INT DEFAULT 0;   

   WHILE var1<num DO   

   SET var1=var1+1;   

  INSERT INTO hwdb.t_huawei(hwid,content,TYPE,creator_id,create_time)  

   VALUES (var1,CONCAT(var1,' - ','Players supposedly said they''d avoidteam facility this summer if Thibodeau remained the '),

0,10013,NOW());   

   END WHILE;    

 

 

   END$$

 

DELIMITER ;

 

 

mysql> show create table t_huawei;

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Table   | Create Table                                                                                                                                                                                                                                       |

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| t_huawei | CREATE TABLE `t_huawei` (

 `hwid` int(11) DEFAULT NULL,

 `content` varchar(4000) DEFAULT NULL,

 `TYPE` char(2) DEFAULT NULL,

  `creator_id`varchar(30) DEFAULT NULL,

 `create_time` datetime DEFAULT NULL

) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1 row in set (0.00 sec)

 

mysql>

 

1.3 开始录入数据

因为要录入千万级别数据，所以用后台mysql进程来录入，如下所示：

nohup /usr/local/mysql/bin/mysql -uroot--password="" -S /usr/local/mysql/mysql.sock -e "select now();truncate table hwdb.t_huawei; call hwdb.pro_insert_hw(10000000); select now();" > /data/call.log &

 

这里因为测试环境机器性能有限，在录入到7023850条的时候，我终止了进程，开始准备测试工作：

mysql> SELECT COUNT(1) FROM `t_huawei` ;

+----------+

| COUNT(1) |

+----------+

| 7023850 |

+----------+

1 row in set (5.52 sec)

 

mysql>

 

 

2，开始测试

 

 

2.1 普通的ORDER
BY RAND()测试很慢很慢，如下所示，需要块1分钟的时间

mysql> SELECT  SQL_NO_CACHE * FROM `t_huawei` AS t1  ORDER BY RAND() LIMIT 5;

+---------+-----------------------------------------------------------------------------------------------------+------+------------+---------------------+

| hwid   | content                                                                                            | TYPE |creator_id | create_time         |

+---------+-----------------------------------------------------------------------------------------------------+------+------------+---------------------+

| 3743323 | 3743323 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 20:43:01|

| 2418491 | 2418491 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 20:19:31|

| 1224667 | 1224667 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:58:35|

| 1639270 | 1639270 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 20:05:50|

| 2756470 | 2756470 - Players supposedlysaid they'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 20:25:28|

+---------+-----------------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (54.92 sec)

 

mysql>

 

 

原因是：MYSQL的官方手册，里面针对RAND()的提示大概意思就是，在ORDER BY从句里面不能使用RAND()函数，因为这样会导致数据列被多次扫描。但是在MYSQL 3.23版本中，仍然可以通过ORDER BY RAND()来实现随机。

 

 

2.2 在JOIN中使用RAND()*
(SELECTMAX(hwid) 的方式来实现

mysql> SELECT SQL_NO_CACHE t1.* FROM`t_huawei` AS t1 LEFT JOIN(SELECT ROUND(RAND()* (SELECT MAX(hwid) FROM  `t_huawei` )) AS id ) AS t2 ONt1.`hwid`>=t2.id ORDER BY t1.`hwid` ASC LIMIT 5;

+------+-----------------------------------------------------------------------------------------------+------+------------+---------------------+

| hwid | content                                                                                      | TYPE | creator_id | create_time        |

+------+-----------------------------------------------------------------------------------------------+------+------------+---------------------+

|   1 | 1 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the  | 0    | 10013     | 2015-06-08 19:37:02 |

|   2 | 2 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the  | 0    | 10013     | 2015-06-08 19:37:02 |

|   3 | 3 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the  | 0    | 10013     | 2015-06-08 19:37:02 |

|   4 | 4 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the  | 0    | 10013     | 2015-06-08 19:37:02 |

|   5 | 5 - Players supposedly said they'd avoid team facility this summerif Thibodeau remained the  | 0    | 10013     | 2015-06-08 19:37:02 |

+------+-----------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (50.04 sec)

 

mysql>

需要50面，比起直接的order by来说少了4秒，但是还是比较耗时的。

 

 

2.3 WHERE子句的方式来优化

mysql> SELECT SQL_NO_CACHE t1.* FROMhwdb.`t_huawei` t1 WHERE t1.`hwid` >=(SELECT FLOOR(RAND() * (SELECTMAX(t2.hwid) FROM hwdb.`t_huawei` t2 ) )) ORDER BY t1.hwid LIMIT 5;

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

| hwid | content                                                                                         | TYPE | creator_id | create_time        |

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

| 1624 | 1624 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:05|

| 6076 | 6076 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:11|

| 7988 | 7988 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:14|

| 8016 | 8016 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:14|

| 8106 | 8106 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:14|

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (7.25 sec)

 

mysql>

 

花了7.25秒，比上一种要快的多很多了。

 

2.4 MAX(hwid)- MIN(hwid)的方式来RAND来操作

mysql> SELECT SQL_NO_CACHE * FROMhwdb.`t_huawei` WHERE hwid >= (SELECT FLOOR(RAND() * ((SELECT MAX(hwid) FROMhwdb.`t_huawei`) - (SELECT MIN(hwid) FROM hwdb.`t_huawei`)) + (SELECT MIN(hwid)FROM hwdb.`t_huawei`))) ORDER BY hwid LIMIT 5;

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

| hwid | content                                                                                          |TYPE | creator_id | create_time         |

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

| 5438 | 5438 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:10|

| 5883 | 5883 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:10|

| 6782 | 6782 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:12|

| 7797 | 7797 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:13|

| 8625 | 8625 - Players supposedly saidthey'd avoid team facility this summer if Thibodeau remained the  | 0   | 10013      | 2015-06-08 19:37:15|

+------+--------------------------------------------------------------------------------------------------+------+------------+---------------------+

5 rows in set (13.50 sec)

 

mysql>

MAX(hwid)- MIN(hwid)的方式来RAND来操作需要13.5秒，比where子句花时间要多一些。

3，总结

WHERE子句的方式效率最高，样例sql：SELECTSQL_NO_CACHE t1.* FROM hwdb.`t_huawei` t1 WHERE t1.`hwid` >=(SELECTFLOOR(RAND() * (SELECT MAX(t2.hwid) FROM hwdb.`t_huawei` t2 ) )) ORDER BYt1.hwid LIMIT 5;