Redis Essentials 读书笔记 - 第一章： Getting Started (The Baby Steps)

Chapter 1. Getting Started (The Baby Steps)

Redis是在内存中运行的NoSQL key-value数据库。

Redis的优势除了内存的高性能外，还有其支持丰富的数据类型，如Strings, Hashes, Lists, Sets, Sorted Sets, Bitmaps, 和 HyperLogLogs。

Redis同时支持持久化，使用的技术为snapshotting 和 journaling。

Redis表示REmote DIctionary Server，此名字也说明其主要用途为look-up，以及其key-value的特性。

Redis最初是Salvatore Sanfilippo在2006年用C语言写的，是一位来自意大利西西里岛的帅哥。



目前客户端支持30多种编程语言。此开源项目可以在https://github.com/antirez/redis中找到，官方网站为http://redis.io。

Redis被很多大公司采用，如Twitter, GitHub, Tumblr, Pinterest, Instagram, Hulu, Flickr, 以及New York Times。

本章讲述如何安装Redis，CLI接口，以及Node.js客户端，然后介绍三种数据类型: Strings, Lists, Hashes.

Installation

Redis官方支持Linux, OS X, OpenBSD, NetBSD和FreeBSD平台。 主流还是Linux，不支持Windows.

Installing from source

源码下载：http://redis.io/download

最新稳定版下载：http://download.redis.io/releases/redis-3.0.7.tar.gz

安装就是下载源码，make编译

[root@tt12c ~]# useradd redis
[root@tt12c ~]# echo ~redis
/home/redis
[root@tt12c ~]# passwd redis

[root@tt12c ~]# su - redis

[redis@tt12c ~]$ cp /mnt/hgfs/TEMP/redis-3.0.7.tar.gz .
[redis@tt12c ~]$ ls -l
total 1344
-rwxrwxr-x 1 redis redis 1375200 Apr 27 19:40 redis-3.0.7.tar.gz

[redis@tt12c ~]$ tar -zxf redis-3.0.7.tar.gz
[redis@tt12c ~]$ ls
redis-3.0.7  redis-3.0.7.tar.gz
[redis@tt12c ~]$ cd redis-3.0.7
[redis@tt12c redis-3.0.7]$ ls
00-RELEASENOTES  COPYING  Makefile   redis.conf       runtest-sentinel  tests
BUGS             deps     MANIFESTO  runtest          sentinel.conf     utils
CONTRIBUTING     INSTALL  README     runtest-cluster  src
[redis@tt12c redis-3.0.7]$ make
cd src && make all
make[1]: Entering directory `/home/redis/redis-3.0.7/src'
rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-dump redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html
(cd ../deps && make distclean)
make[2]: Entering directory `/home/redis/redis-3.0.7/deps'
(cd hiredis && make clean) > /dev/null || true
......
......
Hint: It's a good idea to run 'make test' ;)

make[1]: Leaving directory `/home/redis/redis-3.0.7/src'
[redis@tt12c redis-3.0.7]$ echo $?
0

启动redis服务, 输出中有版本，PID，Port等信息

[redis@tt12c redis-3.0.7]$ src/redis-server
14892:C 27 Apr 20:01:50.255 # Warning: no config file specified, using the default config. In order to specify a config file use src/redis-server /path/to/redis.conf
_._
_.-``__ ''-._
_.-``    `.  `_.  ''-._           Redis 3.0.7 (00000000/0) 64 bit
.-`` .-```.  ```\/    _.,_ ''-._
(    '      ,       .-`  | `,    )     Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
|    `-._   `._    /     _.-'    |     PID: 14892
`-._    `-._  `-./  _.-'    _.-'
|`-._`-._    `-.__.-'    _.-'_.-'|
|    `-._`-._        _.-'_.-'    |           http://redis.io `-._    `-._`-.__.-'_.-'    _.-'
|`-._`-._    `-.__.-'    _.-'_.-'|
|    `-._`-._        _.-'_.-'    |
`-._    `-._`-.__.-'_.-'    _.-'
`-._    `-.__.-'    _.-'
`-._        _.-'
`-.__.-'

14892:M 27 Apr 20:01:50.260 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
14892:M 27 Apr 20:01:50.260 # Server started, Redis version 3.0.7
14892:M 27 Apr 20:01:50.261 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
14892:M 27 Apr 20:01:50.261 * The server is now ready to accept connections on port 6379

Redis的客户端程序为redis-cli

[redis@tt12c redis-3.0.7]$ src/redis-cli
127.0.0.1:6379>  set foo bar
OK
127.0.0.1:6379> get foo
"bar"
127.0.0.1:6379> exit
[redis@tt12c redis-3.0.7]$ du -sk .
46304   .
[redis@tt12c redis-3.0.7]$

make install是将执行程序拷贝到/usr/local/bin下

[redis@tt12c redis-3.0.7]$ sudo make install
[sudo] password for redis:
cd src && make install
make[1]: Entering directory `/home/redis/redis-3.0.7/src'

Hint: It's a good idea to run 'make test' ;)

INSTALL install
INSTALL install
INSTALL install
INSTALL install
INSTALL install
make[1]: Leaving directory `/home/redis/redis-3.0.7/src'
[redis@tt12c redis-3.0.7]$ echo $?
0
[redis@tt12c redis-3.0.7]$ redis-client
-bash: redis-client: command not found
[redis@tt12c redis-3.0.7]$ which redis-cli
/usr/local/bin/redis-cli
[redis@tt12c redis-3.0.7]$ ls -l /usr/local/bin/redis-*
-rwxr-xr-x 1 root root 4587363 Apr 27 20:18 /usr/local/bin/redis-benchmark
-rwxr-xr-x 1 root root   22225 Apr 27 20:18 /usr/local/bin/redis-check-aof
-rwxr-xr-x 1 root root   45443 Apr 27 20:18 /usr/local/bin/redis-check-dump
-rwxr-xr-x 1 root root 4696578 Apr 27 20:18 /usr/local/bin/redis-cli
lrwxrwxrwx 1 root root      12 Apr 27 20:18 /usr/local/bin/redis-sentinel -> redis-server
-rwxr-xr-x 1 root root 6469327 Apr 27 20:18 /usr/local/bin/redis-server

Hello Redis (command-line interface examples)

Redis最重要的可执行程序为redis-server 和 redis-cli。

redis-server就是Redis数据库，可以运行在standalone或cluster模式下。

redis-cli就是Redis 客户端，负责与redis-server交互。

Redis缺省绑定在6379端口, 运行在standalone模式下, 启动命令如下:

$ redis-server

redis客户端运行示例如下：

[redis@tt12c redis-3.0.7]$ redis-cli
127.0.0.1:6379> set city beijing
OK
127.0.0.1:6379> get city
"beijing"
127.0.0.1:6379> help set

SET key value [EX seconds] [PX milliseconds] [NX|XX]
summary: Set the string value of a key
since: 1.0.0
group: string

127.0.0.1:6379> help get

GET key
summary: Get the value of a key
since: 1.0.0
group: string

127.0.0.1:6379> keys c*  <- 支持 glob pattern
1) "city"
127.0.0.1:6379>

redis-cli是不错的用于调试和测试的工具。但本书大部分的例子还是使用了JavaScript和Node.js，一是由于其普遍性，二是为了更好的演示和理解。

Installing Node.js

Node.js下载地址: https://nodejs.org/dist/v4.4.3/node-v4.4.3-linux-x64.tar.xz

mkdir  redis-essentials
cp node-v4.4.3-linux-x64.tar.xz redis-essentials
mkdir redis-essentials
cp node-v4.4.3-linux-x64.tar.xz redis-essentials
cd redis-essentials

unxz node-v4.4.3-linux-x64.tar.xz
tar xvf node-v4.4.3-linux-x64.tar

本书中所有的Node.js例子都需要redis库, 我们可以通过NPM(node.js自带的应用)安装:

[redis@tt12c bin]$ ./npm install redis
redis@2.6.0-1 node_modules/redis
├── double-ended-queue@2.1.0-0
├── redis-commands@1.2.0
└── redis-parser@1.3.0

NPM会创建一个目录node_modules. 客户端运行时通过NODE_PATH寻找这些module，例如：

NODE_PATH=/home/redis/node-v4.4.3-linux-x64/node_modules
export NODE_PATH

JavaScript syntax quick reference guide

JavaScript语法快速参考，不一一解释了。

Use the keyword var to define a variable:

var myAge = 31;

Use // for inline comments and /* */ for multiline comments:

// this is an inline comment
/* this
is a
multi-line
comment
*/

Conditional statements:

if (myAge > 29) {
console.log("I am not in my twenties anymore!");
} else {
console.log("I am still in my twenties!");
}

Defining a function:

function nameOfMyFunction(argument1, argument2) {
console.log(argument1, argument2);
}

Executing a function:

nameOfMyFunction("First Value", "Second Value");

A function can also behave as a class and have methods, properties, and instances. Properties are accessed through the keyword this:

function Car(maxSpeed) {
this.maxSpeed = maxSpeed;
this.currentSpeed = 0;
}

The standard way to create a prototyped method for a function in JavaScript is by using the property prototype:

Car.prototype.brake = function() {
if (this.currentSpeed > 0) {
this.currentSpeed -= 5;
}
};

Car.prototype.accelerate = function() {
if (this.currentSpeed < this.maxSpeed) {
this.currentSpeed += 5;
}
};

To create an instance of a class in JavaScript, use the keyword new:

var car = new Car(100);
car.accelerate();
car.accelerate();
car.brake();

Arrays and objects:

var myArray = [];
var myObject = {};

Callbacks in JavaScript:

var friends = ["Karalyn", "Patrik", "Bernardo"];
friends.forEach(function (name, index) {
console.log(index + 1, name); // 1 Karalyn, 2 Patrik, 3 Bernardo
});

更多JavaScript信息请参考https://developer.mozilla.org/en-US/docs/Web/JavaScript.

Hello World with Node.js and Redis

本书所有的示例程序可以从https://github.com/redis-essentials/book下载，打包下载地址为：https://github.com/redis-essentials/book/archive/master.zip

[redis@tt12c examples]$ cd chapter\ 1
[redis@tt12c chapter 1]$ ls
articles-popularity.js  hash-voting-system.js  producer-worker.js
consumer-worker.js      hello.js               queue.js

[redis@tt12c chapter 1]$ cat hello.js
var redis = require("redis"); // 1
var client = redis.createClient(); // 2
client.set("my_key", "Hello World using Node.js and Redis"); // 3
client.get("my_key", redis.print); // 4
client.quit(); // 5

[redis@tt12c chapter 1]$ node hello.js
Reply: Hello World using Node.js and Redis

Redis data types

只有理解了Redis的数据类型，选择正确的数据类型，才能设计出更好的应用。不同的问题需要不同的数据类型。

Strings

Strings 是最通用的数据类型，根据String中存放的值，String 可用来表示 integer, float, text string, or bitmap。它可用来存放文本 (XML, JSON, HTML, or raw text), integers, floats, or 二进制数据 (视频, 音频, 图像). String存放的值不能超过512 MB。

Strings可使用的场景包括：

* 缓存: 通过SET, GET, MSET, and MGET设置key-value，value可以是文本或二进制数据。

* 带期限的缓存，通过SETEX, EXPIRE, 和 EXPIREAT设置expiration

* 计数，通过INCR, INCRBY，或DECR, DECRBY, INCRFLOATBY

String examples with redis-cli

MSET和MGET读写多个key和value

[redis@tt12c ~]$ redis-cli
127.0.0.1:6379> MSET key1 "value1" key2 "value2"
OK
127.0.0.1:6379> MGET key1 key2
1) "value1"
2) "value2"

EXPIRE为key设置过期时间，单位为秒，到期后，key被删除。TTL显示到期信息，返回值为。

正数: This is the amount of seconds a given key has left to live

-2: If the key is expired or does not exist

-1: If the key exists but has no expiration time set

127.0.0.1:6379> SET key1 "value1"
OK
127.0.0.1:6379> EXPIRE key1 10
(integer) 1
127.0.0.1:6379> GET key1
"value1"
127.0.0.1:6379> TTL key1
(integer) 5
127.0.0.1:6379> TTL key1
(integer) 1
127.0.0.1:6379> TTL key1
(integer) -2
127.0.0.1:6379> GET key1
(nil)

计数的演示

$ redis-cli
127.0.0.1:6379> SET counter 100
OK
127.0.0.1:6379> INCR counter
(integer) 101
127.0.0.1:6379> INCRBY counter 5
(integer) 106
127.0.0.1:6379> DECR counter
(integer) 105
127.0.0.1:6379> DECRBY counter 100
(integer) 5
127.0.0.1:6379> GET counter
"5"
127.0.0.1:6379> INCRBYFLOAT counter 2.4
"7.4"

Redis是单线程的，这点非常重要，在任何时候Redis只会执行一个命令。因此以上的INCR, DECR命令都是原子操作，不会出现冲突和数据一致性问题。

Building a voting system with Strings using Node.js

本例为一个投票计数系统。key是文章名，value是投票数

在运行本例前，需要在redis-cli中设置文章的标题，即后续headlineKey对应的value

$ redis-cli
127.0.0.1:6379> SET article:12345:headline "Google Wants to Turn Your Clothes Into a Computer"
OK
127.0.0.1:6379> SET article:10001:headline "For Millennials, the End of the TV Viewing Party"
OK
127.0.0.1:6379> SET article:60056:headline "Alicia Vikander, Who Portrayed Denmark's Queen, Is Screen Royalty"
OK

[redis@tt12c chapter 1]$ node articles-popularity.js
The article "Google Wants to Turn Your Clothes Into a Computer" has 3 votes
The article "For Millennials, the End of the TV Viewing Party" has 1 votes
The article "Alicia Vikander, Who Portrayed Denmark's Queen, Is Screen Royalty" has 1 votes

[redis@tt12c chapter 1]$ cat articles-popularity.js
var redis = require("redis"); // 1
var client = redis.createClient(); // 2

function upVote(id) { // 3
var key = "article:" + id + ":votes"; // 4
client.incr(key); // 5
}

function downVote(id) { // 1
var key = "article:" + id + ":votes"; // 2
client.decr(key); // 3
}

function showResults(id) {
var headlineKey = "article:" + id + ":headline";
var voteKey = "article:" + id + ":votes";
client.mget([headlineKey, voteKey], function(err, replies) { // 1
console.log('The article "' + replies[0] + '" has', replies[1], 'votes'); // 2
});
}

upVote(12345); // article:12345 has 1 vote
upVote(12345); // article:12345 has 2 votes
upVote(12345); // article:12345 has 3 votes
upVote(10001); // article:10001 has 1 vote
upVote(10001); // article:10001 has 2 votes
downVote(10001); // article:10001 has 1 vote
upVote(60056); // article:60056 has 1 vote
showResults(12345);
showResults(10001);
showResults(60056);
client.quit();

Node.js client是异步的. 所有的Redis命令都可设置callback函数来处理Redis server返回的结果和错误。

在上面的MGET例子, client.mget()中调用callback是唯一可行的方法。

这里还有一个技巧，在以后的例子中会频繁用到，即通过拼接字符串来构建key，例如”article:” + id + “:votes”。因为Redis只支持单key，没有primary key, secondary key的概念，因此可以用：拼接的方法模拟多层key，保证key的唯一性。

Lists

列表是非常灵活的数据类型，可用于表示队列，堆栈和集合，许多事件系统使用Lists来作为队列，因为List的操作时原子操作，并行访问不会有冲突，本质上将并行系统串行化。List有阻塞(block)命令，可以等待空队列插入元素. Redis的List为linked list, 因此在表头和尾的操作时间为固定时间O(1)。

访问列表中元素的时间是线性的，为O(N)。

如果列表中的元素数量小于list-max-ziplist-entries并且元素的大小小于list-max-ziplist-value字节，Redis可以使用内存优化的方式存储。

List的实际使用场景为:

* 时间队列: 许多工具都使用, 包括 Resque, Celery, Logstash

* 存放用户最近的发布的信息，例如Twitter

List examples with redis-cli

List数据结构为链接列表，LPUSH和RPUSH分别在表头和表尾插入数据。LPOP和RPOP删除数据。

LLEN返回列表长度，LINDEX返回第N个元素，从左到右，0表示第一个元素。-1是最后一个元素。

LRANGE返回某一区间的元素。

[redis@tt12c ~]$ redis-cli
127.0.0.1:6379> LPUSH employees susan
(integer) 1
127.0.0.1:6379> LPUSH employees john
(integer) 2
127.0.0.1:6379> RPUSH employees danny
(integer) 3
127.0.0.1:6379> LLEN employees
(integer) 3
127.0.0.1:6379> LINDEX employees 0
"john"
127.0.0.1:6379> LRANGE employees 0 -1
1) "john"
2) "susan"
3) "danny"
127.0.0.1:6379> LPOP employees
"john"
127.0.0.1:6379> RPOP employees
"danny"
127.0.0.1:6379> LRANGE employees 0 -1
1) "susan"

Implementing a generic Queue System

演示如下：

[redis@tt12c chapter 1]$ ls
producer-worker.js consumer-worker.js      queue.js
[redis@tt12c chapter 1]$ node producer-worker.js
Created 5 logs
[redis@tt12c chapter 1]$ node consumer-worker.js
[consumer] Got log: Hello world #0
4 logs left
[consumer] Got log: Hello world #1
3 logs left
[consumer] Got log: Hello world #2
2 logs left
[consumer] Got log: Hello world #3
1 logs left
[consumer] Got log: Hello world #4
0 logs left

producer产生日志（LPUSH），存入FIFO的队列，consumer从列表中取日志（BRPOP）。

BRPOP是block POP，如果队列为空会等待。

从本例我们也可以知道，两个客户端并非直接交互，而是通过redis-server来交互的，redis-server是一个数据集成的点。

Hashes

Hash类型的key对应于几组field和value，非常类似于C语言中的Struct。field和value类型皆为String。

Hash的内部实现可以是ziplist或hash table. 区别如下：

A ziplist is a dually linked list designed to be memory efficient. In a ziplist, integers are stored as real integers rather than a sequence of characters. Although a ziplist has memory optimizations, lookups are not performed in constant time. On the other hand, a hash table has constant-time lookup but is not memory-optimized.

扩展阅读：

Instagram的这个例子说明了Hash比String具有更好的存储效率，原因应该是使用ziplist可以压缩。在InstaGram的例子中, key是bucket，field是media ID，value是user ID。之所以需要尽量减少内存占用，是因为Instagram想把一个shard的数据能被一个EC2的机器（17G）容纳，所以从String的21G减少到5G就有意义了。

关键原理如下:

To take advantage of the hash type, we bucket all our Media IDs into buckets of 1000 (we just take the ID, divide by 1000 and discard the remainder). That determines which key we fall into; next, within the hash that lives at that key, the Media ID is the lookup key within the hash, and the user ID is the value. An example, given a Media ID of 1155315, which means it falls into bucket 1155 (1155315 / 1000 = 1155):

HSET "mediabucket:1155" "1155315" "939"
HGET "mediabucket:1155" "1155315"
"939"

本文也介绍了Instagram Shard的实现，可以借鉴。

Instagram采用 PostgreSQL 作为一个Shard，关键的问题就是如何产生唯一的ID。最终产生的ID为64 bit，组成如下：

41 bits for time in milliseconds (gives us 41 years of IDs with a custom epoch) - timestamp

13 bits that represent the logical shard ID - user ID % 2000(shards)

10 bits that represent an auto-incrementing sequence(数据库产生的sequence), modulus 1024. This means we can generate 1024 IDs, per shard, per millisecond

这个Instagram Engineering Blog推荐后续慢慢的看！

Using Hashes with redis-cli

Hash的命令和String类似，多了个H前缀，如HSET, HMSET, HINCRBY

127.0.0.1:6379> HSET employee age 43
(integer) 1
127.0.0.1:6379> HSET employee sex male
(integer) 1
127.0.0.1:6379> HGET employee
(error) ERR wrong number of arguments for 'hget' command
127.0.0.1:6379> HGET employee age
"43"
127.0.0.1:6379> HINCRBY employee age
(error) ERR wrong number of arguments for 'hincrby' command
127.0.0.1:6379> HINCRBY employee age 1
(integer) 44
127.0.0.1:6379> HMGET employee sex name
1) "male"
2) "john"
127.0.0.1:6379> HDEL employee sex
(integer) 1
127.0.0.1:6379> HGETALL employee
1) "name"
2) "john"
3) "age"
4) "44"
127.0.0.1:6379> HKEYS employee
1) "name"
2) "age"
127.0.0.1:6379> HVALS employee
1) "john"
2) "44"

127.0.0.1:6379> HSCAN employee 0 <- 若field很多，为性能计，建议用HSCAN替代HGETALL
1) "0"
2) 1) "name"
2) "john"
3) "age"
4) "44"

A voting system with Hashes and Node.js

类似于前面String的例子，本例用Hash实现了投票系统。这是一个http://www.reddit.com投票系统的简版实现。

我们注意到，在String的例子中，设置了两组key，headlineKey用于标题，voteKey用于计数，两者之间完全依赖于key的设计关联，即”article:” + id。

var headlineKey = “article:” + id + “:headline”;

var voteKey = “article:” + id + “:votes”;

在Hash的例子中，key为”link:” + id, ，field为”author”，”title”, “link”, “score”。

Hash的实现更简洁，程序更易维护。

Summary

本章讲述了Redis的历史和设计理念，如何安装，以及redis-cli客户端。

本书的例子基于Node.js，因此介绍了JavaScript的简单语法以及如何安装Node.js。

最重要的还是数据类型，本章介绍了String（key对应一个value）, Lists(key对应一组用单向链表连接的value), Hash(key对应一组field:value)。

参考：

* https://scaleyourcode.com/blog/article/14

* https://redislabs.com/blog/redis-keys-in-ram#.Vyk3L-TAM0p