苦逼运维的elk之路(2) ----- 组件基础配置

上文 苦逼运维的elk之路(1) ----- 组件安装篇 说明了各个业务组件的安装
那么,安装完了就可以用了吗? 你不是说开箱即用吗?
我理解的所谓的"开箱即用" 是这样的

1	下载-->解压-->简单配置--->运行

那么,我们至少要配置一下.a
好,先从redis开始, 我喜欢把它注册成服务,这样我就不用纠结它们之间的种种了,省心省力

1. Redis配置
先把它注册成服务,然后再运行配置

工作目录从我们的用户目录开始
/home/user
同样su 进root权限来配置

123456789101112131415161718192021222324

# cp tmp/redis-3.0.6/utils/redis_init_script /etc/rc.d/init.d/redis# vi /etc/rc.d/init.d/redis //然后在第二行插入chkconfig配置,然后修改EXEC和CLI,我的这个文件前几行是这样的 #!/bin/sh# chkconfig: 2345 90 10# Simple Redis init.d script conceived to work on Linux systems# as it does use of the /proc filesystem. REDISPORT=6379EXEC=/usr/local/redis/bin/redis-serverCLIEXEC=/usr/local/redis/bin/redis-cli PIDFILE=/var/run/redis_${REDISPORT}.pidCONF="/etc/redis/${REDISPORT}.conf" //按照上面的修改好了之后保存退出 # mkdir /etc/redis///这个目录用于放我们的配置文件# mkdir /var/rdb///这个目录存放redis的数据库文件

redis源码包中自带redis.conf,但这个只是模版,具体使用还需要自行斟酌具体配置,这里我发出来一个我的配置

1	# vi /etc/redis/6379.conf

然后把下面的配置文件贴进去,保存退出即可

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949

# Redis configuration file example # Note on units: when memory size is needed, it is possible to specify# it in the usual form of 1k 5GB 4M and so forth:## 1k => 1000 bytes# 1kb => 1024 bytes# 1m => 1000000 bytes# 1mb => 1024*1024 bytes# 1g => 1000000000 bytes# 1gb => 1024*1024*1024 bytes## units are case insensitive so 1GB 1Gb 1gB are all the same. ################################## INCLUDES ################################### # Include one or more other config files here. This is useful if you# have a standard template that goes to all Redis servers but also need# to customize a few per-server settings. Include files can include# other files, so use this wisely.## Notice option "include" won't be rewritten by command "CONFIG REWRITE"# from admin or Redis Sentinel. Since Redis always uses the last processed# line as value of a configuration directive, you'd better put includes# at the beginning of this file to avoid overwriting config change at runtime.## If instead you are interested in using includes to override configuration# options, it is better to use include as the last line.## include /path/to/local.conf# include /path/to/other.conf ################################ GENERAL ##################################### # By default Redis does not run as a daemon. Use 'yes' if you need it.# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.daemonize yes # When running daemonized, Redis writes a pid file in /var/run/redis.pid by# default. You can specify a custom pid file location here.# pidfile /var/run/redis.pid # Accept connections on the specified port, default is 6379.# If port 0 is specified Redis will not listen on a TCP socket.port 6379 # TCP listen() backlog.## In high requests-per-second environments you need an high backlog in order# to avoid slow clients connections issues. Note that the Linux kernel# will silently truncate it to the value of /proc/sys/net/core/somaxconn so# make sure to raise both the value of somaxconn and tcp_max_syn_backlog# in order to get the desired effect.tcp-backlog 511 # By default Redis listens for connections from all the network interfaces# available on the server. It is possible to listen to just one or multiple# interfaces using the "bind" configuration directive, followed by one or# more IP addresses.## Examples:## bind 192.168.1.100 10.0.0.1bind 0.0.0.0 # Specify the path for the Unix socket that will be used to listen for# incoming connections. There is no default, so Redis will not listen# on a unix socket when not specified.## unixsocket /tmp/redis.sock# unixsocketperm 700 # Close the connection after a client is idle for N seconds (0 to disable)timeout 0 # TCP keepalive.## If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence# of communication. This is useful for two reasons:## 1) Detect dead peers.# 2) Take the connection alive from the point of view of network# equipment in the middle.## On Linux, the specified value (in seconds) is the period used to send ACKs.# Note that to close the connection the double of the time is needed.# On other kernels the period depends on the kernel configuration.## A reasonable value for this option is 60 seconds.tcp-keepalive 0 # Specify the server verbosity level.# This can be one of:# debug (a lot of information, useful for development/testing)# verbose (many rarely useful info, but not a mess like the debug level)# notice (moderately verbose, what you want in production probably)# warning (only very important / critical messages are logged)loglevel verbose # Specify the log file name. Also the empty string can be used to force# Redis to log on the standard output. Note that if you use standard# output for logging but daemonize, logs will be sent to /dev/nulllogfile "/var/log/redis.log" # To enable logging to the system logger, just set 'syslog-enabled' to yes,# and optionally update the other syslog parameters to suit your needs.# syslog-enabled no # Specify the syslog identity.# syslog-ident redis # Specify the syslog facility. Must be USER or between LOCAL0-LOCAL7.# syslog-facility local0 # Set the number of databases. The default database is DB 0, you can select# a different one on a per-connection basis using SELECT <dbid> where# dbid is a number between 0 and 'databases'-1databases 16 ################################ SNAPSHOTTING ################################## Save the DB on disk:## save <seconds> <changes>## Will save the DB if both the given number of seconds and the given# number of write operations against the DB occurred.## In the example below the behaviour will be to save:# after 900 sec (15 min) if at least 1 key changed# after 300 sec (5 min) if at least 10 keys changed# after 60 sec if at least 10000 keys changed## Note: you can disable saving completely by commenting out all "save" lines.## It is also possible to remove all the previously configured save# points by adding a save directive with a single empty string argument# like in the following example:## save "" save 900 1save 300 10save 60 10000 # By default Redis will stop accepting writes if RDB snapshots are enabled# (at least one save point) and the latest background save failed.# This will make the user aware (in a hard way) that data is not persisting# on disk properly, otherwise chances are that no one will notice and some# disaster will happen.## If the background saving process will start working again Redis will# automatically allow writes again.## However if you have setup your proper monitoring of the Redis server# and persistence, you may want to disable this feature so that Redis will# continue to work as usual even if there are problems with disk,# permissions, and so forth.stop-writes-on-bgsave-error yes # Compress string objects using LZF when dump .rdb databases?# For default that's set to 'yes' as it's almost always a win.# If you want to save some CPU in the saving child set it to 'no' but# the dataset will likely be bigger if you have compressible values or keys.rdbcompression yes # Since version 5 of RDB a CRC64 checksum is placed at the end of the file.# This makes the format more resistant to corruption but there is a performance# hit to pay (around 10%) when saving and loading RDB files, so you can disable it# for maximum performances.## RDB files created with checksum disabled have a checksum of zero that will# tell the loading code to skip the check.rdbchecksum yes # The filename where to dump the DBdbfilename redis_6379.rdb # The working directory.## The DB will be written inside this directory, with the filename specified# above using the 'dbfilename' configuration directive.## The Append Only File will also be created inside this directory.## Note that you must specify a directory here, not a file name.dir /var/rdb/ ################################# REPLICATION ################################# # Master-Slave replication. Use slaveof to make a Redis instance a copy of# another Redis server. A few things to understand ASAP about Redis replication.## 1) Redis replication is asynchronous, but you can configure a master to# stop accepting writes if it appears to be not connected with at least# a given number of slaves.# 2) Redis slaves are able to perform a partial resynchronization with the# master if the replication link is lost for a relatively small amount of# time. You may want to configure the replication backlog size (see the next# sections of this file) with a sensible value depending on your needs.# 3) Replication is automatic and does not need user intervention. After a# network partition slaves automatically try to reconnect to masters# and resynchronize with them.## slaveof <masterip> <masterport> # If the master is password protected (using the "requirepass" configuration# directive below) it is possible to tell the slave to authenticate before# starting the replication synchronization process, otherwise the master will# refuse the slave request.## masterauth <master-password> # When a slave loses its connection with the master, or when the replication# is still in progress, the slave can act in two different ways:## 1) if slave-serve-stale-data is set to 'yes' (the default) the slave will# still reply to client requests, possibly with out of date data, or the# data set may just be empty if this is the first synchronization.## 2) if slave-serve-stale-data is set to 'no' the slave will reply with# an error "SYNC with master in progress" to all the kind of commands# but to INFO and SLAVEOF.#slave-serve-stale-data yes # You can configure a slave instance to accept writes or not. Writing against# a slave instance may be useful to store some ephemeral data (because data# written on a slave will be easily deleted after resync with the master) but# may also cause problems if clients are writing to it because of a# misconfiguration.## Since Redis 2.6 by default slaves are read-only.## Note: read only slaves are not designed to be exposed to untrusted clients# on the internet. It's just a protection layer against misuse of the instance.# Still a read only slave exports by default all the administrative commands# such as CONFIG, DEBUG, and so forth. To a limited extent you can improve# security of read only slaves using 'rename-command' to shadow all the# administrative / dangerous commands.slave-read-only yes # Replication SYNC strategy: disk or socket.## -------------------------------------------------------# WARNING: DISKLESS REPLICATION IS EXPERIMENTAL CURRENTLY# -------------------------------------------------------## New slaves and reconnecting slaves that are not able to continue the replication# process just receiving differences, need to do what is called a "full# synchronization". An RDB file is transmitted from the master to the slaves.# The transmission can happen in two different ways:## 1) Disk-backed: The Redis master creates a new process that writes the RDB# file on disk. Later the file is transferred by the parent# process to the slaves incrementally.# 2) Diskless: The Redis master creates a new process that directly writes the# RDB file to slave sockets, without touching the disk at all.## With disk-backed replication, while the RDB file is generated, more slaves# can be queued and served with the RDB file as soon as the current child producing# the RDB file finishes its work. With diskless replication instead once# the transfer starts, new slaves arriving will be queued and a new transfer# will start when the current one terminates.## When diskless replication is used, the master waits a configurable amount of# time (in seconds) before starting the transfer in the hope that multiple slaves# will arrive and the transfer can be parallelized.## With slow disks and fast (large bandwidth) networks, diskless replication# works better.repl-diskless-sync no # When diskless replication is enabled, it is possible to configure the delay# the server waits in order to spawn the child that transfers the RDB via socket# to the slaves.## This is important since once the transfer starts, it is not possible to serve# new slaves arriving, that will be queued for the next RDB transfer, so the server# waits a delay in order to let more slaves arrive.## The delay is specified in seconds, and by default is 5 seconds. To disable# it entirely just set it to 0 seconds and the transfer will start ASAP.repl-diskless-sync-delay 5 # Slaves send PINGs to server in a predefined interval. It's possible to change# this interval with the repl_ping_slave_period option. The default value is 10# seconds.## repl-ping-slave-period 10 # The following option sets the replication timeout for:## 1) Bulk transfer I/O during SYNC, from the point of view of slave.# 2) Master timeout from the point of view of slaves (data, pings).# 3) Slave timeout from the point of view of masters (REPLCONF ACK pings).## It is important to make sure that this value is greater than the value# specified for repl-ping-slave-period otherwise a timeout will be detected# every time there is low traffic between the master and the slave.## repl-timeout 60 # Disable TCP_NODELAY on the slave socket after SYNC?## If you select "yes" Redis will use a smaller number of TCP packets and# less bandwidth to send data to slaves. But this can add a delay for# the data to appear on the slave side, up to 40 milliseconds with# Linux kernels using a default configuration.## If you select "no" the delay for data to appear on the slave side will# be reduced but more bandwidth will be used for replication.## By default we optimize for low latency, but in very high traffic conditions# or when the master and slaves are many hops away, turning this to "yes" may# be a good idea.repl-disable-tcp-nodelay no # Set the replication backlog size. The backlog is a buffer that accumulates# slave data when slaves are disconnected for some time, so that when a slave# wants to reconnect again, often a full resync is not needed, but a partial# resync is enough, just passing the portion of data the slave missed while# disconnected.## The bigger the replication backlog, the longer the time the slave can be# disconnected and later be able to perform a partial resynchronization.## The backlog is only allocated once there is at least a slave connected.## repl-backlog-size 1mb # After a master has no longer connected slaves for some time, the backlog# will be freed. The following option configures the amount of seconds that# need to elapse, starting from the time the last slave disconnected, for# the backlog buffer to be freed.## A value of 0 means to never release the backlog.## repl-backlog-ttl 3600 # The slave priority is an integer number published by Redis in the INFO output.# It is used by Redis Sentinel in order to select a slave to promote into a# master if the master is no longer working correctly.## A slave with a low priority number is considered better for promotion, so# for instance if there are three slaves with priority 10, 100, 25 Sentinel will# pick the one with priority 10, that is the lowest.## However a special priority of 0 marks the slave as not able to perform the# role of master, so a slave with priority of 0 will never be selected by# Redis Sentinel for promotion.## By default the priority is 100.slave-priority 100 # It is possible for a master to stop accepting writes if there are less than# N slaves connected, having a lag less or equal than M seconds.## The N slaves need to be in "online" state.## The lag in seconds, that must be <= the specified value, is calculated from# the last ping received from the slave, that is usually sent every second.## This option does not GUARANTEE that N replicas will accept the write, but# will limit the window of exposure for lost writes in case not enough slaves# are available, to the specified number of seconds.## For example to require at least 3 slaves with a lag <= 10 seconds use:## min-slaves-to-write 3# min-slaves-max-lag 10## Setting one or the other to 0 disables the feature.## By default min-slaves-to-write is set to 0 (feature disabled) and# min-slaves-max-lag is set to 10. ################################## SECURITY ################################### # Require clients to issue AUTH <PASSWORD> before processing any other# commands. This might be useful in environments in which you do not trust# others with access to the host running redis-server.## This should stay commented out for backward compatibility and because most# people do not need auth (e.g. they run their own servers).## Warning: since Redis is pretty fast an outside user can try up to# 150k passwords per second against a good box. This means that you should# use a very strong password otherwise it will be very easy to break.# # Command renaming.## It is possible to change the name of dangerous commands in a shared# environment. For instance the CONFIG command may be renamed into something# hard to guess so that it will still be available for internal-use tools# but not available for general clients.## Example:## rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52## It is also possible to completely kill a command by renaming it into# an empty string:## rename-command CONFIG ""## Please note that changing the name of commands that are logged into the# AOF file or transmitted to slaves may cause problems. ################################### LIMITS #################################### # Set the max number of connected clients at the same time. By default# this limit is set to 10000 clients, however if the Redis server is not# able to configure the process file limit to allow for the specified limit# the max number of allowed clients is set to the current file limit# minus 32 (as Redis reserves a few file descriptors for internal uses).## Once the limit is reached Redis will close all the new connections sending# an error 'max number of clients reached'.## maxclients 10000 # Don't use more memory than the specified amount of bytes.# When the memory limit is reached Redis will try to remove keys# according to the eviction policy selected (see maxmemory-policy).## If Redis can't remove keys according to the policy, or if the policy is# set to 'noeviction', Redis will start to reply with errors to commands# that would use more memory, like SET, LPUSH, and so on, and will continue# to reply to read-only commands like GET.## This option is usually useful when using Redis as an LRU cache, or to set# a hard memory limit for an instance (using the 'noeviction' policy).## WARNING: If you have slaves attached to an instance with maxmemory on,# the size of the output buffers needed to feed the slaves are subtracted# from the used memory count, so that network problems / resyncs will# not trigger a loop where keys are evicted, and in turn the output# buffer of slaves is full with DELs of keys evicted triggering the deletion# of more keys, and so forth until the database is completely emptied.## In short... if you have slaves attached it is suggested that you set a lower# limit for maxmemory so that there is some free RAM on the system for slave# output buffers (but this is not needed if the policy is 'noeviction').## maxmemory <bytes> # MAXMEMORY POLICY: how Redis will select what to remove when maxmemory# is reached. You can select among five behaviors:## volatile-lru -> remove the key with an expire set using an LRU algorithm# allkeys-lru -> remove any key according to the LRU algorithm# volatile-random -> remove a random key with an expire set# allkeys-random -> remove a random key, any key# volatile-ttl -> remove the key with the nearest expire time (minor TTL)# noeviction -> don't expire at all, just return an error on write operations## Note: with any of the above policies, Redis will return an error on write# operations, when there are no suitable keys for eviction.## At the date of writing these commands are: set setnx setex append# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby# getset mset msetnx exec sort## The default is:## maxmemory-policy noeviction # LRU and minimal TTL algorithms are not precise algorithms but approximated# algorithms (in order to save memory), so you can tune it for speed or# accuracy. For default Redis will check five keys and pick the one that was# used less recently, you can change the sample size using the following# configuration directive.## The default of 5 produces good enough results. 10 Approximates very closely# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.## maxmemory-samples 5 ############################## APPEND ONLY MODE ############################### # By default Redis asynchronously dumps the dataset on disk. This mode is# good enough in many applications, but an issue with the Redis process or# a power outage may result into a few minutes of writes lost (depending on# the configured save points).## The Append Only File is an alternative persistence mode that provides# much better durability. For instance using the default data fsync policy# (see later in the config file) Redis can lose just one second of writes in a# dramatic event like a server power outage, or a single write if something# wrong with the Redis process itself happens, but the operating system is# still running correctly.## AOF and RDB persistence can be enabled at the same time without problems.# If the AOF is enabled on startup Redis will load the AOF, that is the file# with the better durability guarantees.## Please check http://redis.io/topics/persistence for more information. appendonly no # The name of the append only file (default: "appendonly.aof") appendfilename "appendonly.aof" # The fsync() call tells the Operating System to actually write data on disk# instead of waiting for more data in the output buffer. Some OS will really flush# data on disk, some other OS will just try to do it ASAP.## Redis supports three different modes:## no: don't fsync, just let the OS flush the data when it wants. Faster.# always: fsync after every write to the append only log. Slow, Safest.# everysec: fsync only one time every second. Compromise.## The default is "everysec", as that's usually the right compromise between# speed and data safety. It's up to you to understand if you can relax this to# "no" that will let the operating system flush the output buffer when# it wants, for better performances (but if you can live with the idea of# some data loss consider the default persistence mode that's snapshotting),# or on the contrary, use "always" that's very slow but a bit safer than# everysec.## More details please check the following article:# http://antirez.com/post/redis-persistence-demystified.html## If unsure, use "everysec". # appendfsync alwaysappendfsync everysec# appendfsync no # When the AOF fsync policy is set to always or everysec, and a background# saving process (a background save or AOF log background rewriting) is# performing a lot of I/O against the disk, in some Linux configurations# Redis may block too long on the fsync() call. Note that there is no fix for# this currently, as even performing fsync in a different thread will block# our synchronous write(2) call.## In order to mitigate this problem it's possible to use the following option# that will prevent fsync() from being called in the main process while a# BGSAVE or BGREWRITEAOF is in progress.## This means that while another child is saving, the durability of Redis is# the same as "appendfsync none". In practical terms, this means that it is# possible to lose up to 30 seconds of log in the worst scenario (with the# default Linux settings).## If you have latency problems turn this to "yes". Otherwise leave it as# "no" that is the safest pick from the point of view of durability. no-appendfsync-on-rewrite no # Automatic rewrite of the append only file.# Redis is able to automatically rewrite the log file implicitly calling# BGREWRITEAOF when the AOF log size grows by the specified percentage.## This is how it works: Redis remembers the size of the AOF file after the# latest rewrite (if no rewrite has happened since the restart, the size of# the AOF at startup is used).## This base size is compared to the current size. If the current size is# bigger than the specified percentage, the rewrite is triggered. Also# you need to specify a minimal size for the AOF file to be rewritten, this# is useful to avoid rewriting the AOF file even if the percentage increase# is reached but it is still pretty small.## Specify a percentage of zero in order to disable the automatic AOF# rewrite feature. auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb # An AOF file may be found to be truncated at the end during the Redis# startup process, when the AOF data gets loaded back into memory.# This may happen when the system where Redis is running# crashes, especially when an ext4 filesystem is mounted without the# data=ordered option (however this can't happen when Redis itself# crashes or aborts but the operating system still works correctly).## Redis can either exit with an error when this happens, or load as much# data as possible (the default now) and start if the AOF file is found# to be truncated at the end. The following option controls this behavior.## If aof-load-truncated is set to yes, a truncated AOF file is loaded and# the Redis server starts emitting a log to inform the user of the event.# Otherwise if the option is set to no, the server aborts with an error# and refuses to start. When the option is set to no, the user requires# to fix the AOF file using the "redis-check-aof" utility before to restart# the server.## Note that if the AOF file will be found to be corrupted in the middle# the server will still exit with an error. This option only applies when# Redis will try to read more data from the AOF file but not enough bytes# will be found.aof-load-truncated yes ################################ LUA SCRIPTING ############################### # Max execution time of a Lua script in milliseconds.## If the maximum execution time is reached Redis will log that a script is# still in execution after the maximum allowed time and will start to# reply to queries with an error.## When a long running script exceeds the maximum execution time only the# SCRIPT KILL and SHUTDOWN NOSAVE commands are available. The first can be# used to stop a script that did not yet called write commands. The second# is the only way to shut down the server in the case a write command was# already issued by the script but the user doesn't want to wait for the natural# termination of the script.## Set it to 0 or a negative value for unlimited execution without warnings.lua-time-limit 5000 ################################ REDIS CLUSTER ################################# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++# WARNING EXPERIMENTAL: Redis Cluster is considered to be stable code, however# in order to mark it as "mature" we need to wait for a non trivial percentage# of users to deploy it in production.# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++## Normal Redis instances can't be part of a Redis Cluster; only nodes that are# started as cluster nodes can. In order to start a Redis instance as a# cluster node enable the cluster support uncommenting the following:## cluster-enabled yes # Every cluster node has a cluster configuration file. This file is not# intended to be edited by hand. It is created and updated by Redis nodes.# Every Redis Cluster node requires a different cluster configuration file.# Make sure that instances running in the same system do not have# overlapping cluster configuration file names.## cluster-config-file nodes-6379.conf # Cluster node timeout is the amount of milliseconds a node must be unreachable# for it to be considered in failure state.# Most other internal time limits are multiple of the node timeout.## cluster-node-timeout 15000 # A slave of a failing master will avoid to start a failover if its data# looks too old.## There is no simple way for a slave to actually have a exact measure of# its "data age", so the following two checks are performed:## 1) If there are multiple slaves able to failover, they exchange messages# in order to try to give an advantage to the slave with the best# replication offset (more data from the master processed).# Slaves will try to get their rank by offset, and apply to the start# of the failover a delay proportional to their rank.## 2) Every single slave computes the time of the last interaction with# its master. This can be the last ping or command received (if the master# is still in the "connected" state), or the time that elapsed since the# disconnection with the master (if the replication link is currently down).# If the last interaction is too old, the slave will not try to failover# at all.## The point "2" can be tuned by user. Specifically a slave will not perform# the failover if, since the last interaction with the master, the time# elapsed is greater than:## (node-timeout * slave-validity-factor) + repl-ping-slave-period## So for example if node-timeout is 30 seconds, and the slave-validity-factor# is 10, and assuming a default repl-ping-slave-period of 10 seconds, the# slave will not try to failover if it was not able to talk with the master# for longer than 310 seconds.## A large slave-validity-factor may allow slaves with too old data to failover# a master, while a too small value may prevent the cluster from being able to# elect a slave at all.## For maximum availability, it is possible to set the slave-validity-factor# to a value of 0, which means, that slaves will always try to failover the# master regardless of the last time they interacted with the master.# (However they'll always try to apply a delay proportional to their# offset rank).## Zero is the only value able to guarantee that when all the partitions heal# the cluster will always be able to continue.## cluster-slave-validity-factor 10 # Cluster slaves are able to migrate to orphaned masters, that are masters# that are left without working slaves. This improves the cluster ability# to resist to failures as otherwise an orphaned master can't be failed over# in case of failure if it has no working slaves.## Slaves migrate to orphaned masters only if there are still at least a# given number of other working slaves for their old master. This number# is the "migration barrier". A migration barrier of 1 means that a slave# will migrate only if there is at least 1 other working slave for its master# and so forth. It usually reflects the number of slaves you want for every# master in your cluster.## Default is 1 (slaves migrate only if their masters remain with at least# one slave). To disable migration just set it to a very large value.# A value of 0 can be set but is useful only for debugging and dangerous# in production.## cluster-migration-barrier 1 # By default Redis Cluster nodes stop accepting queries if they detect there# is at least an hash slot uncovered (no available node is serving it).# This way if the cluster is partially down (for example a range of hash slots# are no longer covered) all the cluster becomes, eventually, unavailable.# It automatically returns available as soon as all the slots are covered again.## However sometimes you want the subset of the cluster which is working,# to continue to accept queries for the part of the key space that is still# covered. In order to do so, just set the cluster-require-full-coverage# option to no.## cluster-require-full-coverage yes # In order to setup your cluster make sure to read the documentation# available at http://redis.io web site. ################################## SLOW LOG ################################### # The Redis Slow Log is a system to log queries that exceeded a specified# execution time. The execution time does not include the I/O operations# like talking with the client, sending the reply and so forth,# but just the time needed to actually execute the command (this is the only# stage of command execution where the thread is blocked and can not serve# other requests in the meantime).## You can configure the slow log with two parameters: one tells Redis# what is the execution time, in microseconds, to exceed in order for the# command to get logged, and the other parameter is the length of the# slow log. When a new command is logged the oldest one is removed from the# queue of logged commands. # The following time is expressed in microseconds, so 1000000 is equivalent# to one second. Note that a negative number disables the slow log, while# a value of zero forces the logging of every command.slowlog-log-slower-than 10000 # There is no limit to this length. Just be aware that it will consume memory.# You can reclaim memory used by the slow log with SLOWLOG RESET.slowlog-max-len 128 #Fixd VM Config by feng root@moonstack.org##vm-enabled no#vm-swap-file /moonstack/tmp/redis/redis.swap#vm-max-memory 0#vm-page-size 32#vm-pages 134217728#vm-max-threads 4 ################################ LATENCY MONITOR ############################## # The Redis latency monitoring subsystem samples different operations# at runtime in order to collect data related to possible sources of# latency of a Redis instance.## Via the LATENCY command this information is available to the user that can# print graphs and obtain reports.## The system only logs operations that were performed in a time equal or# greater than the amount of milliseconds specified via the# latency-monitor-threshold configuration directive. When its value is set# to zero, the latency monitor is turned off.## By default latency monitoring is disabled since it is mostly not needed# if you don't have latency issues, and collecting data has a performance# impact, that while very small, can be measured under big load. Latency# monitoring can easily be enabled at runtime using the command# "CONFIG SET latency-monitor-threshold <milliseconds>" if needed.latency-monitor-threshold 0 ############################# EVENT NOTIFICATION ############################## # Redis can notify Pub/Sub clients about events happening in the key space.# This feature is documented at http://redis.io/topics/notifications## For instance if keyspace events notification is enabled, and a client# performs a DEL operation on key "foo" stored in the Database 0, two# messages will be published via Pub/Sub:## PUBLISH __keyspace@0__:foo del# PUBLISH __keyevent@0__:del foo## It is possible to select the events that Redis will notify among a set# of classes. Every class is identified by a single character:## K Keyspace events, published with __keyspace@<db>__ prefix.# E Keyevent events, published with __keyevent@<db>__ prefix.# g Generic commands (non-type specific) like DEL, EXPIRE, RENAME, ...# $ String commands# l List commands# s Set commands# h Hash commands# z Sorted set commands# x Expired events (events generated every time a key expires)# e Evicted events (events generated when a key is evicted for maxmemory)# A Alias for g$lshzxe, so that the "AKE" string means all the events.## The "notify-keyspace-events" takes as argument a string that is composed# of zero or multiple characters. The empty string means that notifications# are disabled.## Example: to enable list and generic events, from the point of view of the# event name, use:## notify-keyspace-events Elg## Example 2: to get the stream of the expired keys subscribing to channel# name __keyevent@0__:expired use:## notify-keyspace-events Ex## By default all notifications are disabled because most users don't need# this feature and the feature has some overhead. Note that if you don't# specify at least one of K or E, no events will be delivered.notify-keyspace-events "" ############################### ADVANCED CONFIG ############################### # Hashes are encoded using a memory efficient data structure when they have a# small number of entries, and the biggest entry does not exceed a given# threshold. These thresholds can be configured using the following directives.hash-max-ziplist-entries 512hash-max-ziplist-value 64 # Similarly to hashes, small lists are also encoded in a special way in order# to save a lot of space. The special representation is only used when# you are under the following limits:list-max-ziplist-entries 512list-max-ziplist-value 64 # Sets have a special encoding in just one case: when a set is composed# of just strings that happen to be integers in radix 10 in the range# of 64 bit signed integers.# The following configuration setting sets the limit in the size of the# set in order to use this special memory saving encoding.set-max-intset-entries 512 # Similarly to hashes and lists, sorted sets are also specially encoded in# order to save a lot of space. This encoding is only used when the length and# elements of a sorted set are below the following limits:zset-max-ziplist-entries 128zset-max-ziplist-value 64 # HyperLogLog sparse representation bytes limit. The limit includes the# 16 bytes header. When an HyperLogLog using the sparse representation crosses# this limit, it is converted into the dense representation.## A value greater than 16000 is totally useless, since at that point the# dense representation is more memory efficient.## The suggested value is ~ 3000 in order to have the benefits of# the space efficient encoding without slowing down too much PFADD,# which is O(N) with the sparse encoding. The value can be raised to# ~ 10000 when CPU is not a concern, but space is, and the data set is# composed of many HyperLogLogs with cardinality in the 0 - 15000 range.hll-sparse-max-bytes 3000 # Active rehashing uses 1 millisecond every 100 milliseconds of CPU time in# order to help rehashing the main Redis hash table (the one mapping top-level# keys to values). The hash table implementation Redis uses (see dict.c)# performs a lazy rehashing: the more operation you run into a hash table# that is rehashing, the more rehashing "steps" are performed, so if the# server is idle the rehashing is never complete and some more memory is used# by the hash table.## The default is to use this millisecond 10 times every second in order to# actively rehash the main dictionaries, freeing memory when possible.## If unsure:# use "activerehashing no" if you have hard latency requirements and it is# not a good thing in your environment that Redis can reply from time to time# to queries with 2 milliseconds delay.## use "activerehashing yes" if you don't have such hard requirements but# want to free memory asap when possible.activerehashing yes # The client output buffer limits can be used to force disconnection of clients# that are not reading data from the server fast enough for some reason (a# common reason is that a Pub/Sub client can't consume messages as fast as the# publisher can produce them).## The limit can be set differently for the three different classes of clients:## normal -> normal clients including MONITOR clients# slave -> slave clients# pubsub -> clients subscribed to at least one pubsub channel or pattern## The syntax of every client-output-buffer-limit directive is the following:## client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>## A client is immediately disconnected once the hard limit is reached, or if# the soft limit is reached and remains reached for the specified number of# seconds (continuously).# So for instance if the hard limit is 32 megabytes and the soft limit is# 16 megabytes / 10 seconds, the client will get disconnected immediately# if the size of the output buffers reach 32 megabytes, but will also get# disconnected if the client reaches 16 megabytes and continuously overcomes# the limit for 10 seconds.## By default normal clients are not limited because they don't receive data# without asking (in a push way), but just after a request, so only# asynchronous clients may create a scenario where data is requested faster# than it can read.## Instead there is a default limit for pubsub and slave clients, since# subscribers and slaves receive data in a push fashion.## Both the hard or the soft limit can be disabled by setting them to zero.client-output-buffer-limit normal 0 0 0client-output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60 # Redis calls an internal function to perform many background tasks, like# closing connections of clients in timeout, purging expired keys that are# never requested, and so forth.## Not all tasks are performed with the same frequency, but Redis checks for# tasks to perform according to the specified "hz" value.## By default "hz" is set to 10. Raising the value will use more CPU when# Redis is idle, but at the same time will make Redis more responsive when# there are many keys expiring at the same time, and timeouts may be# handled with more precision.## The range is between 1 and 500, however a value over 100 is usually not# a good idea. Most users should use the default of 10 and raise this up to# 100 only in environments where very low latency is required.hz 10 # When a child rewrites the AOF file, if the following option is enabled# the file will be fsync-ed every 32 MB of data generated. This is useful# in order to commit the file to the disk more incrementally and avoid# big latency spikes.aof-rewrite-incremental-fsync yes requirepass 8ff35947f8efe8db806622f6a98a1ea3

其实我本想只发精简后的配置文件来着,但是为了以后改配置,还是贴出来完整的好了.

上面的这个配置文件有几个参数我们需要注意,
requirepass 这个是redis的密码,这里我用了一串md5
bind 是redis服务所监听的地址, 因为我们的redis在业务中是Broker,所以我用了 0.0.0.0 ,根据业务的不同,如果不需要外网服务,那么请果断写入 127.0.0.1 不要犹豫
daemonize 是否作为守护程序运行,前面我们配了服务启动脚本,这里当然必须yes

好了,基本上就这么些要注意的配置,细节优化还是要移步官方访问并获取各项参数的具体配置信息.

保存退出后,我们继续敲命令

123456

# chmod +x /etc/rc.d/init.d/redis//给丫执行权限# chkconfig redis --add//添加开机启动//然后就是启动了service redis start

2. logstash配置
本文中,我们做了AIO(all in one) 所有的业务都在1台主机上，那么我需要两个logstash节点
1，Indexer；2，Shipper；
其中， Shipper可以理解为监控节点，将所有的（分布式业务中）业务产生的消息（log）发送给Broker，然后，Indexer从Broker读取数据并推送给ES，我们就可以用kibana中获取想要的信息了

好,我们先配indexer.conf

12345678910111213141516171819

input { redis { host => "127.0.0.1" port => 6379 password => "8ff35947f8efe8db806622f6a98a1ea3" type => "redis-input" data_type => "list" key => "key_count" }} output { stdout {} elasticsearch { cluster => "elasticsearch" codec => "json" protocol => "http" }}

因为我们是AIO,所以,这里host我写了127.0.0.1,因为我们的redis监听了0.0.0.0,redis和这个indexer在同一台主机上,所以我写了本地ip.
这个配置文件中,我使用了两个插件,redis和elasticsearch,分别制定了输入数据源和输出目的对象.
当前,我们对配置中的参数还并不了解,我们暂时不去管它.
上面的这个配置文件保存在/etc/logstash/conf.d/目录中,这里我起名为indexer.conf

然后我们再配shipper.conf

1234567891011121314151617

input { file { path => ["/var/log/*.log", "/var/log/message"] type => "system" start_position => "beginning" }}output { stdout {} redis { host => "127.0.0.1" port => 6379 password => "8ff35947f8efe8db806622f6a98a1ea3" data_type => "list" key => "key_count" }}

这个文件我们统一保存在/etc/logstash/conf.d/目录中,这里起名为shipper.conf
这里我们纠结一下input的file配置:
首先,这个file插件只是在进程运行的注册阶段初始化的一个FileWatch对象,它并不能支持l类似fluentd那样的

1	path => "/path/to/%{+yyyy/MM/dd/hh}.log"

type则是我们要告诉elk的这个log的类型,这个type参数是个字符串,官方的原文是

12

The type is stored as part of the event itself, so you can also use the type to search for it in Kibana.If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.

我们暂且理解为字段类型.

接下来的start_position参数则是logstash 从什么位置开始读取文件数据，默认是结束位置，也就是说 logstash 进程会以类似 tail -F 的形式运行。如果你是要导入原有数据，把这个设定改成 "beginning"，logstash 进程就从头开始读取，有点类似 cat，但是读到最后一行不会终止，而是继续变成 tail -F。

当我们把这两个配置文件保存到/etc/logstash/conf.d/目录中后,执行下面的命令即可

12	service logstash stopservice logstash start

为神马我们这么方便? 因为我们rpm包安装滴..~~

3.elasticsearch 启动
接下来,启动elasticsearch

1	service elasticsearch start

啊噢~ 居然报错了,我这里报错

12	which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME

可是,我们明明安装了java啊, 也可以运行,
根据提示,我们这样做

1	ln -sv /usr/local/java/bin/java /usr/bin/java

当然你把软链接建立到sbin目录也可以.
在service启动, 这次ok了....

也许你会问,这里不需要配置吗? 至少现在不需要配置

4.kibana启动
还记得kibana我们是怎么安装的吗? 也在系统path目录下设置了软链接

果断敲入下面的神秘代码, 启动kibana

1

kibana &

我为什么加了个 " & " 号, 让它去后台执行嘛

ok, 这时候,我们在浏览器输入
http://ip:5601 就可以访问啦

它是这个样子








ok, 填坑过程第二篇简单配置先到此为止, 下一篇我们细说logstash的配置和kibana的使用,开始我们的系统监控.
本文出自 “mooooooooon” 博客，请务必保留此出处/article/4513137.html