秦悦明的运维笔记

SRE职责

SRE 职责

  1. 可用性改进
  2. 延迟优化
  3. 性能优化
  4. 效率优化
  5. 变更管理
  6. 监控
  7. 紧急事务处理
  8. 容量规划与管理

摘自[SRE google运维解密]，2017年努力践行SRE。

mysql-5.7编译安装

1. 编译

下载:
直接用国内的源，速度杠杠的:

1
http://mirrors.sohu.com/mysql/MySQL-5.7/mysql-boost-5.7.17.tar.gz

最新源码包是

1
2
http://mirrors.sohu.com/mysql/MySQL-5.7/mysql-5.7.18.tar.gz
http://mirrors.sohu.com/mysql/MySQL-5.7/mysql-boost-5.7.18.tar.gz

依赖(centos7为例):

1
sudo yum -y install make gcc-c++ cmake bison-devel bison ncurses-devel

编译跟5.6稍微有点不一样，依赖新版的boost了。不要cmake时候自动下载，会被墙掉。直接下载boost版本的源码包就行了，剩下的套路跟5.6的一样。

1
2
3
4
cd BUILD
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql57  -DWITH_BOOST=../boost ..
make 
make install

2.单实例初始化和运行

mysql_install_db已经被废弃了。直接用

1
2
3
4
5
6
7
# bin/mysqld --initialize --user=mysql --datadir=/vdb/mysql1
2017-06-19T01:18:19.299580Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-06-19T01:18:19.458771Z 0 [Warning] InnoDB: New log files created, LSN=45790
2017-06-19T01:18:19.482412Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2017-06-19T01:18:19.537139Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 2de7b52f-548d-11e7-b57b-52560acdcad4.
2017-06-19T01:18:19.537944Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2017-06-19T01:18:19.538602Z 1 [Note] A temporary password is generated for root@localhost: C#lF1kb*KkV/

运行，

1
2
3
4
5
6
7
8
9
10
11
# cp support-files/mysql.server .
修改mysql.server中的
basedir=xxx
datadir=xxx
修改 /etc/my.cnf 中配置。
vim /etc/my.cnf
# ./mysql.server start 
Starting MySQL.Logging to '/vdb/mysql1/10-205-202-212.err'.  [  OK  ]

用上面系统生成的随机密码进去，然后修改成自己的密码完成整个初始化

1
alter user root@localhost identified by '123123';

3.多实例部署

主要是用到了mysqld_multi,要在my.cnf里面填上多个实例的信息，格式如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[mysqld_multi]
mysqld     = /usr/local/mysql57/bin/mysqld_safe
mysqladmin = /usr/local/mysql57/bin/mysqladmin
user       = multi_admin
password   = my_password
[mysqld2]
socket     = /tmp/mysql.sock2
port       = 3307
pid-file   = /usr/local/mysql57/data2/hostname.pid2
datadir    = /usr/local/mysql57/data2
language   = /usr/local/mysql57/share/mysql/english
user       = unix_user1
[mysqld3]
mysqld     = /path/to/mysqld_safe
ledir      = /path/to/mysqld-binary/
mysqladmin = /path/to/mysqladmin
socket     = /tmp/mysql.sock3
port       = 3308
pid-file   = /usr/local/mysql57/data3/hostname.pid3
datadir    = /usr/local/mysql57/data3
language   = /usr/local/mysql57/share/mysql/swedish
user       = unix_user2

用 mysqld_multi –example可以直接查看。很方便,mysqld_multi里面的user和password 主要是用来关闭实例时候用的mysql 用户，所以先要附权限

1
GRANT SHUTDOWN ON *.* TO multi_admin@localhost IDENTIFIED BY 'password'

然后配置上就可以了~很方便。

mysql-GTID

1. GTID概念

简化了很多主从复制上面的问题。不需要像之前版本需要通过binlog的pos来确定从哪开始复制。
GTID = source_id:transaction_id
source_id是一个mysql自动生成的uuid，在datadir下面的auto.cnf里面,例如:

1
server-uuid=6cdd37fa-d3d8-11e6-b843-00163e0030cf

transaction_id是事务id，递增的一个序列。连起来标识一个事务。

2. master上面设置

1
2
3
4
5
6
7
binlog-format=ROW
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
log-bin=master-bin
log-bin-index=master-bin.index
server_id = 1

主要就是加了gtid-mode,enforce-gtid-consistency

3. slave设置

1
2
3
4
5
6
7
8
server_id = 222
gtid-mode=on
log-bin=slave-bin
log-slave-updates=true
enforce-gtid-consistency=true
binlog-format=ROW
log-bin-index=slave-bin.index
relay-log=slave1-relay-bin

4. grant权限

1
2
CREATE USER 'repl'@'%' IDENTIFIED BY 'slavepass';
grant REPLICATION SLAVE ON *.* TO 'repl'@'%';

5. change master

1
change master to master_host='10.168.105.153', master_user='repl',master_password='slavepass',master_auto_position=1;

6. 启动slave

1
start slave;

Alisql安装与压测

alisql试用

alisql简介

专访丁奇：阿里云即将开源AliSQL，超大并发、针对秒杀优化
本质上是一个mysql分支。

安装

alisql是从mysql5.6fork出来的版本，所以安装和5.6的一样，比较套路跟之前的mysql安装没有什么区别:

1
2
3
4
5
yum -y install make gcc-c++ cmake bison-devel bison ncurses-devel
cd BUILD
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql56 ..
make
make install

sysbench压测

压测很关键，不同机器配置，不通server参数下得出的结果是很不同的，压测得到的数据有直观的说服力。
上系统之前先压一下，了解db的极限在哪。
sysbench安装

1
2
3
4
5
6
git clone https://github.com/akopytov/sysbench
git checkout origin/0.5
yum install -y automake libtool
./autogen.sh
./configure
make

sysbench测试

有多套lua脚本来测不同场景。

1
2
3
tests/db/bulk_insert.lua  tests/db/insert.lua       tests/db/parallel_prepare.lua      tests/db/select_random_ranges.lua
tests/db/common.lua       tests/db/oltp.lua         tests/db/select.lua                tests/db/update_index.lua
tests/db/delete.lua       tests/db/oltp_simple.lua  tests/db/select_random_points.lua  tests/db/update_non_index.lua
1
2
3
4
./sysbench --test=./tests/db/oltp.lua --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-db=dba_test --oltp-table-size=500000 --oltp_tables_count=3 --rand-init=on --mysql-user=test --mysql-password=test prepare
./sysbench --test=./tests/db/oltp.lua --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-db=dba_test --num-threads=8 --oltp-read-only=off --report-interval=10  --max-requests=0 --percentile=99 --oltp-table-size=500000 --oltp_tables_count=3 --rand-init=on --max-time=100 --mysql-user=test --mysql-password=test run

hadoop-oldapi以及运行mapreduce程序

1. 程序

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
package c3;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.*;
import java.io.IOException;
import java.util.Iterator;
/**
 * Created by gqdw on 28/12/2016.
 */
public class WordCountOldApi {
    public static class MyMapper extends MapReduceBase implements
            Mapper<LongWritable, Text, Text, IntWritable>{
        public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
                        throws IOException{
            output.collect(new Text(value.toString()), new IntWritable(1));
        }
    }
    public static class MyReducer extends MapReduceBase implements
            Reducer<Text, IntWritable, Text, IntWritable>{
        public void reduce(Text key, Iterator<IntWritable> values,
                           OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException{
            int sum = 0;
            while(values.hasNext()) {
                sum += values.next().get();
            }
            output.collect(key, new IntWritable(sum));
        }
    }
    public static void main(String[] args) throws Exception {
        JobConf conf = new JobConf(WordCountOldApi.class);
        conf.setJobName("wordcount");
        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);
        conf.setMapperClass(MyMapper.class);
        conf.setCombinerClass(MyReducer.class);
        conf.setNumReduceTasks(1);
        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);
        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));
        JobClient.runJob(conf);
    }
}

maven里面加上依赖:

1
2
3
4
5
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.7.3</version>
</dependency>

编译完了以后得到一个jar包，prohadoop-1.0-SNAPSHOT.jar

2. 运行

其实很简单:

1
hadoop jar prohadoop-1.0-SNAPSHOT.jar c3.WordCountOldApi [input] [output]

input output是指hdfs上面的路径。
首先将文件上传到hdfs

1
hdfs dfs -put test.txt /test

之后即可运行:

1
hadoop jar prohadoop-1.0-SNAPSHOT.jar c3.WordCountOldApi  /test/test.txt /test/output

mapreduce程序将结果输出到了output目录，

1
2
-rw-r--r--   1 gqdw supergroup          0 2016-12-28 17:00 /test/output/_SUCCESS
-rw-r--r--   1 gqdw supergroup         27 2016-12-28 17:00 /test/output/part-00000

part-00000里面既是结果

1
2
3
4
java	1
job	1
test	3
word	1

About

weclome to qym's blog

Tags

Tag Cloud

Spring c++ django docker flask hadoop haproxy java linux mysql network nginx python redis salt spring tcp zabbix 负载均衡

Archives

Recents