秦悦明的运维笔记

spring-restfull

1. maven依赖

能用ide的用ide,或者用spring的命令行生成项目文件。

1
2
3
4
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-rest</artifactId>
</dependency>

2. 定义对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public class Greeting {
private final long id;
private final String content;
public Greeting(long id, String content) {
this.id = id;
this.content = content;
}
public long getId() {
return id;
}
public String getContent() {
return content;
}
}

3. 定义controller

GreetingController.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class GreetingController {
private static final String template = "Hello, %s!";
private final AtomicLong counter = new AtomicLong();
@RequestMapping("/greeting")
public Greeting greeting(@RequestParam(value="name", defaultValue="World") String name) {
return new Greeting(counter.incrementAndGet(),
String.format(template, name));
}
}

@RequestMapping 映射url地址,RequestMapping里面可以定义http方法@RequestMapping(method=GET).

RequestParam解析请求参数。

这里用了个String.format,套一个模板,还有就是id用了AtomicLong 原子对象。返回一个Greeting对象,spring boot自动配置生成json格式,里面的原理是data-rest依赖里面有com.jayway.jsonpath:json-path的依赖,spring的 MappingJackson2HttpMessageConverter自动将Greeting对象转成JSON了。

再详细了解一下@SpringBootApplication注解,其实他有3个注解组成

  1. Configuration 表明是个java配置类。
  2. EnableAutoConfiguration 打开springboot自动配置。
  3. ComponentScan 组件扫描。

4. run

运行访问得到结果:

1
2
3
4
5
http://localhost:8080/greeting
{"id":1,"content":"Hello, World!"}
http://localhost:8080/greeting?name=User
{"id":2,"content":"Hello, User!"}

spring data JPA

1. JPA提供的功能

JPA主要用于在java中处理持久化操作。对ORM特性和功能进行标准化。
分别定义了用来将对象模型映射到关系模型API,可以在对象上面执行CRUD操作,一种对象查询语言以及通过对象图获取数据的标准API。

简单来说就是ORM的java规范,比jdbc要方便太多太多。

JPA的实现有很多种,比如Hibernate,openJPA等。

2. maven依赖

1
2
3
4
5
6
7
8
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
</dependency>

3.1 定义一个entity

跟普通的BEAN对象差别不大,就是多了一些JPA的注解。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
package hello;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
@Entity
public class Customer {
@Id
@GeneratedValue(strategy=GenerationType.AUTO)
private Long id;
private String firstName;
private String lastName;
protected Customer() {}
public Customer(String firstName, String lastName) {
this.firstName = firstName;
this.lastName = lastName;
}
@Override
public String toString() {
return String.format(
"Customer[id=%d, firstName='%s', lastName='%s']",
id, firstName, lastName);
}
}

3.1 几个关键的注解

@Entity 定义实体,如果一个对象在数据库端与一个带有主键的纪录相对于,那么该对象就称为实体。可以用@Table注解指定表名。

@Id 标记了主键属性。

@GeneratedValue注解告诉JPA,应用程序将不负责分配主键值,stratepy属性指定实体类的ID生成策略。

4 定义一个repository接口

repository是spring dat jpa里面的特性,很magic,实现了CrudRepository接口就能自动实现CRUD操作了,不用自己去实现方法! amazing! I like it!

1
2
3
4
5
6
7
8
9
import java.util.List;
import org.springframework.data.repository.CrudRepository;
public interface CustomerRepository extends CrudRepository<Customer, Long> {
List<Customer> findByLastName(String lastName);
}

将Entity和Id的值传到CrudRepository泛型中去即可。也可以自己定义查询,比如上面定义的findByLastName。很方便。

5.调用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@Bean
public CommandLineRunner demo(CustomerRepository repository){
return (args)-> {
repository.save(new Customer("Jack","Bauer"));
repository.save(new Customer("Chloe", "O'Brian"));
repository.save(new Customer("Kim", "Bauer"));
repository.save(new Customer("David", "Palmer"));
repository.save(new Customer("Michelle", "Dessler"));
logger.info("Customers found with findAll():");
logger.info("--------------------------------");
for (Customer customer : repository.findAll()){
logger.info(customer.toString());
}
logger.info("");
Customer customer = repository.findOne(1L);
logger.info("Customer found with findOne(1L):");
logger.info("---------------------------------");
logger.info(customer.toString());
logger.info("");
logger.info("Customer found with findByLastName('Bauer'):");
logger.info("-------------------------------------------");
for (Customer bauer : repository.findByLastName("Bauer")){
logger.info(bauer.toString());
}
logger.info("");
};

直接开个CustomerRepository的对象,各种增删改查就可以了。

java-gson简单应用

1.gson

官网https://github.com/google/gson

1.1 maven

1
2
3
4
5
6
7
8
9
<dependencies>
<!-- Gson: Java to Json conversion -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
<scope>compile</scope>
</dependency>
</dependencies>

2.基本方法

  1. gson.toJson 将对象转换成json字符串
  2. gson.fromJson 将json字符串转换为object对象

2.1 toJson方法

定义一个对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class Res1 {
private String jsonrpc;
private String result;
private Integer id;
public String getResult() {
return result;
}
public void setId(Integer id) {
this.id = id;
}
public void setJsonrpc(String jsonrpc) {
this.jsonrpc = jsonrpc;
}
public void setResult(String result) {
this.result = result;
}
}

可以不用定义setter函数,用构造函数赋值,然后:

1
2
3
4
5
Res1 r1 = new Res1();
Res1.setId()
...
Gson gson = new Gson();
Res1 r1 = gson.toJson();

就可以转成json字符串。

2.2 fromJson方法

1
2
3
String j1 = JSON_STRING
Gson gson = new Gson();
Res1 res1 = gson.fromJson(JSON_STRING, Res1.class)

java-HttpClient

1. apache HttpClient

用于替换jdk的httpclient,
官网文档 http://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/fundamentals.htm

2.基本套路

1
2
3
4
5
6
7
8
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
<...>
} finally {
response.close();
}

2.1 http请求

一个http请求对象,有HttpGet,HttpPost等,HttpGet接收一个uri作参数,可以用字符串比如 “http://localhost", 注意加上http://, 也可以用URIBuilder来生成,如下:

1
2
3
4
5
6
7
8
9
10
11
URI uri = new URIBuilder()
.setScheme("http")
.setHost("www.google.com")
.setPath("/search")
.setParameter("q", "httpclient")
.setParameter("btnG", "Google Search")
.setParameter("aq", "f")
.setParameter("oq", "")
.build();
HttpGet httpget = new HttpGet(uri);
System.out.println(httpget.getURI());

如果想设置内容,可以用 httpPost.setEntity()方法,最简单的是用StringEntity声明一个,

1
2
3
String j1 = "xx"
StringEntity entity1 = new StringEntity(j1);
httpPost.setEntity(entity1);

也有setHeader这类实用的方法,

1
httpPost.setHeader("Content-Type","application/json-rpc");

2.2 http response回复

response一般是httpclient执行execute() HttpPost返回的对象。
可以用response.getStatusLine().toString()得到返回码。

2.3 http entity实体

一般用EntityUtils的toString(myEntity)将实体转换为字符串。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
HttpEntity entity = response.getEntity();
if (entity != null) {
long len = entity.getContentLength();
if (len != -1 && len < 2048) {
System.out.println(EntityUtils.toString(entity));
} else {
// Stream content out
}
}
} finally {
response.close();
}

还能用低级的instream流来处理实体

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try {
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
try {
// do something useful
} finally {
instream.close();
}
}
} finally {
response.close();
}

2.4 basic认证套路

1
2
3
4
5
6
7
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
UsernamePasswordCredentials credentials =
new UsernamePasswordCredentials("admin","PASSWORD");
credentialsProvider.setCredentials(AuthScope.ANY, credentials);
CloseableHttpClient client = HttpClientBuilder.create()
.setDefaultCredentialsProvider(credentialsProvider)
.build();

创建CredentialsProvider对象,UsernamePasswordCredentials对象,然后用setCredentials方法将credentialsProvider设置。
之后使用HttpClientBuilder方法来构建一个添加过认证的client,之后的调用没有任何区别。

redis与ssdb对比压测

1. ssdb简介

“一个高性能的支持丰富数据结构的 NoSQL 数据库, 用于替代 Redis.”官网上面是这么说的。
个人理解是一个兼容redis数据结构,底层用leveldb的nosql数据库。

2. ssdb安装

1
2
3
4
5
6
7
yum install -y autoconf gcc-c++
wget --no-check-certificate https://github.com/ideawu/ssdb/archive/master.zip
unzip master
cd ssdb-master
make
# optional, install ssdb in /usr/local/ssdb
make install

先得把gcc-c++安装好。如果有问题,目录删了重新编译。。如果说在编译鲁棒性上面redis说完虐ssdb,应该没人会反对吧。

3. 配置

基本都是默认配置,redis测了开aof的和不开aof,当然开aof qps会下降一些,后面有详细数据。
ssdb也是默认配置,开了binlog。
注意一下redis对内核设置有要求:

1
2
sysctl vm.overcommit_memory=1
echo never > /sys/kernel/mm/transparent_hugepage/enabled

两个版本分别为 4.0.6 和1.9.6。
机器是在8核16G的机器上面测的。

4. 压测注意事项

很有意思的是2个程序都自带了压测工具,redis的叫redis-benchmark,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests>] [-k <boolean>]
-h <hostname> Server hostname (default 127.0.0.1)
-p <port> Server port (default 6379)
-s <socket> Server socket (overrides host and port)
-a <password> Password for Redis Auth
-c <clients> Number of parallel connections (default 50)
-n <requests> Total number of requests (default 100000)
-d <size> Data size of SET/GET value in bytes (default 3)
--dbnum <db> SELECT the specified db number (default 0)
-k <boolean> 1=keep alive 0=reconnect (default 1)
-r <keyspacelen> Use random keys for SET/GET/INCR, random values for SADD
Using this option the benchmark will expand the string __rand_int__
inside an argument with a 12 digits number in the specified range
from 0 to keyspacelen-1. The substitution changes every time a command
is executed. Default tests use this to hit random keys in the
specified range.
-P <numreq> Pipeline <numreq> requests. Default 1 (no pipeline).
-e If server replies with errors, show them on stdout.
(no more than 1 error per second is displayed)
-q Quiet. Just show query/sec values
--csv Output in CSV format
-l Loop. Run the tests forever
-t <tests> Only run the comma separated list of tests. The test
names are the same as the ones produced as output.
-I Idle mode. Just open N idle connections and wait.

1
2
3
4
5
6
7
8
9
10
11
ssdb-bench - SSDB benchmark tool, 1.9.6
Copyright (c) 2013-2015 ssdb.io
Usage:
tools/ssdb-bench [ip] [port] [requests] [clients]
Options:
ip server ip (default 127.0.0.1)
port server port (default 8888)
requests Total number of requests (default 10000)
clients Number of parallel connections (default 50)

注意,client都是开50个线程,request却差了10倍。redis默认是10w,ssdb默认是1w,所以ssdb测时候指定了10w个连接。

1
2
redis-bench
tools/ssdb-bench 127.0.0.1 8888 100000 50

5. 结论

ssdb的结果:

redis:

PING_INLINE PING_BULK SET GET INCR LPUSH RPUSH LPOP RPOP SADD HSET SPOP LPUSH LRANGE_100 LRANGE_300 LRANGE_500 LRANGE_600 MSET
152207.00 137741.05 120192.30 141843.97 168634.06 179211.45 140056.03 150150.14 136798.91 144717.80 132978.73 142653.36 173010.38 49236.83 20898.64 15415.45 11823.13 116414.43
146627.56 158478.61 162866.44 149031.30 147275.41 149031.30 169491.53 175438.59 163398.70 166112.95 142045.45 159489.64 152905.20 63856.96 23218.02 14925.37 10511.93 74515.65

第二行是开了aof的数据,set反而高了。

ssdb:

set get del hset hget hdel zset zget zdel qpush qpop
82281 43160 65208 41184 43745 48159 38246 42734 36934 42031 30591

redis不论set,get都是15w左右的qps,ssdb差不多是3,4w左右,差距还是有一些的,redis基本上是全面碾压ssdb。
当然我个人觉得如果是实际业务的话,也应该在技术选型的时候压测一把,特定业务场景的压测非常重要。另外redis比ssdb优势的一点是其社区更加壮大,可运维性大于ssdb,当然这是题外话了。