redis value为对象的排序、分页的一种实现
2013-04-27 09:23
447 查看
-------基于HBase存储引擎并实现了排序、分页的Redis缓存策略
思路:
1、
当前系统缓存json,根据key存取value,key
和value都是String类型,直接返回前端。
2、
考虑支持分页,想到了缓存排好序的ResultSet,每一次前端请求,将所有数据缓存到redis,根据filter返回某页的数据,此部分数据是封装好的json。
考虑到使用list会增加开发的工作量,所以暂不用。
3、
考虑缓存ResultSet对象,所以该类要实现序列化,用hadoop的Writable,不用java本身的Serializable。
4、
前端需求不需要分页的仍然用原来接口,缓存json,用不同的annotation设置到查询方法上区分。
注:ResultSet为自己实现了的将hbase列存储数据转为按关系数据库行存储的数据结构。
具体实现:
1、由于每次都传给前端少量数据,所以要传给前端总记录条数方便其确定分页信息,可封转在json里。
2、rediscache操作类增加支持序列化的存取方法:
[java]
view plaincopyprint?
@Override
public ResultSet getResultSetCache(String key) {
try {
log.info("get object from redisCache :"+key);
ShardedJedis jedis = pool.getResource();
ResultSet value = SerializableUtils.deSerialize(jedis.get(key.getBytes()), ResultSet.class);
pool.returnResource(jedis);
return value;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public boolean setResultSetCache(String key, ResultSet value) {
try {
log.info("add object to redisCache :"+key);
ShardedJedis jedis = pool.getResource();
jedis.set(key.getBytes(),SerializableUtils.serialize(value));
pool.returnResource(jedis);
return true;
} catch (Exception e) {
e.printStackTrace();
}
return false;
}
其中SerializableUtils工具类实现对writable子类的序列化、反序列化操作:
[java]
view plaincopyprint?
public class SerializableUtils {
public static<T
extends Writable> byte[] serialize(Object value)
throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
DataOutputStream oos = new DataOutputStream(bos);
if (value instanceof Writable) {
((T) value).write(oos);
} else {
throw new IOException("not instanceof Writable!");
}
return bos.toByteArray();
}
public static <T
extends Writable> T deSerialize(byte[] byteCode,Class<T> classType)
throws IOException{
ByteArrayInputStream bis = new ByteArrayInputStream(byteCode);
T object = null;
try {
DataInputStream dis = new DataInputStream(bis);
object = (T)classType.newInstance();
object.readFields(dis);
return object;
} catch (IOException e) {
e.printStackTrace();
} catch (InstantiationException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
}
return null;
}
}
java本身的序列化反序列化工具类:
[java]
view plaincopyprint?
public class SerializableUtil {
public static<T
extends Serializable> byte[] serialize(Object value){
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(value);
} catch (IOException e) {
e.printStackTrace();
}
return bos.toByteArray();
}
public static <T
extends Serializable> T deSerialize(byte[] bytes){
ByteArrayInputStream bIs = new ByteArrayInputStream(bytes);
try {
ObjectInputStream oIs = new ObjectInputStream(bIs);
return (T)oIs.readObject();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
return null;
}
}
3、 增加两个annotation及两个切面处理类,annotation用以在具体查询方法上添加,区分需不需要排序,需要排序的缓存ResultSet,不需要的缓存Json。
需要分页的切面类主要代码:
[java]
view plaincopyprint?
String md5key = MD5Util.getMD5(filter.getValue().toString());
Object value = redisCacheManager.getResultSetCache(md5key);
String srcClzzName = "";
boolean flag = false;
if (null != value) {
srcClzzName = pjp.getTarget().getClass().getSimpleName();
JSONObject json = redisCacheManager.getJson(value,filter,srcClzzName);
return json;
} else if ("null".equals(value)) {
return null;
} else { //执行hbase查询
ResultSet orderRs = redisCacheManager.getResultSet(filter,srcClzzName);
boolean ifNeedCache = orderRs.size()>0?true:false;
if (ifNeedCache ==
true){
redisCacheManager.setResultSetCache(md5key, orderRs);
}
return redisCacheManager.getJson(orderRs, filter, srcClzzName);
}
getResultSet部分代码:
[java]
view plaincopyprint?
public ResultSet getOrderdResultSet(QueryFilter filter) {
SaleDetailsFilter Salefilter = (SaleDetailsFilter) filter;
String shopId = Salefilter.getShopId();
String date = Salefilter.getDate();//
int proType = Salefilter.getProType();//
byte sku =
1, spu = 0;
byte isSKU = Salefilter.getIsMergeSKU() ? spu : sku;
long[] proNum = Salefilter.getProNums();//
String[] itemNums = Salefilter.getItemNums();//
ResultSet result = new ResultSet();
try {
// set start row;
ByteBuffer rowkey = ByteBuffer.allocate(13);
rowkey.putInt(Integer.parseInt(shopId));
rowkey.putInt(Integer.parseInt(date));
// set product type, default 0
rowkey.putInt(proType);
rowkey.put(isSKU);
byte[] endkey = CommonUtil.getEndKey(rowkey.array(),"int,int,int,byte");
Where where = null;
List<OrderBy> orderlist = null;
if ((proNum != null && proNum.length >
0)) {根据前端传入的商品id查询记录
//拼where条件
for (int i =
0; i < proNum.length; i++) {
if(i>0){
where = or(eq("d:ProID",proNum[i]),where);
continue;
}
where = eq("d:ProID",proNum[i]);
}
orderlist = new ArrayList<OrderBy>();
Map<String,Integer> ordermap = Salefilter.getOrderBy();
if(null!=ordermap){
Set<String> keys = ordermap.keySet();
for(Iterator<String> it = keys.iterator();it.hasNext();){
String key = it.next();
Integer value = ordermap.get(key);
if(value==0){
orderlist.add(asc(key));
}else if(value==1){
orderlist.add(desc(key));
}
}
result = client.simpleQuery(rowkey.array(),endkey, select, tableName, where, orderlist,-1);
}else{
result = client.simpleQuery(rowkey.array(),endkey, select, tableName, where,
null,-1);
}
return result;
} else if (itemNums !=
null && itemNums.length > 0) {
..... } else {
...... }
} catch (Exception e) {
logger.error(e.getMessage(), e);
} catch (Throwable e) {
logger.error(e.getMessage(), e);
e.printStackTrace();
}
return null;
}
当然,用到的很多方法在cp基础上自定义的对hbase的查询接口。
4、
查询接口类增加相应属性、方法,并且hbase表要加入coprocessor的支持,因为用到了聚合查询。
从缓存得到某一页的数据返回前端:
[java]
view plaincopyprint?
public JSONObject getPagedJson(Object value, QueryFilter filter,String srcClzzName)
throws JSONException {
JSONObject result = new JSONObject();
ResultSet tempRs = (ResultSet) value;
long size = tempRs.size();
ResultSet realRs = new ResultSet();
SaleDetailsFilter sdf = (SaleDetailsFilter) filter;
int pageNum = sdf.getPageNum();
int start = (pageNum -
1) * PAGE;
int maxend = PAGE * pageNum-1;
long end = size-(PAGE*pageNum)>=0?maxend:start+size-(pageNum-1)*PAGE-1;
for (int i = start; i <= end; i++) {
realRs.add(tempRs.get(i));
}
JSONObject temp = new JSONObject();
if (realRs.size() >
0) {
temp = trans.toJSONObject(realRs,select);
}
result.put(Transformer.DTL, temp);
result.put(Transformer.SIZE, size);
return result;
}
5、
查询接口类对应的filter增加两个属性,orderby和pageNum。Orderby表示排序的列,map类型,key表示列,值为0或1,表示升序或降序,默认为空,表示不排序。PageNum表示页号,默认第一页。
并将orderby纳入到查询条件中作为生成redis key的json。
pageNum是不能纳入的,因为前端翻页时应该对应redis中的一份数据。redis中的数据的份数由查询的种类决定,每一份都是查询的全量数据。后续考虑用list或索引的方式实现。
6、
redis集群安装—略。
7、
redis java客户端配置文件在web-inf下,包括连接池配置,属性配置,切面配置。
思路:
1、
当前系统缓存json,根据key存取value,key
和value都是String类型,直接返回前端。
2、
考虑支持分页,想到了缓存排好序的ResultSet,每一次前端请求,将所有数据缓存到redis,根据filter返回某页的数据,此部分数据是封装好的json。
考虑到使用list会增加开发的工作量,所以暂不用。
3、
考虑缓存ResultSet对象,所以该类要实现序列化,用hadoop的Writable,不用java本身的Serializable。
4、
前端需求不需要分页的仍然用原来接口,缓存json,用不同的annotation设置到查询方法上区分。
注:ResultSet为自己实现了的将hbase列存储数据转为按关系数据库行存储的数据结构。
具体实现:
1、由于每次都传给前端少量数据,所以要传给前端总记录条数方便其确定分页信息,可封转在json里。
2、rediscache操作类增加支持序列化的存取方法:
[java]
view plaincopyprint?
@Override
public ResultSet getResultSetCache(String key) {
try {
log.info("get object from redisCache :"+key);
ShardedJedis jedis = pool.getResource();
ResultSet value = SerializableUtils.deSerialize(jedis.get(key.getBytes()), ResultSet.class);
pool.returnResource(jedis);
return value;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public boolean setResultSetCache(String key, ResultSet value) {
try {
log.info("add object to redisCache :"+key);
ShardedJedis jedis = pool.getResource();
jedis.set(key.getBytes(),SerializableUtils.serialize(value));
pool.returnResource(jedis);
return true;
} catch (Exception e) {
e.printStackTrace();
}
return false;
}
@Override public ResultSet getResultSetCache(String key) { try { log.info("get object from redisCache :"+key); ShardedJedis jedis = pool.getResource(); ResultSet value = SerializableUtils.deSerialize(jedis.get(key.getBytes()), ResultSet.class); pool.returnResource(jedis); return value; } catch (Exception e) { e.printStackTrace(); } return null; } @Override public boolean setResultSetCache(String key, ResultSet value) { try { log.info("add object to redisCache :"+key); ShardedJedis jedis = pool.getResource(); jedis.set(key.getBytes(),SerializableUtils.serialize(value)); pool.returnResource(jedis); return true; } catch (Exception e) { e.printStackTrace(); } return false; }
其中SerializableUtils工具类实现对writable子类的序列化、反序列化操作:
[java]
view plaincopyprint?
public class SerializableUtils {
public static<T
extends Writable> byte[] serialize(Object value)
throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
DataOutputStream oos = new DataOutputStream(bos);
if (value instanceof Writable) {
((T) value).write(oos);
} else {
throw new IOException("not instanceof Writable!");
}
return bos.toByteArray();
}
public static <T
extends Writable> T deSerialize(byte[] byteCode,Class<T> classType)
throws IOException{
ByteArrayInputStream bis = new ByteArrayInputStream(byteCode);
T object = null;
try {
DataInputStream dis = new DataInputStream(bis);
object = (T)classType.newInstance();
object.readFields(dis);
return object;
} catch (IOException e) {
e.printStackTrace();
} catch (InstantiationException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
}
return null;
}
}
public class SerializableUtils { public static<T extends Writable> byte[] serialize(Object value) throws IOException { ByteArrayOutputStream bos = new ByteArrayOutputStream(); DataOutputStream oos = new DataOutputStream(bos); if (value instanceof Writable) { ((T) value).write(oos); } else { throw new IOException("not instanceof Writable!"); } return bos.toByteArray(); } public static <T extends Writable> T deSerialize(byte[] byteCode,Class<T> classType) throws IOException{ ByteArrayInputStream bis = new ByteArrayInputStream(byteCode); T object = null; try { DataInputStream dis = new DataInputStream(bis); object = (T)classType.newInstance(); object.readFields(dis); return object; } catch (IOException e) { e.printStackTrace(); } catch (InstantiationException e) { e.printStackTrace(); } catch (IllegalAccessException e) { e.printStackTrace(); } return null; } }
java本身的序列化反序列化工具类:
[java]
view plaincopyprint?
public class SerializableUtil {
public static<T
extends Serializable> byte[] serialize(Object value){
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(value);
} catch (IOException e) {
e.printStackTrace();
}
return bos.toByteArray();
}
public static <T
extends Serializable> T deSerialize(byte[] bytes){
ByteArrayInputStream bIs = new ByteArrayInputStream(bytes);
try {
ObjectInputStream oIs = new ObjectInputStream(bIs);
return (T)oIs.readObject();
} catch (IOException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
return null;
}
}
public class SerializableUtil { public static<T extends Serializable> byte[] serialize(Object value){ ByteArrayOutputStream bos = new ByteArrayOutputStream(); try { ObjectOutputStream oos = new ObjectOutputStream(bos); oos.writeObject(value); } catch (IOException e) { e.printStackTrace(); } return bos.toByteArray(); } public static <T extends Serializable> T deSerialize(byte[] bytes){ ByteArrayInputStream bIs = new ByteArrayInputStream(bytes); try { ObjectInputStream oIs = new ObjectInputStream(bIs); return (T)oIs.readObject(); } catch (IOException e) { e.printStackTrace(); } catch (ClassNotFoundException e) { e.printStackTrace(); } return null; } }
3、 增加两个annotation及两个切面处理类,annotation用以在具体查询方法上添加,区分需不需要排序,需要排序的缓存ResultSet,不需要的缓存Json。
需要分页的切面类主要代码:
[java]
view plaincopyprint?
String md5key = MD5Util.getMD5(filter.getValue().toString());
Object value = redisCacheManager.getResultSetCache(md5key);
String srcClzzName = "";
boolean flag = false;
if (null != value) {
srcClzzName = pjp.getTarget().getClass().getSimpleName();
JSONObject json = redisCacheManager.getJson(value,filter,srcClzzName);
return json;
} else if ("null".equals(value)) {
return null;
} else { //执行hbase查询
ResultSet orderRs = redisCacheManager.getResultSet(filter,srcClzzName);
boolean ifNeedCache = orderRs.size()>0?true:false;
if (ifNeedCache ==
true){
redisCacheManager.setResultSetCache(md5key, orderRs);
}
return redisCacheManager.getJson(orderRs, filter, srcClzzName);
}
String md5key = MD5Util.getMD5(filter.getValue().toString()); Object value = redisCacheManager.getResultSetCache(md5key); String srcClzzName = ""; boolean flag = false; if (null != value) { srcClzzName = pjp.getTarget().getClass().getSimpleName(); JSONObject json = redisCacheManager.getJson(value,filter,srcClzzName); return json; } else if ("null".equals(value)) { return null; } else { //执行hbase查询 ResultSet orderRs = redisCacheManager.getResultSet(filter,srcClzzName); boolean ifNeedCache = orderRs.size()>0?true:false; if (ifNeedCache == true){ redisCacheManager.setResultSetCache(md5key, orderRs); } return redisCacheManager.getJson(orderRs, filter, srcClzzName); }
getResultSet部分代码:
[java]
view plaincopyprint?
public ResultSet getOrderdResultSet(QueryFilter filter) {
SaleDetailsFilter Salefilter = (SaleDetailsFilter) filter;
String shopId = Salefilter.getShopId();
String date = Salefilter.getDate();//
int proType = Salefilter.getProType();//
byte sku =
1, spu = 0;
byte isSKU = Salefilter.getIsMergeSKU() ? spu : sku;
long[] proNum = Salefilter.getProNums();//
String[] itemNums = Salefilter.getItemNums();//
ResultSet result = new ResultSet();
try {
// set start row;
ByteBuffer rowkey = ByteBuffer.allocate(13);
rowkey.putInt(Integer.parseInt(shopId));
rowkey.putInt(Integer.parseInt(date));
// set product type, default 0
rowkey.putInt(proType);
rowkey.put(isSKU);
byte[] endkey = CommonUtil.getEndKey(rowkey.array(),"int,int,int,byte");
Where where = null;
List<OrderBy> orderlist = null;
if ((proNum != null && proNum.length >
0)) {根据前端传入的商品id查询记录
//拼where条件
for (int i =
0; i < proNum.length; i++) {
if(i>0){
where = or(eq("d:ProID",proNum[i]),where);
continue;
}
where = eq("d:ProID",proNum[i]);
}
orderlist = new ArrayList<OrderBy>();
Map<String,Integer> ordermap = Salefilter.getOrderBy();
if(null!=ordermap){
Set<String> keys = ordermap.keySet();
for(Iterator<String> it = keys.iterator();it.hasNext();){
String key = it.next();
Integer value = ordermap.get(key);
if(value==0){
orderlist.add(asc(key));
}else if(value==1){
orderlist.add(desc(key));
}
}
result = client.simpleQuery(rowkey.array(),endkey, select, tableName, where, orderlist,-1);
}else{
result = client.simpleQuery(rowkey.array(),endkey, select, tableName, where,
null,-1);
}
return result;
} else if (itemNums !=
null && itemNums.length > 0) {
..... } else {
...... }
} catch (Exception e) {
logger.error(e.getMessage(), e);
} catch (Throwable e) {
logger.error(e.getMessage(), e);
e.printStackTrace();
}
return null;
}
public ResultSet getOrderdResultSet(QueryFilter filter) { SaleDetailsFilter Salefilter = (SaleDetailsFilter) filter; String shopId = Salefilter.getShopId(); String date = Salefilter.getDate();// int proType = Salefilter.getProType();// byte sku = 1, spu = 0; byte isSKU = Salefilter.getIsMergeSKU() ? spu : sku; long[] proNum = Salefilter.getProNums();// String[] itemNums = Salefilter.getItemNums();// ResultSet result = new ResultSet(); try { // set start row; ByteBuffer rowkey = ByteBuffer.allocate(13); rowkey.putInt(Integer.parseInt(shopId)); rowkey.putInt(Integer.parseInt(date)); // set product type, default 0 rowkey.putInt(proType); rowkey.put(isSKU); byte[] endkey = CommonUtil.getEndKey(rowkey.array(),"int,int,int,byte"); Where where = null; List<OrderBy> orderlist = null; if ((proNum != null && proNum.length > 0)) {根据前端传入的商品id查询记录 //拼where条件 for (int i = 0; i < proNum.length; i++) { if(i>0){ where = or(eq("d:ProID",proNum[i]),where); continue; } where = eq("d:ProID",proNum[i]); } orderlist = new ArrayList<OrderBy>(); Map<String,Integer> ordermap = Salefilter.getOrderBy(); if(null!=ordermap){ Set<String> keys = ordermap.keySet(); for(Iterator<String> it = keys.iterator();it.hasNext();){ String key = it.next(); Integer value = ordermap.get(key); if(value==0){ orderlist.add(asc(key)); }else if(value==1){ orderlist.add(desc(key)); } } result = client.simpleQuery(rowkey.array(),endkey, select, tableName, where, orderlist,-1); }else{ result = client.simpleQuery(rowkey.array(),endkey, select, tableName, where, null,-1); } return result; } else if (itemNums != null && itemNums.length > 0) { ..... } else { ...... } } catch (Exception e) { logger.error(e.getMessage(), e); } catch (Throwable e) { logger.error(e.getMessage(), e); e.printStackTrace(); } return null; }
当然,用到的很多方法在cp基础上自定义的对hbase的查询接口。
4、
查询接口类增加相应属性、方法,并且hbase表要加入coprocessor的支持,因为用到了聚合查询。
从缓存得到某一页的数据返回前端:
[java]
view plaincopyprint?
public JSONObject getPagedJson(Object value, QueryFilter filter,String srcClzzName)
throws JSONException {
JSONObject result = new JSONObject();
ResultSet tempRs = (ResultSet) value;
long size = tempRs.size();
ResultSet realRs = new ResultSet();
SaleDetailsFilter sdf = (SaleDetailsFilter) filter;
int pageNum = sdf.getPageNum();
int start = (pageNum -
1) * PAGE;
int maxend = PAGE * pageNum-1;
long end = size-(PAGE*pageNum)>=0?maxend:start+size-(pageNum-1)*PAGE-1;
for (int i = start; i <= end; i++) {
realRs.add(tempRs.get(i));
}
JSONObject temp = new JSONObject();
if (realRs.size() >
0) {
temp = trans.toJSONObject(realRs,select);
}
result.put(Transformer.DTL, temp);
result.put(Transformer.SIZE, size);
return result;
}
public JSONObject getPagedJson(Object value, QueryFilter filter,String srcClzzName) throws JSONException { JSONObject result = new JSONObject(); ResultSet tempRs = (ResultSet) value; long size = tempRs.size(); ResultSet realRs = new ResultSet(); SaleDetailsFilter sdf = (SaleDetailsFilter) filter; int pageNum = sdf.getPageNum(); int start = (pageNum - 1) * PAGE; int maxend = PAGE * pageNum-1; long end = size-(PAGE*pageNum)>=0?maxend:start+size-(pageNum-1)*PAGE-1; for (int i = start; i <= end; i++) { realRs.add(tempRs.get(i)); } JSONObject temp = new JSONObject(); if (realRs.size() > 0) { temp = trans.toJSONObject(realRs,select); } result.put(Transformer.DTL, temp); result.put(Transformer.SIZE, size); return result; }
5、
查询接口类对应的filter增加两个属性,orderby和pageNum。Orderby表示排序的列,map类型,key表示列,值为0或1,表示升序或降序,默认为空,表示不排序。PageNum表示页号,默认第一页。
并将orderby纳入到查询条件中作为生成redis key的json。
pageNum是不能纳入的,因为前端翻页时应该对应redis中的一份数据。redis中的数据的份数由查询的种类决定,每一份都是查询的全量数据。后续考虑用list或索引的方式实现。
6、
redis集群安装—略。
7、
redis java客户端配置文件在web-inf下,包括连接池配置,属性配置,切面配置。
相关文章推荐
- redis value为对象的排序、分页的一种实现
- redis value为对象的排序、分页的一种实现
- 在ASP.NET MVC5中实现具有服务器端过滤、排序和分页的GridView
- 在struts中分页的一种实现
- DataGrid排序错误:必须至少有一个对象实现
- JavaScript基于对象方法实现数组去重及排序操作示例
- 基于spark排序的一种更廉价的实现方案-附基于spark的性能测试
- SpringBoot JPA实现增删改查、分页、排序、事务操作等功能
- 79 1.编写实现链表排序的一种算法。说明为什么你会选择用这样的方法?
- 花旗软件 电面问题(一) 根据对象的某一属性排序一组对象(JAVA实现)
- 快速排序,一种实现
- collections 接口存储一组不唯一,无序的对象。提供了对集合进行排序,遍历等对种算法的实现
- Spring Data JPA(2)--利用PagingAndSortingRespository实现分页和排序
- 一种支持html中table排序的跨平台实现
- hibernate实现单个对象的增删改查,以及实现分页查询
- Vue.js bootstrap前端实现分页和排序
- GridView 实现自定义分页、排序、查询、添加、编辑、多选删除
- 利用JQuery方便实现基于Ajax的数据查询、排序和分页功能
- (新人上路)成绩单实现对象三种排序方法 及归档解归档方法
- [ECSide文档] ECSide基于数据库的分页、排序、过滤的实现