Android 上的数据格式 FlatBuffers
2016-07-19 15:03
417 查看
原文:http://gold.xitu.io/entry/55dd1e3b60b27e6cd500d266
迪鲁宾 分享:Android 工程师 @ 阿里巴巴
FlatBuffers 是一个由谷歌提出的跨平台的数据格式,最大的好处在于不需要反序列化,直接读出数据,减少了decode的计算和内存使用。专门为游戏应用设计,目前已经被 Facebook 使用在 Facebook 应用中。
原文链接
本文已经翻译成中文《 Android 上的序列化库 FlatBuffers 介绍》,欢迎参加「掘金翻译计划」,翻译优质的技术文章。
JSON - probably everyone knows this lightweight data format used in almost all modern servers. It weights less, is more human-readable and in general is more dev-friendly than old-fashined, horrible xml. JSON is language-independend data format but parsing
data and transforming it to e.g. Java objects costs us time and memory resources.
Several days ago Facebook announced big performance improvement in data handling in its Android app. It was connected with dropping JSON format and replacing it with FlatBuffers in almost entire app. Please check
this articleto get some basic knowledge about FlatBuffers and results of transition to it from JSON.
While the results are very promising, at the first glance the implementation isn’t too obvious. Also facebook didn’t say too much. That’s why in this post I’d like to show how we can start our work with FlatBuffers.
16ms rule of smooth and responsive UI in Android.
But hey, before you throw everything to migrate all your data to FlatBuffers, just make sure that you need this. Sometimes the impact on performance will be imperceptible and sometimes
data safety will be more important than a tens of milliseconds difference in computation speed.
What makes FlatBuffers so effective?
Serialized data is accessed without parsing because of flat binary buffer, even for hierarchical data. Thanks to this we don’t need to initialize parsers (what means to build complicated field mappings) and parse this data, which also takes time.
FlatBuffers data doesn’t need to allocate more memory than it’s used by buffer itself. We don’t need to allocate extra objects for whole hierarchy of parsed data like it’s done in JSON.
For the real numbers just check again
facebook article about migrating to FlatBuffers or
Google documentation itself.
JSON data is converted to FlatBuffer format somewhere outside the app (e.g. bin ary file is delivered as a file or returned directly from API)
Data model (Java classes) is generated by hand, with flatc (FlatBuffer compiler)
There are some limitations for JSON file (null fields cannot be used, Date format is parsed as a String)
Probably in the future we’ll prepare more complex solution.
flatbuffers repository. Let’s download/clone it. Whole build process is described on
FlatBuffers Building documentation. If you are Mac user all you have to do is:
Open downloaded source code on
Run flatc scheme (should be selected by default) by pressing
Play button or
flatc executable will appear in project root directory.
Now we’re able to
use schema compiler which among the others can generate model classes for given schema (in Java, C#, Python, GO and C++) or convert JSON to FlatBuffer binary file.
Here is a part of our JSON file:
Full version is available
here. It’s a bit modified version of data which can be taken from Github API call:
https://api.github.com/users/google/repos.
Writing a FlatBuffer schema is very well
documented, so I won’t delve into this. Also in our case schema won’t be very complicated. All we have to do is to create 3 tables:
Full schema file is
available here.
available in our repository):
If everything goes well, here is a list of generated files:
repos_json.bin (we’ll rename it to repos_flat.bin)
Repos/Repo.java
Repos/ReposList.java
Repos/User.java
ProgressBar will be used only to show how incorrect data handling (in UI thread) can affect smoothness of user interface.
Of course it’s not necessary to use Rx or ButterKnife in our example, but why not to make this app a bit nicer
迪鲁宾 分享:Android 工程师 @ 阿里巴巴
FlatBuffers 是一个由谷歌提出的跨平台的数据格式,最大的好处在于不需要反序列化,直接读出数据,减少了decode的计算和内存使用。专门为游戏应用设计,目前已经被 Facebook 使用在 Facebook 应用中。
原文链接
本文已经翻译成中文《 Android 上的序列化库 FlatBuffers 介绍》,欢迎参加「掘金翻译计划」,翻译优质的技术文章。
JSON - probably everyone knows this lightweight data format used in almost all modern servers. It weights less, is more human-readable and in general is more dev-friendly than old-fashined, horrible xml. JSON is language-independend data format but parsing
data and transforming it to e.g. Java objects costs us time and memory resources.
Several days ago Facebook announced big performance improvement in data handling in its Android app. It was connected with dropping JSON format and replacing it with FlatBuffers in almost entire app. Please check
this articleto get some basic knowledge about FlatBuffers and results of transition to it from JSON.
While the results are very promising, at the first glance the implementation isn’t too obvious. Also facebook didn’t say too much. That’s why in this post I’d like to show how we can start our work with FlatBuffers.
FlatBuffers
In short, FlatBuffers is a cross-platform serialization library from Google, created specifically for game development and, as Facebook showed, to follow the16ms rule of smooth and responsive UI in Android.
But hey, before you throw everything to migrate all your data to FlatBuffers, just make sure that you need this. Sometimes the impact on performance will be imperceptible and sometimes
data safety will be more important than a tens of milliseconds difference in computation speed.
What makes FlatBuffers so effective?
Serialized data is accessed without parsing because of flat binary buffer, even for hierarchical data. Thanks to this we don’t need to initialize parsers (what means to build complicated field mappings) and parse this data, which also takes time.
FlatBuffers data doesn’t need to allocate more memory than it’s used by buffer itself. We don’t need to allocate extra objects for whole hierarchy of parsed data like it’s done in JSON.
For the real numbers just check again
facebook article about migrating to FlatBuffers or
Google documentation itself.
Implementation
This article will cover the simplest way of using FlatBuffers in Android app:JSON data is converted to FlatBuffer format somewhere outside the app (e.g. bin ary file is delivered as a file or returned directly from API)
Data model (Java classes) is generated by hand, with flatc (FlatBuffer compiler)
There are some limitations for JSON file (null fields cannot be used, Date format is parsed as a String)
Probably in the future we’ll prepare more complex solution.
FlatBuffers compiler
At the beginning we have to get flatc - FlatBuffers compiler. It can be built from source code hosted in Google’sflatbuffers repository. Let’s download/clone it. Whole build process is described on
FlatBuffers Building documentation. If you are Mac user all you have to do is:
Open downloaded source code on
\{extract directory}\build\XcodeFlatBuffers.xcodeproj
Run flatc scheme (should be selected by default) by pressing
Play button or
⌘ + R
flatc executable will appear in project root directory.
Now we’re able to
use schema compiler which among the others can generate model classes for given schema (in Java, C#, Python, GO and C++) or convert JSON to FlatBuffer binary file.
Schema file
Now we have to prepare schema file which defines data structures we want to de-/serialize. This schema will be used with flatc to create Java models and to transform JSON into Flatbuffer binary file.Here is a part of our JSON file:
{ "repos": [ { "id": 27149168, "name": "acai", "full_name": "google/acai", "owner": { "login": "google", "id": 1342004, ... "type": "Organization", "site_admin": false }, "private": false, "html_url": "https://github.com/google/acai", "description": "Testing library for JUnit4 and Guice.", ... "watchers": 21, "default_branch": "master" }, ... ] }
Full version is available
here. It’s a bit modified version of data which can be taken from Github API call:
https://api.github.com/users/google/repos.
Writing a FlatBuffer schema is very well
documented, so I won’t delve into this. Also in our case schema won’t be very complicated. All we have to do is to create 3 tables:
ReposList,
Repoand
User, and define
root_type. Here is the important part of this schema:
table ReposList {
repos : [Repo];
}
table Repo {
id : long;
name : string;
full_name : string;
owner : User;
//...
labels_url : string (deprecated);
releases_url : string (deprecated);
}
table User {
login : string;
id : long;
avatar_url : string;
gravatar_id : string;
//...
site_admin : bool;
}
root_type ReposList;
Full schema file is
available here.
FlatBuffers data file
Great, now all we have to do is to convertrepos_json.jsonto FlatBuffers binary file and generate Java models which will be able to represent our data in Java-friendly style (all files required in this operation are
available in our repository):
$ ./flatc -j -b repos_schema.fbs repos_json.json
If everything goes well, here is a list of generated files:
repos_json.bin (we’ll rename it to repos_flat.bin)
Repos/Repo.java
Repos/ReposList.java
Repos/User.java
Android app
Now let’s create our example app to check how FlatBuffers format works in practice. Here is the screenshot of it:ProgressBar will be used only to show how incorrect data handling (in UI thread) can affect smoothness of user interface.
app/build.gradlefile of our app will look like this:
apply plugin: 'com.android.application' apply plugin: 'com.jakewharton.hugo' android { compileSdkVersion 22 buildToolsVersion "23.0.0 rc2" defaultConfig { applicationId "frogermcs.io.flatbuffs" minSdkVersion 15 targetSdkVersion 22 versionCode 1 versionName "1.0" } buildTypes { release { minifyEnabled false proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' } } } dependencies { compile fileTree(dir: 'libs', include: ['*.jar']) compile 'com.android.support:appcompat-v7:22.2.1' compile 'com.google.code.gson:gson:2.3.1' compile 'com.jakewharton:butterknife:7.0.1' compile 'io.reactivex:rxjava:1.0.10' compile 'io.reactivex:rxandroid:1.0.0' }
Of course it’s not necessary to use Rx or ButterKnife in our example, but why not to make this app a bit nicer
相关文章推荐
- Android Volley框架的简单get和post demo
- 获取Android应用专属缓存存储目录
- Android 进程常驻(0)----MarsDaemon使用说明
- EditText详细介绍
- Android:如何从堆栈中还原ProGuard混淆后的代码
- Android自定义控件:进度条的四种实现方式
- Android平板上开发应用的一点心得——精确适配不同的dpi和屏幕尺寸
- 本地资源的图片文件 转换成bitmap
- Android开发经典书籍
- 安卓分享纯图片
- Android中Recyclerview使用6----添加条目得到点击事件和长按事件(另一种写法,较简单)
- Android ListView 隐藏自带分割线
- android--xUtils开发框架
- Android逆向分析案例——某地图的定位请求分析
- Android中定时器Timer和TimerTask的启动,停止,暂停,继续等操作实例
- GreenDao_2.介绍GreenDao
- 推荐几款实用的Android Studio 插件
- android simpledraweeview 获取已经加载的图片bitmap
- Android listview 点击问题
- 初识Android之(一)-自定义标题栏控件