to improve sqlite performance
2015-11-06 10:19
323 查看
INSERT is really slow - I can only do few dozen INSERTs per second
http://www.sqlite.org/faq.html#q19
threadsafe / sqlite in multithread
https://www.sqlite.org/threadsafe.html
I read everywhere that creating transactions is the solution to slow SQLite writes, but it can be long and painful to rewrite your code and wrap all your SQLite writes in transactions.
I found a much simpler, safe and very efficient method: I enable a (disabled by default) SQLite 3.7.0 optimisation : the Write-Ahead-Log (WAL). The documentation says it works in all unix (i.e. Linux and OSX) and Windows systems.
How ? Just run the following commands after initializing your SQLite connection:
My code now runs ~600% faster : my test suite now runs in 38 seconds instead of 4 minutes :)
http://www.sqlite.org/faq.html#q19
Actually, SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. Transaction speed is limited by the rotational speed of your disk drive. A transaction normally requires two complete rotations of the disk platter, which on a 7200RPM disk drive limits you to about 60 transactions per second.
Transaction speed is limited by disk drive speed because (by default) SQLite actually waits until the data really is safely stored on the disk surface before the transaction is complete. That way, if you suddenly lose power or if your OS crashes, your data is still safe. For details, read about atomic commit in SQLite..
By default, each INSERT statement is its own transaction. But if you surround multiple INSERT statements with BEGIN...COMMIT then all the inserts are grouped into a single transaction. The time needed to commit the transaction is amortized over all the enclosed insert statements and so the time per insert statement is greatly reduced.
Another option is to run PRAGMA synchronous=OFF. This command will cause SQLite to not wait on data to reach the disk surface, which will make write operations appear to be much faster. But if you lose power in the middle of a transaction, your database file might go corrupt.
http://blogs.msdn.com/b/andy_wigley/archive/2013/11/21/how-to-massively-improve-sqlite-performance-using-sqlwinrt.aspx http://blog.quibb.org/2010/08/fast-bulk-inserts-into-sqlite/ http://www.codeproject.com/Articles/853842/Csharp-Avoiding-Performance-Issues-with-Inserts-in http://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite Transaction speed is limited by disk drive speed because (by default) SQLite actually waits until the data really is safely stored on the disk surface before the transaction is complete. That way, if you suddenly lose power or if your OS crashes, your data is still safe. For details, read about atomic commit in SQLite..
By default, each INSERT statement is its own transaction. But if you surround multiple INSERT statements with BEGIN...COMMIT then all the inserts are grouped into a single transaction. The time needed to commit the transaction is amortized over all the enclosed insert statements and so the time per insert statement is greatly reduced.
Another option is to run PRAGMA synchronous=OFF. This command will cause SQLite to not wait on data to reach the disk surface, which will make write operations appear to be much faster. But if you lose power in the middle of a transaction, your database file might go corrupt.
threadsafe / sqlite in multithread
https://www.sqlite.org/threadsafe.html
I read everywhere that creating transactions is the solution to slow SQLite writes, but it can be long and painful to rewrite your code and wrap all your SQLite writes in transactions.
I found a much simpler, safe and very efficient method: I enable a (disabled by default) SQLite 3.7.0 optimisation : the Write-Ahead-Log (WAL). The documentation says it works in all unix (i.e. Linux and OSX) and Windows systems.
How ? Just run the following commands after initializing your SQLite connection:
PRAGMA journal_mode = WAL PRAGMA synchronous = NORMAL
My code now runs ~600% faster : my test suite now runs in 38 seconds instead of 4 minutes :)
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- MySQL:建表及字段类型选取
- 真的是数据库连接池C3P0的错?
- Redis缓存服务器的配置
- MySQL:存储引擎
- 数据库分离
- Mysql 性能优化
- MYSQL用法(十六) MySQL按指定字段自定义列表排序
- sql语句，查询昨天的数据
- MySQL:触发器
- sql 中常用日期转换Convert(Datetime)
- SQLserver行转列，列转行
- sql语句备份
- SQL UNION 和 UNION ALL 操作符
- 内存数据库Redis小Demo 包括持久性测试
- 数据库热备
- PL/SQL Developer远程连接Oracle数据库
- 数据库练习
- mysql将timestamp类型转换成date类型数据
- RHCE 系列（五）：如何在 RHEL 7 中管理系统日志（配置、轮换以及导入到数据库）
- MYSQL 注射精华