您的位置：首页 > 编程语言 > Python开发

python MySQLdb 问题 - 程序已的后台运行每90秒获取数据，却无法获取最新的值

2013-05-24 09:34 771 查看

这是我在stackoverflow.com 中发现的处理方法，通过 autocommit解决问题！
URL: http://stackoverflow.com/questions/5943418/chronic-stale-results-using-mysqldb-in-python

Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.

Chronic stale results using MySQLdb in Python

up vote6down votefavorite3

My Python program queries a set of tables in a MySQL DB, sleeps for 30 seconds, then queries them again, etc. The tables in question are continuously updated by a third-party, and (obviously) I would like to see the new results every 30 seconds.
Let's say my query looks like this:

"select * from A where A.key > %d"% maxValueOfKeyFromLastQuery

Regularly I will see that my program stops finding new results after one or two iterations, even though new rows are present in the tables. I know new rows are present in the tables because I can see them when I issue the identical query from interactive mysql (i.e. not from Python).
I found that the problem goes away in Python if I terminate my connection to the database after each query and then establish a new one for the next query.
I thought maybe this could be a server-side caching issue as discussed here: Explicit disable MySQL query cache in some parts of program
However:
When I check the interactive mysql shell, it says that caching is on. (So if this is a caching problem, how come the interactive shell doesn't suffer from it?)

If I explicitly execute

SET SESSION query_cache_type = OFF

from within my Python program, the problem still occurs.

Creating a new DB connection for each query is the only way I've been able to make the problem go away.
How can I get my queries from Python to see the new results that I know are there?
python mysql caching

share|improve this question

edited May 9 '11 at 23:00

ThiefMaster♦
103k16117198

asked May 9 '11 at 22:56

dg99
857

You should not use the

format operator in queries. Use

%s

for all placeholders (no matter what their type is) and pass a tuple/list with the values so they can be sanitized properly. – ThiefMaster♦May 9 '11 at 23:01

Just a guess: Make sure that your queries that read from the db are not isolated from other changes by a transaction. I.e. create a new transaction before each read. – jsw May 9 '11 at 23:14

@ThiefMaster:

%d

is actually okay in queries, because it forces the substitution to be an integer, which cannot be used for SQL injection attacks. – Jonathan May 10 '11 at 14:37

3 Answers

active oldest votes

up vote6down voteaccepted

This website and this website contain information on the same problem. In order to keep your tables up to date, you must commit your transactions. Use

db.commit()

to do this.
As mentioned by the post below me, you can remove the need for this by enabling auto-commit. this can be done by running

db.autocommit(True)

Also, auto-commit is enabled in the interactive shell, so this explains why you didn't have the problem there.

share|improve this answer

edited May 10 '11 at 20:34

answered May 9 '11 at 23:10

Zonedabone
38029

Thanks for the tip! This was exactly the problem. I'm too used to Oracle, I suppose. :) – dg99 May 10 '11 at 14:28

I also benefited from this. But could you care to explain why autocommit helps SELECT statements? My source (in dev) is PHPMyAdmin which always returns the updated results. But my Python program did not, until enabling the autocommit. – aitchnyu Oct 12 '11 at 5:25

I have the same issue as @aitchnyu, my python process only does

select

s. I have a completely other process doing the writing, I put the

commit()

right after the

execute()

and before the

fetchall()

and it works. This is cool, but why do you do

commit()

on read statements? – Landon Nov 16 '12 at 1:03

up vote1down vote

You can enable auto-commit automatically in MySQLdb! Try the following:

conn = MySQLdb.Connect("host","user","password")conn.autocommit(True)

This gives you the same behavior that you are used to in the interactive shell.

share|improve this answer

answered May 10 '11 at 14:33

Jonathan
927

up vote-1down vote

You may want to check the transaction isolation level of your database. The behavior you describe is what you may expect if it is set to REPEATABLE-READ. You may want to change it to READ-COMMITTED.
Since the original poster of the problem mentions that he is merely querying the database, it cannot be a commit that was forgotten. Inserting a commit seems to be a workaround though since it causes a new transaction to begin; and a new snapshot may need to be established. Still, having to insert a commit before every select doesn't sound like good programming practices to me.
There is no python code to show since the solution lies in correctly configuring the database.
DO check MySQL documentation at http://dev.mysql.com/doc/refman/5.5/en/set-transaction.html.
REPEATABLE READ
This is the default isolation level for InnoDB. For consistent reads, there is an important difference from the READ COMMITTED isolation level: All consistent reads within the same transaction read the snapshot established by the first read. This convention means that if you issue several plain (nonlocking) SELECT statements within the same transaction, these SELECT statements are consistent also with respect to each other. See Section 14.3.9.2, “Consistent Nonlocking Reads”.

READ COMMITTED
A somewhat Oracle-like isolation level with respect to consistent (nonlocking) reads: Each consistent read, even within the same transaction, sets and reads its own fresh snapshot. See Section 14.3.9.2, “Consistent Nonlocking Reads”.

Checking the configured isolation level:

>mysql >SELECT@@GLOBAL.tx_isolation,@@tx_isolation;+-----------------------+-----------------+|@@GLOBAL.tx_isolation |@@tx_isolation  |+-----------------------+-----------------+| REPEATABLE-READ       | REPEATABLE-READ |+-----------------------+-----------------+1rowinset(0.01 sec)

Setting the transaction isolation level to READ-COMMITTED

mysql>SET GLOBAL tx_isolation='READ-COMMITTED';Query OK,0rows affected (0.00 sec)mysql>SET SESSION tx_isolation='READ-COMMITTED';Query OK,0rows affected (0.00 sec)mysql>SELECT@@GLOBAL.tx_isolation,@@tx_isolation;+-----------------------+----------------+|@@GLOBAL.tx_isolation |@@tx_isolation |+-----------------------+----------------+| READ-COMMITTED        | READ-COMMITTED |+-----------------------+----------------+1rowinset(0.01 sec)mysql>

And run the application again …

This website and this website contain information on the same problem. In order to keep your tables up to date, you must commit your transactions. Use

db.commit()

to do this.
As mentioned by the post below me, you can remove the need for this by enabling auto-commit. this can be done by running

db.autocommit(True)

Also, auto-commit is enabled in the interactive shell, so this explains why you didn't have the problem there.

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签：

相关文章推荐

新的分享

章节导航