How to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer
2016-06-17 08:09
656 查看
How to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer 1 About DB Query AnalyzerDB Query Analyzer is presented by Master Genfeng, Ma from Chinese Mainland. It has English version named ‘DB Query Analyzer’ and Simplified Chinese version named
. DB Query Analyzer is one of the few excellent Client Tools in the world for its’ powerful function, friendly interface, easy operation and applicability to every production of RDBMS. ① It lets you query ODBC data sources, author SQL scripts and queries, return query results to a grid or free-form text or a file, retrieve ODBC driver information, ②execute multiple SQL scripts or stored procedures simultaneously, ③save the result to files with high efficiency which is as quick as tools provided by database system. ④ What is more, from 6.01,DB Query Analyzer provides SQL Execute Schedule function to execute SQL scripts in certain time. If you select the option ‘SQL Schedule Reconnect’ in configurations window, the Schedule SQL scripts will be executed as long as the Database Server is in use, even though the Database Server had been down. ⑤From version 6.03, without changing any Windows OS settings or configurations, DB Query Analyzer can run on any Microsoft Windows OS directly such as Windows 10,Windows 8, Windows 7, Windows Vista, Windows 2003, Windows XP, Windows 2000,Windows NT, Windows ME, Windows 9X. In the New products & Tools reviews of programmer second issue of 2007, DB Query Analyzer had been strongly recommended. Now the Simplified Chinese version of DB Query Analyzer is the top 50 database application software in the famous software website http://xiazai.zol.com.cn/download_order/sub_550.html .In most case it lies the top 10 and it has been downloaded more than 110,000 times. 85 technical articles about DB Query Analyzer have been published in computer journal, BAIDU Library, the CSDN resource or my three blog-websites. 2 Contents reported regularly
First in Chinese there is a time unit, which is defined as a period of ten days in a month. One month has three sections. The first period of ten days is from the 1st dayto the 10th day, and the second period of ten days means the 11stday to the 20th day. In addition, the 21st day to the last day is the third period of a month.If you have experience in big data processing, you will know the complex of getting all the data required in the table.In fact I am familiar with big data processing. Every month 1.2 billion records will occur in the database according to the vehicles that pass by the high way in Guangdong province. So I told an officer from Guangdong Transport Department in a transport conference held in Beijing last month that I would design more efficient algorithm to generate the data required in the table every period of ten days.As Guangdong is adeveloped province in China, the income from Guangdong of a series of state-owned company always occupies one sixth of the whole state industry income.If there are only a few records in a table, you will not meet with any difficulties in data processing whatever algorithm you choose. Considering the big data to generate the result in the table above, I have no choice but to design more efficient algorithm. 3 Algorithm design by myself and steps of big data processing Considering the complex of the data required by Ministry of Transport of P.R.C every period of tendays, I design the following algorithm to generate the data required.To begin with, get theinter mediate results by the five group SQL scripts in schedule with DB Query Analyzer. In addition, the final data will generate with DB Query Analyzer by accessing the six text files also in schedule.
Figure 1 Five section to get the intermiediate results 3.1 How to get the newly added toll station in this month.The definition of newly added toll station is that occurs from the 1st day to the end of this period of ten days this month compare with the same month in last year. As aresult, the maximum record number is all the record occurred in this month when processing the third period of ten days. Traffic volume in Guangdong is about 1.2 billion a month. So in this case, I have to process 1.2 billion records this month and 1.2 billion records the same month last year to get the set of newly added toll station.
Figure 2 Get the set of newly added toll station in schedule by DB Query Analyzer From 6.01, DB Query Analyzer provides SQL Execute Schedule function to execute SQL scripts in certain time. If you select the option ‘SQL Schedule Reconnect’ in configurations window, the Schedule SQL scripts will be executed as long as the Database Server is in use, even though the Database Server had been down. 3.2 How to get a series of traffic data of trunk this period of ten days
Figure 3 Get a series of traffic data of trunk this period of ten days in schedule by DB Query Analyzer Similarly, other traffic data of trunk can be gotten with corresponding SQL scripts in schedule by DB Query Analyzer as well. 3.3 All the intermediate result generated The sixth text files are the intermediate result and five files are created by DB Query Analyzer inschedule. At last, the intermediate result is used to generate the final result.
Figure 4 The sixth text files , the intermediate result 3.4 How to configure the ODBC data source to the directory in Figure 4 DB Query Analyzer can access text/csv files with SQL script by ODBC data source. So if you can configure a ODBC data source firstly, then you can copy some text/csv files thatyou want to access to the directory. Consequently, you can access those files with SQL scripts by DB Query Analyzer.
Figure 5 Configure the ODBC data source to the directory
Figure 6 Configure the ODBC datasource to the directory 3.5 Access six text files to generate all the data required
Figure 7 Access 6 text files in schedule to generate all the data
Figure 8 Open the result file with EXCEL and you can paste them to the table 4 The advantage of my algorithm designFirstly, you have to choose the strategy to disassemble the complex task into a few easier sub tasks when facing big data, which prevent you from locking tables for quite long time.Secondly, you need not copy intermediate data to other machine and create intermediate tables in database which you have to fill in quite a few tables for approval process. Also you need not to insert data into tables you will create.Thirdly, it is very convenient to test SQL scripts, find logic errors in your SQL scripts which may result in error data, since you can open text/csv files with NotePad, Wordpad as well as DB Query Analyzer using SQL scripts. 5 SummaryToday I just demonstrate that DB Query Analyzer is very good tool in big data processing.What’s more, 85 technical articles about DB Query Analyzer which have been published in computer journal, BAIDU Library, the CSDN resource or my there blog-websites. At present, DB QueryAnalyzer is widely used in our company and our colleagues are fond of it toaccess DB2 database.DB Query Analyzer 6.04 download URL: http://xiazai.zol.com.cn/detail/43/420901.shtmlhttp://www.unitedpowersoft.com/UpFile/DBQueryAnalyzer_English_604.rar DB Query Analyzer Simplified Chinese version 6.04 download URL:http://xiazai.zol.com.cn/detail/27/264455.shtml My blogs are http://blog.csdn.net/magenfenghttp://blog.sina.com.cn/magenfeng http://user.qzone.qq.com/630414817 Interview with CSDN: http://www.csdn.net/article/2014-08-09/2821124
85 technical articles about DB Query Analyzer have been published in computer journal, Baidu Library, the CSDN resource and my three blog-websites. The following 7 articles have been published in computer journals.Guangdong Highway settlement business analysis system based on Star Schema in Data Warehousing according to national standardization in Computer Era 7th issue of 2015 in Hangzhou.DB Query Analyzer to cancel a running SQL statement in Computer Era 12th issue of 2011 in Hangzhou.DB Query Analyzer Returns Information in More Detail While Batch SQL Statement End of Execution in Computer Programming Skill & Maintenance 24th issue of 2011 in BeijingThe Application of the Transactions Control in DB2 with DB Query Analyzer in Computer Programming Skill & Maintenance 22nd issue of 2011 in Beijing.How does DB Query Analyzer cancel the SQL statement committed to DBMS in Computer Engineering & Software 6th issue of 2011 in Tianjin.The Application of the Transactions Control in Oracle with DB Query Analyzer in Microcomputer Applications 11th issue of 2011 in Shanghai. DB Query Analyzer had been strongly recommended In the New products & Tools reviews of programmer 2nd issue of 2007. The following 78 articles have been published in Baidu Library, the CSDN resource and my three blog-websites.The DBMS that DB Query Analyzer Users often use cover all kinds of DBMS have been published.How to configure ODBC DSN in Client to access remote DB2 for WindowsHow to configure ODBC DSN to access local DB2 for WindowsWhich SQL statement is the trump card to the senior software developerExepacker prevent DB Query Analyzer from being debugged have been published.16 articles from The 1st tip of DB Query Analyze to The 16th tips of DB QueryAnalyzer have been published.The table name must be enclosed in double quotation marks or square bracket while accessing EXCEL by DB Query AnalyzerHow to download the installation package by ZOL DownerDB Query Analyzer, the most excellent Universal database Access tools on any Microsoft Windows OSDemonstration of DB Query Analyzer 6.03 Installation and Running on Microsoft Windows 8How to easily create DB2 user function by DB Query Analyzer 6.03Install and run DB Query Analyzer 6.04 on Microsoft Windows 10A universal algorithm design of locating fixed length stringHow to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer
. DB Query Analyzer is one of the few excellent Client Tools in the world for its’ powerful function, friendly interface, easy operation and applicability to every production of RDBMS. ① It lets you query ODBC data sources, author SQL scripts and queries, return query results to a grid or free-form text or a file, retrieve ODBC driver information, ②execute multiple SQL scripts or stored procedures simultaneously, ③save the result to files with high efficiency which is as quick as tools provided by database system. ④ What is more, from 6.01,DB Query Analyzer provides SQL Execute Schedule function to execute SQL scripts in certain time. If you select the option ‘SQL Schedule Reconnect’ in configurations window, the Schedule SQL scripts will be executed as long as the Database Server is in use, even though the Database Server had been down. ⑤From version 6.03, without changing any Windows OS settings or configurations, DB Query Analyzer can run on any Microsoft Windows OS directly such as Windows 10,Windows 8, Windows 7, Windows Vista, Windows 2003, Windows XP, Windows 2000,Windows NT, Windows ME, Windows 9X. In the New products & Tools reviews of programmer second issue of 2007, DB Query Analyzer had been strongly recommended. Now the Simplified Chinese version of DB Query Analyzer is the top 50 database application software in the famous software website http://xiazai.zol.com.cn/download_order/sub_550.html .In most case it lies the top 10 and it has been downloaded more than 110,000 times. 85 technical articles about DB Query Analyzer have been published in computer journal, BAIDU Library, the CSDN resource or my three blog-websites. 2 Contents reported regularly
First in Chinese there is a time unit, which is defined as a period of ten days in a month. One month has three sections. The first period of ten days is from the 1st dayto the 10th day, and the second period of ten days means the 11stday to the 20th day. In addition, the 21st day to the last day is the third period of a month.If you have experience in big data processing, you will know the complex of getting all the data required in the table.In fact I am familiar with big data processing. Every month 1.2 billion records will occur in the database according to the vehicles that pass by the high way in Guangdong province. So I told an officer from Guangdong Transport Department in a transport conference held in Beijing last month that I would design more efficient algorithm to generate the data required in the table every period of ten days.As Guangdong is adeveloped province in China, the income from Guangdong of a series of state-owned company always occupies one sixth of the whole state industry income.If there are only a few records in a table, you will not meet with any difficulties in data processing whatever algorithm you choose. Considering the big data to generate the result in the table above, I have no choice but to design more efficient algorithm. 3 Algorithm design by myself and steps of big data processing Considering the complex of the data required by Ministry of Transport of P.R.C every period of tendays, I design the following algorithm to generate the data required.To begin with, get theinter mediate results by the five group SQL scripts in schedule with DB Query Analyzer. In addition, the final data will generate with DB Query Analyzer by accessing the six text files also in schedule.
Figure 1 Five section to get the intermiediate results 3.1 How to get the newly added toll station in this month.The definition of newly added toll station is that occurs from the 1st day to the end of this period of ten days this month compare with the same month in last year. As aresult, the maximum record number is all the record occurred in this month when processing the third period of ten days. Traffic volume in Guangdong is about 1.2 billion a month. So in this case, I have to process 1.2 billion records this month and 1.2 billion records the same month last year to get the set of newly added toll station.
Figure 2 Get the set of newly added toll station in schedule by DB Query Analyzer From 6.01, DB Query Analyzer provides SQL Execute Schedule function to execute SQL scripts in certain time. If you select the option ‘SQL Schedule Reconnect’ in configurations window, the Schedule SQL scripts will be executed as long as the Database Server is in use, even though the Database Server had been down. 3.2 How to get a series of traffic data of trunk this period of ten days
Figure 3 Get a series of traffic data of trunk this period of ten days in schedule by DB Query Analyzer Similarly, other traffic data of trunk can be gotten with corresponding SQL scripts in schedule by DB Query Analyzer as well. 3.3 All the intermediate result generated The sixth text files are the intermediate result and five files are created by DB Query Analyzer inschedule. At last, the intermediate result is used to generate the final result.
Figure 4 The sixth text files , the intermediate result 3.4 How to configure the ODBC data source to the directory in Figure 4 DB Query Analyzer can access text/csv files with SQL script by ODBC data source. So if you can configure a ODBC data source firstly, then you can copy some text/csv files thatyou want to access to the directory. Consequently, you can access those files with SQL scripts by DB Query Analyzer.
Figure 5 Configure the ODBC data source to the directory
Figure 6 Configure the ODBC datasource to the directory 3.5 Access six text files to generate all the data required
Figure 7 Access 6 text files in schedule to generate all the data
Figure 8 Open the result file with EXCEL and you can paste them to the table 4 The advantage of my algorithm designFirstly, you have to choose the strategy to disassemble the complex task into a few easier sub tasks when facing big data, which prevent you from locking tables for quite long time.Secondly, you need not copy intermediate data to other machine and create intermediate tables in database which you have to fill in quite a few tables for approval process. Also you need not to insert data into tables you will create.Thirdly, it is very convenient to test SQL scripts, find logic errors in your SQL scripts which may result in error data, since you can open text/csv files with NotePad, Wordpad as well as DB Query Analyzer using SQL scripts. 5 SummaryToday I just demonstrate that DB Query Analyzer is very good tool in big data processing.What’s more, 85 technical articles about DB Query Analyzer which have been published in computer journal, BAIDU Library, the CSDN resource or my there blog-websites. At present, DB QueryAnalyzer is widely used in our company and our colleagues are fond of it toaccess DB2 database.DB Query Analyzer 6.04 download URL: http://xiazai.zol.com.cn/detail/43/420901.shtmlhttp://www.unitedpowersoft.com/UpFile/DBQueryAnalyzer_English_604.rar DB Query Analyzer Simplified Chinese version 6.04 download URL:http://xiazai.zol.com.cn/detail/27/264455.shtml My blogs are http://blog.csdn.net/magenfenghttp://blog.sina.com.cn/magenfeng http://user.qzone.qq.com/630414817 Interview with CSDN: http://www.csdn.net/article/2014-08-09/2821124
85 technical articles about DB Query Analyzer have been published in computer journal, Baidu Library, the CSDN resource and my three blog-websites. The following 7 articles have been published in computer journals.Guangdong Highway settlement business analysis system based on Star Schema in Data Warehousing according to national standardization in Computer Era 7th issue of 2015 in Hangzhou.DB Query Analyzer to cancel a running SQL statement in Computer Era 12th issue of 2011 in Hangzhou.DB Query Analyzer Returns Information in More Detail While Batch SQL Statement End of Execution in Computer Programming Skill & Maintenance 24th issue of 2011 in BeijingThe Application of the Transactions Control in DB2 with DB Query Analyzer in Computer Programming Skill & Maintenance 22nd issue of 2011 in Beijing.How does DB Query Analyzer cancel the SQL statement committed to DBMS in Computer Engineering & Software 6th issue of 2011 in Tianjin.The Application of the Transactions Control in Oracle with DB Query Analyzer in Microcomputer Applications 11th issue of 2011 in Shanghai. DB Query Analyzer had been strongly recommended In the New products & Tools reviews of programmer 2nd issue of 2007. The following 78 articles have been published in Baidu Library, the CSDN resource and my three blog-websites.The DBMS that DB Query Analyzer Users often use cover all kinds of DBMS have been published.How to configure ODBC DSN in Client to access remote DB2 for WindowsHow to configure ODBC DSN to access local DB2 for WindowsWhich SQL statement is the trump card to the senior software developerExepacker prevent DB Query Analyzer from being debugged have been published.16 articles from The 1st tip of DB Query Analyze to The 16th tips of DB QueryAnalyzer have been published.The table name must be enclosed in double quotation marks or square bracket while accessing EXCEL by DB Query AnalyzerHow to download the installation package by ZOL DownerDB Query Analyzer, the most excellent Universal database Access tools on any Microsoft Windows OSDemonstration of DB Query Analyzer 6.03 Installation and Running on Microsoft Windows 8How to easily create DB2 user function by DB Query Analyzer 6.03Install and run DB Query Analyzer 6.04 on Microsoft Windows 10A universal algorithm design of locating fixed length stringHow to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer
相关文章推荐
- Java工具类:(2)生成UUID
- TaableView之自定义Cell
- intptr_t, uintptr_t
- 用实战为你讲解UIKit性能调优
- hdu 5538 House Building 2015ACM/ICPC亚洲区长春站-重现赛
- iOS UiICollectionView中cell间的边距的设置
- win32gui.CeateWindow的参数
- win32gui中操作任务栏托盘区的函数
- win32gui win32gui.CreatePopupMenu int = CreatePop
- 自定义UINavigationController push和pop动画
- Mysql操作数据库异常Ignoring query to other database
- 排序算法-10-算法-分治法(Divide and Conquer)
- 关于用UITabelView发现的问题以及解决
- Abobe Flash cs6 和 Abobe Flash Builder4.6 破解和许可证过期
- UIImage知识点汇总(二)
- UIView 的常用方法及注意事项
- 详细讲解UIKit性能调优(续)
- UICollectionView
- UITableView的使用(UITableViewCell\section页眉\section页脚复用、sectionHeaderView点击效果)
- IOS 开发笔记-基础 UI(8)控件连线的 strong 和 weak 设定