Fetching verly large result set in mySql
Posted by: Pranay Choudhary
Date: February 27, 2011 10:34PM

Hi, I'm using the streaming technique to fetch very large result set from mysql using Java JDBC connection.
objStatement = con.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
objStatement.setFetchSize(Integer.MIN_VALUE);

I'm able to get the results successfully. My query returns 6*(10^10)+ rows and takes about and hour to fetch all results. My requirement is to cancel the query and clean close the db connection if the query execution takes more than 1 minute. I tried using the following approaches:
1. Statement.setQueryTimeOut(60)
2. connection property netTimeoutForStreamingResults=60
but neither of them works. Mysql document claims that it supports both of these methods.

Here is the code that i used:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;


public class MySqlConnector {
	private static final int nTimeOut = 60;
	
	private static Connection getConnection() throws SQLException, ClassNotFoundException {
		System.out.println("Getting Connection...");
		String driver = "com.mysql.jdbc.Driver";
		Class.forName(driver);
		String dbURL = "jdbc:mysql://localhost:3306/mydatabase";
		String dbUsername = "root";
		String dbPassword = "password";
		
		Properties connProperties = new Properties();
		connProperties.put("user", dbUsername);
		connProperties.put("password", dbPassword);
		connProperties.put("netTimeoutForStreamingResults", String.valueOf(nTimeOut));

		Connection conn = DriverManager.getConnection(dbURL, connProperties);
		System.out.println("Connected!");
		return conn;
	}
	
	public static void main(String[] args) throws SQLException, ClassNotFoundException {
		
		
		Connection con = getConnection();
		
		String strQuery = "SELECT vwdynmiscsecprices.compname as vwdynmiscsecprices_compname, vwdynmiscsecprices.ticker as vwdynmiscsecprices_ticker, vwdynmiscsecprices.exchange as vwdynmiscsecprices_exchange, vwdynmiscsecprices.currency as vwdynmiscsecprices_currency, vwdynmiscsecprices.pricedate as vwdynmiscsecprices_pricedate, vwdynmiscsecprices.price as vwdynmiscsecprices_price, vwdynmiscfxrates.currency as vwdynmiscfxrates_currency, vwdynmiscfxrates.fxdate as vwdynmiscfxrates_fxdate, vwdynmiscfxrates.fxrate as vwdynmiscfxrates_fxrate FROM  vwdynmiscsecprices   JOIN  vwdynmiscfxrates";
		
		Statement objStatement=null;
		ResultSet objResultSet=null;
		try {
			objStatement = con.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
			objStatement.setFetchSize(Integer.MIN_VALUE);
			objStatement.setQueryTimeout(nTimeOut);
			
			objResultSet = objStatement.executeQuery(strQuery);
			long count=0;
			while(objResultSet.next()) {
				count++;
				if(count%1000000==0) {
					System.out.println(count);
				}
			}
			System.out.println(count);
		} catch (SQLException e) {
			e.printStackTrace();
		} finally {
			try {
				objResultSet.close();
				objStatement.cancel();
				objStatement.close();
			} catch (SQLException e) {
				e.printStackTrace();
			} finally {
				con.close();
			}
		}
	}
}

Does mysql provides these facility? I'm using MySQL Community Server 5.1.51 and Connector/J version 5.1.15.
If yes, why doesn't it work? Do i need to change some configuration in mysql server?

Thanks in advance.

Options: ReplyQuote


Subject
Written By
Posted
Fetching verly large result set in mySql
February 27, 2011 10:34PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.