MySQL Forums
Forum List  »  Perl

Huge memusage when operating on large hash/arrays in perl
Posted by: Apachez
Date: April 20, 2006 05:11PM

Perhaps a more Perl oriented question than MySQL oritented but since my script uses MySQL as backend I hope that someone in here might have runned into the same/similar problems as I have...

When I am loading searchword data from mysql into hash + arrays. The perl process then takes approx 61 meg.

When the "core functions" (pasted below) are commented out the total memusage is approx 140 meg (first stage is to load the searchword stuff into mem in perl, second stage is to fetch 100.000 rows from the db which will be indexed).

When they are not commented out the usage grows to 1.6 gig...

Commented (140 meg total memusage):

if(exists($searchword{$array}))
{
# $searchwordfreq[$searchword{$array}]++;
# $vector = Compress::LZO::decompress($searchvector[$searchword{$array}]);
# vec($vector, $sql_NewsPost, 1) = 1;
# $searchvector[$searchword{$array}] = Compress::LZO::compress($vector, 1);
# $searchupdated[$searchword{$array}] = 1;
}
else
{
# $searchwordid++;
# $vector = '';
# vec($vector, $sql_NewsPost, 1) = 1;
# $searchword{$array} = $searchwordcount;
# push(@searchwordid, $searchwordid);
# push(@searchwordfreq, 1);
# push(@searchvector, Compress::LZO::compress($vector, 1));
# push(@searchupdated, 1);
# $searchwordcount++;
}

Not commented:

if(exists($searchword{$array}))
{
$searchwordfreq[$searchword{$array}]++;
$vector = Compress::LZO::decompress($searchvector[$searchword{$array}]);
vec($vector, $sql_NewsPost, 1) = 1;
$searchvector[$searchword{$array}] = Compress::LZO::compress($vector, 1);
$searchupdated[$searchword{$array}] = 1;
}
else
{
$searchwordid++;
$vector = '';
vec($vector, $sql_NewsPost, 1) = 1;
$searchword{$array} = $searchwordcount;
push(@searchwordid, $searchwordid);
push(@searchwordfreq, 1);
push(@searchvector, Compress::LZO::compress($vector, 1));
push(@searchupdated, 1);
$searchwordcount++;
}

The mysql data on disk is about 22 meg (which when loaded in perl into hash + arrays becomes roughly 61 meg but that is due to how perl creates and stores hash and arrays).

The load function for the first stage is:

$main::sth2->bind_columns(undef, \$sql2_SearchWordId, \$sql2_SearchWordFreq, \$sql2_SearchWord, \$sql2_SearchVector);
while($main::sth2->fetchrow_arrayref())
{
$searchwordid = $sql2_SearchWordId;
$searchword{$sql2_SearchWord} = $searchwordcount;
push(@searchwordid, $sql2_SearchWordId);
push(@searchwordfreq, $sql2_SearchWordFreq);
push(@searchvector, $sql2_SearchVector);
push(@searchupdated, 0);
$searchwordcount++;
}

Which obviously works without problems, so why do the "core-functions" make the perl to start eating memory out of the blue ?

Options: ReplyQuote


Subject
Written By
Posted
Huge memusage when operating on large hash/arrays in perl
April 20, 2006 05:11PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.