mysql - Perl mailqueueing with threads 'out of memory' -



mysql - Perl mailqueueing with threads 'out of memory' -

i created mailqueue script checks mysql table mail service has got timestamp in past send. can there more mail service in 'mailqueue' send in 1 time used threads send mail service in batches. when there not much mail service send scheme running steady, other times mailqueue process running (through bash script calls every 30 seconds) getting killed because scheme running out of memory. prevent 'mailqueue' of running out of memory.

i inquire of take code underneath maybe doing wrong.

thanks in advance.

~$ htop pid user pri ni virt res shr s cpu% mem% time+ command 5675 root 20 0 969m 528m 3744 s 0.0 14.1 2:33.05 perl /path/mailqueue.pl server specs: 1 vcpu; 3.75 gib memory; sub start_threads { @threads; # records found $found = 0; $sth = $dbh->prepare("select mail_queue_id, project_id, type, name, email mail_queue timestamp < now() , active = 1 order timestamp asc limit 10"); $sth->execute(); while (my $ref = $sth->fetchrow_hashref()) { # set if records found $found = 1; # set email variables $id = $ref->{'mail_queue_id'}; $project_id = $ref->{'project_id'}; $type = $ref->{'type'}; $name = $ref->{'name'}; $email = $ref->{'email'}; # create array info @select_arr = ($id, $project_id, $type, $name, $email); # start thread send mail service $t = threads->new(\&sendmail, @select_arr); push(@threads,$t); } foreach (@threads) { # mail_queue_id $id = $_->join; print "set email $id in queue inactive\n"; # set mail_queue record inactive -> mysql(event) mailqueue cleanup every 10 minutes $sth = $dbh->prepare("update mail_queue set active = 0 mail_queue_id = ? "); $sth->execute($id); } if($found eq 1) { # homecoming rows in mail_queue < 1 sleep(10); &start_threads; } else { # skip thread, wait 1 minut = sleep(1) select new rows; sleep(30); &start_threads; } } # prepare send e-mail sub sendmail { @select_arr = @_; # queue variables $id = $select_arr[0]; $project_id = $select_arr[1]; $type = $select_arr[3]; $name = $select_arr[4]; $email = $select_arr[5]; print "started sending email " . $id . " \n"; # phone call function sends mail service out $send = &email(@select_arr); # if mail service sent if($send eq 1) { print "done sending email " . $id . "\n"; sleep (1); # homecoming unique id homecoming $id; } } &start_threads;

what you're doing potentially quite expensive - when perl threads creates re-create of process - including imported modules, info states, etc. if table returns lot of rows, eat memory quickly.

you can see doing ps -eft.

for you're doing, way you're doing bad idea. suggest 2 alternatives:

stick threads, start fixed number (say, 10) , utilize thread::queue serialise data. limits number of process copies, , thread startup overhead.

switch using fork(). parallel::forkmanager more or less want here. fork() more efficient way of process cloning - it'll re-create memory on demand, meaning sub processes much smaller.

i'll offer examples gave in previous answer: perl daemonize kid daemons

mysql multithreading bash perl email

Comments

Post a Comment

Popular posts from this blog

Delphi change the assembly code of a running process -

Delphi change the assembly code of a running process - i'm trying alter address 00741fa5 has push test.009e721c . alter push test.009e71c8 . procedure callback; asm force $9e71c8 end; procedure tform2.btn1click(sender: tobject); var ppid :dword; pprocess : cardinal; begin getwindowthreadprocessid(findwindow(nil,pchar('test')), @ppid); pprocess := openprocess(process_all_access, false, ppid); injectasm(pprocess, $741fa5, 10, integer(@callback), 0); end; here injectasm function (with error checking) function injectasm(process: longword; injectaddress, injectsize, codeaddress, codesize : integer): pointer; var csize, rsize : integer; replaced : array of byte; jmp : array [0..4] of byte; jmpaddress : integer; nopv : byte; : integer; nbr: ulong_ptr; begin //injectsize must equal or greater 5, because need space our //(far) jmp wwxxyyzz instruction //if there's no space, inject in place of few instructions if injectsize ...
Read more

json - Hibernate and Jackson (java.lang.IllegalStateException: Cannot call sendError() after the response has been committed) -

json - Hibernate and Jackson (java.lang.IllegalStateException: Cannot call sendError() after the response has been committed) - hello have problem hibernate , jackson. have 2 pojos @entity @table(name = "users") public class user implements serializable { static final long serialversionuid = 4235637833262722256l; @id @generatedvalue private int id; @column(name = "creation_date") private date creationdate; @column(name = "first_name") private string firstname; @column(name = "last_name") private string lastname; @version @column(name = "version") private int version; @column(name = "e_mail") private string email; @column(name = "enabled") private boolean enabled; @column(name = "login") private string login; @column(name = "password") private string password; @jsonmanagedreference("role") @onetomany(mappedby = "user", cascade = cascadetype.all,fetch = fetchtyp...
Read more

C++ 11 "class" keyword -

C++ 11 "class" keyword - i've started using c++ 11 little bit more , have few questions on special uses of class keyword. know it's used declare class, there's 2 instances have seen don't understand: method<class t>(); and class class_name *var; why precede typename keyword class in first example, , keyword pointer in sec example? that known elaborated type specifier , necessary when class name "shadowed" or "hidden" , need explicit. class t { }; // love of god don't t t; t t2; if compiler smart, give these warnings: main.cpp:15:5: error: must utilize 'class' tag refer type 't' in scope t t2; ^ class main.cpp:14:7: note: class 't' hidden non-type declaration of 't' here t t; ^ if class not defined, deed forwards declaration. for example: template <typename t> void method() { using type = typename t::value_type; } method...
Read more