Background processes ( How dangerous it could be)

As a software developer we face some situation when user response time is very high. For example sometime we need to send 10,000 mails at a time or converting 1000 files to some other formats at a time.

In such cases users need to wait a long time to get response. To avoid this we usually run the task as a background process and show user a message like “Your request is being processed”.

In my case also there was similar situation and I was handling in this way-

for ($i=0;$i<10000;$i++){
  exec("php convertFileAndMail.php $i >/dev/null 2>&1 & echo $!",, &$output);
}

And the server went into inoperable state and and once even my database server which was running in same system crashed.

Finding the reason:

To find the reason I created 2 files

1) loop.php
2) sleep.php

Codes of sleep.php: (sleep for 5 sec and log 2^nd argument passed)

<?php
  sleep(5);
  file_put_contents('log.txt',$argv[1]."\n", FILE_APPEND);
?>

Codes of loop.php: (loop and execute sleep.php)

<?php
  $c=100000;
  for($i=0;$i<$c;$i++) {
     exec("php sleep.php $i >/dev/null 2>&1 & echo $!",, &$output);
  }
?>

And then in terminal if you run following command –

[php loop.php]

Within a few seconds this will hang a normal system. 😦

Reason:

If we look into the files there are almost 0 processing power needed to execute the sleep.php because the file only sleeps for 5 sec and write a number to a file.

Then what is wrong?

The file loop.php adding tasks as background processes and sleep.php takes 5 sec to complete due to sleep. Within this 5 sec loop.php adding millions of background processes.

So in the certain time number of simultaneous processes is huge and it takes lots of memory. Which may even cause system breakdown.

Way out and suggestions:

From above example it is very clear that we have to keep number of simultaneous processes less.

There are a few ways to reach it according to your need-

• Avoid creating background processes.
• But sometime we need background processes as mentioned above, in that case try to create less number of background processes. May be in one background process you can do all you need.
• There are some scenario where we will be needing multiple background processes. Here we have few tools like GEARMAN (http://gearman.org) to handle simultaneous background processes. (I will come up with Gearman concept in another blog later).

Hope this will help you to avoid a pothole in your way. 🙂