Running comet on a cluster - Revision history

Markdayel at 19:47, 14 May 2009

2009-05-14T19:47:32Z

Markdayel at 17:10, 28 April 2009

2009-04-28T17:10:07Z

Markdayel: moved Recommendations for running comet to Running comet on a cluster

2009-04-13T18:09:11Z

moved Recommendations for running comet to Running comet on a cluster

Markdayel at 17:59, 13 April 2009

2009-04-13T17:59:55Z

Markdayel at 17:59, 13 April 2009

2009-04-13T17:59:25Z

Markdayel: Created page with 'We highly recommend running 'comet' on a cluster of machines, rather than one. The program takes some time to run (an hour or two for early events like symmetry breaking; overni...'

2009-04-13T17:56:14Z

Created page with 'We highly recommend running 'comet' on a cluster of machines, rather than one. The program takes some time to run (an hour or two for early events like symmetry breaking; overni...'

New page

We highly recommend running 'comet' on a cluster of machines, rather than one. The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.) Running the program concurrently on at least 5 machines (I used 9) lets you test the effect of changing one parameter through a range of values. You can set it going then come back the following day and have the whole thing layed out for you. I've found this very useful, and have included a set of scripts to automate the process.

I recommend putting a default '''cometparams.ini''' into a main directory for the data e.g. '''~/runs'''. In that directory run the '''varyset''' script (included with the source):
varyset <parameter> <startval> <endval> <number of steps>
This will create a subdirectory within '''runs''' that contains subdirectories numbered 1,2,3,etc. each containing a version of the '''cometparams.ini''' file with the <parameter> value varying in linear steps between <startval> and <endval>. It will also add information to run the individual comet jobs into ~/joblist.

If you have access to a cluster with a working job control system, you might want to use that. I had trouble with the job control system on the cluster I was using, and ended up writing my own:

On the head node, I have the
startnewjobs
script running as a cron job. This checks to see if the worker nodes are idle (5 min load average below a certain threshold) and starts the next job if they are.

If you don't have access to a cluster, there is a single computer version of this
startjobsloop
which will check ~/joblist for new jobs and run them sequentially.

← Older revision		Revision as of 17:10, 28 April 2009
Line 1:		Line 1:
−	We highly recommend running 'comet' on a cluster of machines, rather than just one. The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.) Running the program concurrently on at least 5 machines (I used 9) lets you efficiently test the effect of changing one parameter through a range of values. You can set it going then come back the following day and have the whole thing ~~layed~~ out for you. I've found this very useful, and have included a set of scripts to automate the process, including automatically generating the montage of images seen in the robustness section so you can quickly scan the effect of varying the parameter.	+	We highly recommend running 'comet' on a cluster of machines, rather than just one. The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.) Running the program concurrently on at least 5 machines (I used 9) lets you efficiently test the effect of changing one parameter through a range of values. You can set it going then come back the following day and have the whole thing laid out for you. I've found this very useful, and have included a set of scripts to automate the process, including automatically generating the montage of images seen in the robustness section so you can quickly scan the effect of varying the parameter.

← Older revision		Revision as of 17:59, 13 April 2009
Line 1:		Line 1:
−	We highly recommend running 'comet' on a cluster of machines, rather than one. The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.) Running the program concurrently on at least 5 machines (I used 9) lets you test the effect of changing one parameter through a range of values. You can set it going then come back the following day and have the whole thing layed out for you. I've found this very useful, and have included a set of scripts to automate the process, including automatically generating the montage of images seen in the robustness section so you can quickly scan the effect of varying the parameter.	+	We highly recommend running 'comet' on a cluster of machines, rather than just one. The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.) Running the program concurrently on at least 5 machines (I used 9) lets you efficiently test the effect of changing one parameter through a range of values. You can set it going then come back the following day and have the whole thing layed out for you. I've found this very useful, and have included a set of scripts to automate the process, including automatically generating the montage of images seen in the robustness section so you can quickly scan the effect of varying the parameter.

@@ Line 10: / Line 10: @@
 On the head node, I have the
   startnewjobs
-script running as a cron job.  This checks to see if the worker nodes are idle (5 min load average below a certain threshold) and starts the next job if they are.
+script running as a cron job every 15 minutes.  This checks to see if the worker nodes are idle (5 min load average below a certain threshold) and starts the next job if they are.
-If you don't have access to a cluster, there is a single computer version of this
   startjobsloop
 which will check ~/joblist for new jobs and run them sequentially.

← Older revision	Revision as of 18:09, 13 April 2009
(No difference)

@@ Line 1: / Line 1: @@
-We highly recommend running 'comet' on a cluster of machines, rather than one.  The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.)  Running the program concurrently on at least 5 machines (I used 9) lets you test the effect of changing one parameter through a range of values.  You can set it going then come back the following day and have the whole thing layed out for you.  I've found this very useful, and have included a set of scripts to automate the process.
+We highly recommend running 'comet' on a cluster of machines, rather than one.  The program takes some time to run (an hour or two for early events like symmetry breaking; overnight if you want to look at the later motility, pulsatile motion etc.)  Running the program concurrently on at least 5 machines (I used 9) lets you test the effect of changing one parameter through a range of values.  You can set it going then come back the following day and have the whole thing layed out for you.  I've found this very useful, and have included a set of scripts to automate the process, including automatically generating the montage of images seen in the robustness section so you can quickly scan the effect of varying the parameter.
@@ Line 16: / Line 16: @@
   startjobsloop
 which will check ~/joblist for new jobs and run them sequentially.