How do I avoid having to allocate N copies of my data if I want to execute N jobs?

When using peercellfun, you need to provide the arguments to your function as cell arrays. Each cell array should have one element per job that you want to execute. This can be problematic if you need to provide the same large array (e.g., electrophysiological data) as input to each of your jobs, since this can require impossible amounts of memory.

One way to take care of this is by using input- and outputfiles. See the FAQ on how_can_i_combine_fieldtrip_with_peer_distributed_computing for this approach.

Another way to take care of this is by avoiding to use peercellfun, and instead relying on the lower-level peerfeval and peerget. Peerfeval will submit a single job to an available peerslave, and return a job ID. Peerget should be provided with this job ID and will then retrieve the output of your job.

To use peerfeval and peerget properly, you will need to write some job-scheduling scaffolding code. Please refer to peercellfun's source code for an example of how to do this.

faq/how_do_i_avoid_having_to_allocate_n_copies_of_my_data_if_i_want_to_execute_n_jobs.txt · Last modified: 2011/09/07 12:44 by lilla

You are here: startfaqhow_do_i_avoid_having_to_allocate_n_copies_of_my_data_if_i_want_to_execute_n_jobs
CC Attribution-Share Alike 3.0 Unported Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0
This DokuWiki features an Anymorphic Webdesign theme, customised by Eelke Spaak and Stephen Whitmarsh.
Mobile Analytics Website Security Test