Best Practices

Using ACME inside a Python script requires some precautions. Please follow this guide in case you’re experiencing strange RuntimeError or recursion exceptions.

Use a __main__ Block

First, please ensure your script contains a main module block that defines a central entrypoint for the Python interpreter. That means instead of just lining up code blocks one after another in a file, encapsulate those parts of the code, that are supposed to be executed only once inside a __main__ block. Thus, if your script looks like this

script_nomain.py

# Do something, then otherthing

import package1
import package2

def do_this(...):
    ...
    return something

def do_that(...):
    ...
    return otherthing

x = ...
y = ...

z = do_this(...x,y,...)
...
w = do_that(...,z,...)

re-structure it using a __main__ block to clearly delineate which parts are intended to be executed just once:

script_withmain.py

# Do something, then otherthing

import package1
import package2

def do_this(...):
    ...
    return something

def do_that(...):
    ...
    return otherthing

if __name__ == "__main__":
    x = ...
    y = ...

    z = do_this(...x,y,...)
    ...
    w = do_that(...,z,...)

If script_withmain.py is called directly (e.g., via run script_withmain.py in iPython) the interpreter uses the main module block as its entry point of code execution.

Isolate Your Function

To avoid any recursive importing errors or accidental spawning of workers by workers, it is further strongly recommended to capsulate your function func in a dedicated file separate from the ParallelMap call. Thus, instead of using a single script

my_script.py

# My processing script

import numpy as np
from acme import ParallelMap

def func(x, y, z=3):
    return np.dot(x, y) + z

if __name__ == "__main__":
    y = np.array([[4, 1], [2, 2]])
    x1 = np.arange(4).reshape(2,2)
    x2 = np.arange(4, 8).reshape(2,2)
    with ParallelMap(func, [x1, x2], y) as pmap:
        results = pmap.compute()

split up the definition of func and its ACME parallelization:

my_func.py:

# My processing function

import numpy as np

def func(x, y, z=3):
    return np.dot(x, y) + z

acme_script.py:

# My ACME script for func

import numpy as np
from acme import ParallelMap
from my_func import func

if __name__ == "__main__":
    y = np.array([[4, 1], [2, 2]])
    x1 = np.arange(4).reshape(2,2)
    x2 = np.arange(4, 8).reshape(2,2)
    with ParallelMap(func, [x1, x2], y) as pmap:
        results = pmap.compute()

Then simply launching acme_script.py via iPython does the trick:

>>> run acme_script.py

Note Just like any regular Python module, my_func.py permits to define several distinct functions. This means, if, e.g., func requires additional helper routines, they can all be migrated to my_func.py. For instance:

my_func.py:

# My processing function

import numpy as np

def func(x, y, z=3):
    help_here(...)
    help_there(...)
    return np.dot(x, y) + z

def help_here(...):
    ...

def help_there(...):
    ...