Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/joblib/joblib/issues/883 below:

"timeout or by a memory leak.", UserWarning · Issue #883 · joblib/joblib · GitHub

Mac OS 10.14.4
Python 3.6
Joblib 0.12.5

I have 32 big batches, and each batch contains unequal number of works. I am using joblib within a loop for the big batch, and the joblib function is to run the works within one batch at the same time. The errors come when I run the loop. However, when I run the batch one by one, there is no such error.

pseudo code:

results_list=[]
for batch in batch_list:
    results=Parallel(n_jobs=num_cores,verbose=1)(delayed(func)(x[i]) for i in batch )))
    result_list.append(results)

/Users/x/miniconda3/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py:700: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  "timeout or by a memory leak.", UserWarning

• I have tried gc.collection() to release the memory. But it does not help.
• I checked the total memory, and the memory is only 10% of the total I have.
• When I change the worker number to a smaller number (from 20 to 10), more warning comes.
• During the loop, the chance and frequency for the warning to show is larger if that batch has more works inside.
• Based on my previous validation on some cases, even with this warning, the results look the same as the one generated without warning (when I do it individually)

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4