Re: Parallel Processing of subsets

Hi Daniel,

On Mon 19 Jun, 2023 at 22:09, Daniel Berger wrote:

Hello,

I want to do computations on subsets of a given set, that is quite large (around 600+ elements). Since there are quite many subsets, I do not necessarily want to process them all, but still as many as possible given my computational resources, using parallel processing.

For smaller sets (~25 elements), i have successfully used parfor, iterating over the number of subsets and using vecextract to get to the subsets themselves.

However, I don't think this approach is optimal for two reasons:

Each computation only takes a very short amount of time (few ms), so the added overhead wastes resources (or so I assume after having read the resources on using thread in PARI/GP), I think it would be better to split up the data into chunks and let each core compute a bunch before returning and getting a new batch of data.

It would be better for me to start with smaller subsets (like forsubset works) and the above approach doesn't do this.

So my questions are: Is my assumption above correct? What is the easiest and/or best way to do this in PARI/GP?

You might be able to split to set in two subsets A and B, do the parfor on the

subsets S_A of A, and iterate in the threads over the subsets S_B of B, and

take S_A\cup S_B as the subset you study.

Loïc