Downloading large outputs using Get-IshPublicationOutputData is slow, is there anything I can do to speed it up

I am currently using Get-IshPublicationOutputData to download output from a bunch of publications. I use this with powershell's foreach-object -parallel, which executes jobs in parallel without waiting for the current item in the for loop to complete. This works quite well for outputs where are relatively small, less than 100MB. I can download 300+ distinct outputs in a couple of minutes.

However, we also have some outputs that are 1GB+, and when trying to download those outputs using Get-IshPublicationOutputData, it is painfully slow to the point of being unusable. Is there anything I can do to speed this up?

emoji
Parents
  • By default, the -Parallel parameter in PowerShell's ForEach-Object cmdlet sets the maximum number of parallel jobs to the number of available logical processors. You can try increasing this value to allow for more concurrent downloads. For example, you can set it to a specific number like -Parallel 10 to have up to 10 parallel downloads at a time. Be cautious not to set it too high, as it may overwhelm the system.

    basket random

    emoji
  • The magic of -Parallel happens on the client machine, your laptop. Making your client machine trigger 10 parallel requests (e.g. 10 times 'Print a 1000 page PDF') will not make the server go faster in handling 10 requests (of 'Print a 1000 page PDF'). Just trying to be clear that doing 10 simple requests in parallel on the client goes fast... but the actual execution is on the server which has other limitations.

    emoji
Reply
  • The magic of -Parallel happens on the client machine, your laptop. Making your client machine trigger 10 parallel requests (e.g. 10 times 'Print a 1000 page PDF') will not make the server go faster in handling 10 requests (of 'Print a 1000 page PDF'). Just trying to be clear that doing 10 simple requests in parallel on the client goes fast... but the actual execution is on the server which has other limitations.

    emoji
Children
No Data