Author |
Message |
Keith Myers Send message
Joined: 13 Dec 17 Posts: 1360 Credit: 7,913,813,181 RAC: 5,493,935 Level
Scientific publications
|
For Windows users attempting to run Python on GPU tasks so that they do not fail it is recommended to change the default pagefile settings.
Change from default or system managed settings to Custom with a minimum starting pagefile size of 35000 MB and the upper limit to be 512000 MB.
See this post for location images in Windows.
https://www.gpugrid.net/forum_thread.php?id=5336&nowrap=true#59029
|
|
|
|
Useful notice.
But in a good way, this information should be in project preferences when choosing applications to crunch(how it's done in WCG project).
Or at least in the top post of announcing new application's thread.
A lot of time and effort could be saved(by many people) if such important information was placed in the right place : /
|
|
|
djo5296Send message
Joined: 31 Jul 18 Posts: 1 Credit: 3,374,175 RAC: 0 Level
Scientific publications
|
Is it safe to allocate such a huge virtual memory on my SSD? Will it not affect the lifespan of my SSD? If yes, can I divide the recommended virtual memory size to multiple drives and maybe even put the bulk of the virtual memory on my HDD instead? |
|
|
Keith Myers Send message
Joined: 13 Dec 17 Posts: 1360 Credit: 7,913,813,181 RAC: 5,493,935 Level
Scientific publications
|
There is no danger to your SSD's. That trope is a long disproven fallacy about too many writes to an SSD killing your drive.
Many articles have tested the longevity of modern SSD drives and the none of them have any failures in literally petabytes of data written to them.
The figure in SSD specifications to pay attention to is Total Drive Writes per day or DWPD and TBW or Terabytes Written. The higher the number the better.
For typical consumer drives in the 256GB-2TB range drive life expectancy is on the order of tens of years.
A Samsung 850 Pro 1TB drive will last before failure in 342 years for example.
You would kill the HDD much faster than you ever would using the SSD.
https://www.enterprisestorageforum.com/hardware/ssd-lifespan-how-long-will-your-ssd-work/ |
|
|
WPrionSend message
Joined: 30 Apr 13 Posts: 96 Credit: 3,275,571,611 RAC: 15,307,185 Level
Scientific publications
|
Since we're talking about pagefile, I assume this storage won't be used until most/all of RAM is consumed.
What size RAM are those pagefile requirements based upon?
For example, if a computer has 128GB of RAM is a max pagefile of 512GB still needed or could that be less?
In other words, what's the max combined RAM + pagefile needed?
Thanks,
W |
|
|
KAMasudSend message
Joined: 27 Jul 11 Posts: 138 Credit: 526,581,194 RAC: 188,310 Level
Scientific publications
|
50GB is recommended. Where did the 512GB come from? |
|
|
Keith Myers Send message
Joined: 13 Dec 17 Posts: 1360 Credit: 7,913,813,181 RAC: 5,493,935 Level
Scientific publications
|
I don't have an answer for you since I don't run Windows.
Linux handles the issue much differently than Windows.
On my Linux hosts with 128GB of system RAM, I have not used any of the default 2GB pagefile.
On my Linux hosts with 32GB of system RAM, I see less than a 1GB of pagefile in use with the host using 17-18GB of the system RAM.
On this daily driver with 32GB of system RAM I see around 24-26GB of RAM in use and around 2.2-2.4 GB of the 6GB pagefile in use.
This host also supports two Gridcoin wallets along with BOINC usage and is why it is using more memory and pagefile than just my simple BOINC crunchers. |
|
|
Keith Myers Send message
Joined: 13 Dec 17 Posts: 1360 Credit: 7,913,813,181 RAC: 5,493,935 Level
Scientific publications
|
50GB is recommended. Where did the 512GB come from?
Yes, it looks like a max 50GB pagefile is being sufficient for Windows.
No need for a huge 512GB pagefile size. Assume typo here in play. |
|
|
WPrionSend message
Joined: 30 Apr 13 Posts: 96 Credit: 3,275,571,611 RAC: 15,307,185 Level
Scientific publications
|
50GB is recommended. Where did the 512GB come from?
The original post in this FAQ stated 512000 MB
|
|
|
Keith Myers Send message
Joined: 13 Dec 17 Posts: 1360 Credit: 7,913,813,181 RAC: 5,493,935 Level
Scientific publications
|
50GB is recommended. Where did the 512GB come from?
The original post in this FAQ stated 512000 MB
That's an upper limit for the pagefile to dynamically grow. Not that it would ever need one that large but it certainly would be a safe value.
I was just saying if you set it that large for the upper limit you would never have the issue.
And since I don't do Windows, I was just making a safe guess. With further comments from actual Windows users the upper safe limit is 50GB or 50000 MB. |
|
|
WPrionSend message
Joined: 30 Apr 13 Posts: 96 Credit: 3,275,571,611 RAC: 15,307,185 Level
Scientific publications
|
Thanks!
Now I'll circle back to my original question, which I guess is re: Windows only.
Is that 50GB upper limit for pagefile PLUS RAM, or just pagefile?
If a computer has 64GB or 128GB of RAM is any pagefile needed?? |
|
|
Keith Myers Send message
Joined: 13 Dec 17 Posts: 1360 Credit: 7,913,813,181 RAC: 5,493,935 Level
Scientific publications
|
Thanks!
Now I'll circle back to my original question, which I guess is re: Windows only.
Is that 50GB upper limit for pagefile PLUS RAM, or just pagefile?
If a computer has 64GB or 128GB of RAM is any pagefile needed??
Again, I state I don't run Windows so don't know all the particulars for everyone running Windows that has been successful or what their hosts have for hardware.
I am just stating that 50GB of pagefile seems to work. I don't know how much RAM they are using.
The reason why this issue only occurs for Windows is here:
https://www.gpugrid.net/forum_thread.php?id=5322&nowrap=true#58908
Has to do with how Windows enforces memory allocation. |
|
|