Hey folks,

I bought an RTX 2080 on Amazon*, being delivered tomorrow, and was wondering if you might help confirm my plans for my GPUs.

I have 7 GPUs worth crunching with:
2080, 1050 Ti, 980 Ti, 980 Ti, 970, 660 Ti, 660 Ti.

I have 2 rigs each capable of housing 3 GPUs:
- 3-yr-old Alienware Area-51 R2 main gaming rig
- 9-yr-old XPS 730x basement cruncher.

My goals are, in this order:
- Use the 2080 for gaming
- Keep a 980 Ti for gaming and testing
- Optimize ability to crunch GPU Grid tasks on all GPUs, knowing that 2080 isn't supported and may cause tasks to not be issued on the PC that has it until acemd is updated (right?)
- Use the GPUs to crunch other projects if GPUGrid isn't setup yet to support me

I'm thinking of doing this:

Area-51 R2:
- 2080 (for sure staying in this PC)
- 980 Ti (for sure staying in this PC)
- 660 Ti
* reasoning: Game on 2080, 980 Ti as backup/testing games, and all 3 crunch BOINC for non-GPU-Grid projects until GPUGrid gets fixed

XPS 730x:
- 1050 Ti
- 980 Ti
- 970
* reasoning: Maximum GPUGrid with remaining GPUs

Shelf:
- 660 Ti
- 460 (gets a friend on the shelf)

Does this sound like it'd "best optimize" my goals? Let me know. Thanks.

* Great deal here, in my opinion - $850, on sale for $50 for $800, and a 5% %40 coupon checkbox, for final total $760 pre-tax, includes 2 games, Battlefield V and Anthem.
https://www.amazon.com/gp/product/B07GHVK4KN

Also consider CUDA GPU Capability:
2080, >6.1
1050 Ti, 6.1
980 Ti, 5.2
970, 5.2
660 Ti, 3.0
460, 2.1
https://developer.nvidia.com/cuda-gpus
I'd put them:
980 Ti + 970
2080 + 1050 Ti
Capability is one of the factors the work server can consider in assigning WUs. This may get more of the same kind of work to all cards.
Just a thought.

BTW, I donate my legacy cards, etc to the New-2-U charity here.

Power supply/cooling capability/spacing would be a bigger concern for me if that was my setup.

You can tell BOINC to ignore GPUGrid for just the 2080 until Turing cards work with GPUGrid. No need to not run GPUGrid on all 3 cards just because one doesn't work.

Hmm, I thought that wouldn't work because BOINC reports to the projects the "biggest card", so GPUGrid would think I have 3 2080 GPUs and thus wouldn't give me any work. Are you sure your proposal would work and still allow me to do BOINC work from other projects on the 2080? If so, how?

Do you have this line in your cc_config.xml ???

Hmm, I thought that wouldn't work because BOINC reports to the projects the "biggest card", so GPUGrid would think I have 3 2080 GPUs and thus wouldn't give me any work.

Are you sure your proposal would work and still allow me to do BOINC work from other projects on the 2080?

If so, how?

Hmm, I thought that wouldn't work because BOINC reports to the projects the "biggest card", so GPUGrid would think I have 3 2080 GPUs and thus wouldn't give me any work. Are you sure your proposal would work and still allow me to do BOINC work from other projects on the 2080? If so, how?

:) Did you guys know that I'm responsible for <exclude_gpu> being included in BOINC? I know how it works and how to use it.

I didn't know that GPUGrid was giving work to 20-series GPUs, even though it would fail on them. That ends up being good for me, in a way, because I can get GPUGrid work for the other 2 GPUs in the system, and BOINC should get work from other projects for the 2080.

Thanks for helping to clarify that for me. I'll definitely be adding the GPU exclusion.

I'm currently employing gpu_exclude statements for both GPUGrid and Einstein for one of my 4 card hosts with a RTX 2080. Works fine preventing that card from being used.

But that causes issues with a project_max_concurrent statement for Seti.

It prevents all Seti cpu tasks from running leaving only the four gpu tasks running.

I have a thread in the Linux/Unix section of the Questions and Answers forum at Seti.

https://setiathome.berkeley.edu/forum_thread.php?id=83645

For now I have to remove the project_max_concurrent statement from cc_config and use the cpu limitation in Local Preferences to limit the number of cores to 16.

Why do you have an exclude for Einstein?

Why do you have an exclude for Einstein?

Thanks. I confirmed that at least one of the Einstein task types failed immediately on my 2080, so I also added them to my GPU Exclusion list for that GPU. Pity, really.

:) Did you guys know that I'm responsible for <exclude_gpu> being included in BOINC? I know how it works and how to use it.

:) I'm a rock star at breaking things, for sure!
I am happy to hear that you find the feature as useful as I do!

Power supply/cooling capability/spacing would be a bigger concern for me if that was my setup.

:)

I'm not too concerned with cooling. My cramped GPUs run hot, and I know it. For my main rig, I just don't do GPU work on it during the day, and instead let it run at night. Even with a max fan curve, it routinely runs around 75-80*C overnight, and the small office becomes very well heated. But I already stress tested the overclocks, and know it is 100% stable, and the GPU fans have proven to be very durable too.

What can I say? I like it hot and I like it loud. ;)

There's something to be said for white noise...

Power connectors.

The basement cruncher, my old XPS 730x, has 4 6-pin connectors, and I previously made 2-more using molex adapters, for a total of 6 6-pin connectors. But 0 8-pin connectors.

My GPU additional power requirements are:
6+8 : EVGA GeForce RTX 2080 XC ULTRA GAMING
None: EVGA GeForce GTX 1050 Ti SSC GAMING
8+8 : EVGA GeForce GTX 980 Ti FTW GAMING ACX 2.0+
6+8 : Dell GTX 980 Ti
6+6 : EVGA GeForce GTX 970 FTW ACX 2.0
6+6 : EVGA GeForce GTX 660 Ti FTW+ 3GB w/Backplate
6+6 : MSI GTX 660 Ti TwinFrozr III OC 3GB

So, I think this means I'm going to:
Area 51 R2: 2080, 980 Ti, 980 Ti
XPS 730x: 970, 1050, 660 Ti
Shelf: 660 Ti

Fun!

:) I'm a rock star at breaking things, for sure!
I am happy to hear that you find the feature as useful as I do!

BTW, the new Ryzens coming out mid year will be PCIe 4.0, so again double the speed of PCIe 3.0. You'll need a 500 series MB for PCIe 4.0, on the older boards they'll still run at PCIe 3.0.

:) I'm a rock star at breaking things, for sure!
I am happy to hear that you find the feature as useful as I do!

I about to join you as a "rock star" for breaking things to apparently.

The client code commit that DA wrote to fix my original problem is going to cause major problems for anyone using a max_concurrent or project_max_concurrent statement.

The unintended consequence of the code change prevents requesting work fetch task replacement until the hosts cache is empty.

Only then does the host report all finished work and then asks for work to refill the cache. So the end of keeping your cache topped up at every 5 minute scheduler connection.

The PR2918 commit is close to be being accepted into the master branch.

I have voiced my displeasure but since only DA usually authorizes pull requests into the master branch, that decision is up to him. Richard Haselgrove also has voiced his concerns.

:) I'm a rock star at breaking things, for sure!
I am happy to hear that you find the feature as useful as I do!

I about to join you as a "rock star" for breaking things to apparently.

The client code commit that DA wrote to fix my original problem is going to cause major problems for anyone using a max_concurrent or project_max_concurrent statement.

The unintended consequence of the code change prevents requesting work fetch task replacement until the hosts cache is empty.

Only then does the host report all finished work and then asks for work to refill the cache. So the end of keeping your cache topped up at every 5 minute scheduler connection.

The PR2918 commit is close to be being accepted into the master branch.

I have voiced my displeasure but since only DA usually authorizes pull requests into the master branch, that decision is up to him. Richard Haselgrove also has voiced his concerns.

It sounds like you were using "max concurrent" to mean "only run this many at the same time, but allow fetch of more."

David is likely arguing that, if you can't run more than that many simultaneously, then why buffer more? Consider tasks that take 300 days to complete (yes, RNA World has them). If you're set to only run 3 as "max concurrent", then why would you want to get a 4th task that would sit there for 300 days?

You might consider asking for a separation of functionality --- "max_concurrent_to_schedule" [which is what you want] vs "max_concurrent_to_fetch" [which is what David is changing max_concurrent to mean]. Then you could set the first one to a value, and leave the second one unbound, and get back your desired behavior.

I hope this makes sense to you. Please feel free to add the text/info to the PR.

Note: I doubt it waits until the cache is completely exhausted of max_blah items before asking more. I'm betting, instead, that work fetch will still top off, even if you have some of that task type, but only up to the max_blah setting.

max_concurrent
The maximum number of tasks of this application to run at a given time.

project_max_concurrent
A limit on the number of running jobs for this project.

:) I'm a rock star at breaking things, for sure!
I am happy to hear that you find the feature as useful as I do!

I about to join you as a "rock star" for breaking things to apparently.

The client code commit that DA wrote to fix my original problem is going to cause major problems for anyone using a max_concurrent or project_max_concurrent statement.

The unintended consequence of the code change prevents requesting work fetch task replacement until the hosts cache is empty.

Only then does the host report all finished work and then asks for work to refill the cache. So the end of keeping your cache topped up at every 5 minute scheduler connection.

The PR2918 commit is close to be being accepted into the master branch.

I have voiced my displeasure but since only DA usually authorizes pull requests into the master branch, that decision is up to him. Richard Haselgrove also has voiced his concerns.

It sounds like you were using "max concurrent" to mean "only run this many at the same time, but allow fetch of more."

David is likely arguing that, if you can't run more than that many simultaneously, then why buffer more? Consider tasks that take 300 days to complete (yes, RNA World has them). If you're set to only run 3 as "max concurrent", then why would you want to get a 4th task that would sit there for 300 days?

You might consider asking for a separation of functionality --- "max_concurrent_to_schedule" [which is what you want] vs "max_concurrent_to_fetch" [which is what David is changing max_concurrent to mean]. Then you could set the first one to a value, and leave the second one unbound, and get back your desired behavior.

I hope this makes sense to you. Please feel free to add the text/info to the PR.

Note: I doubt it waits until the cache is completely exhausted of max_blah items before asking more. I'm betting, instead, that work fetch will still top off, even if you have some of that task type, but only up to the max_blah setting.

max_concurrent
The maximum number of tasks of this application to run at a given time.

project_max_concurrent
A limit on the number of running jobs for this project.

Thanks for the comments Jacob. I have added your observations to my post and will await what Richard has to say about your new classifications when the new day for him begins.

[Edit] I should also comment that I have been using <project_max_concurrent> statements in all my hosts for years.

They have never been limited to the N number of tasks in those statements in their caches. I have always maintained the server side 100 cpu +100 tasks per gpu limit for the caches on all hosts. Never had any issues fetching replacement work for my hosts to keep topped up at the cache limit.

The problem I incurred in my original post was when I added the gpu_exclude statements which prevented all cpu tasks from running.

You're welcome. Since max_concurrent documentation has a 'documented meaning' already, you might use that to suggest (gently push) toward keeping it the same, and putting any work fetch max limits into a new variable. I could see it ending up that way maybe.

Hi Keith and Jacob!

Slightly odd place for this conversation, but I hope other posters will excuse me joining in.

There's a problem with <max_concurrent>. I noticed a couple of years ago that it could prevent full use of your computer, if you tried to run both GPU and CPU versions of the same app at the same time. Last year, Keith fell over the same problem, and with two independent reports David started work on fixing it.

This work is explicitly about the proper definition of max_concurrent, but in fixing it, David realised that if you do less work concurrently, you'll do less work overall, and you might over-fill the cache and run into deadline problems.

So, for the time being, he's put in a rather crude kludge. I've seen and documented a case where a project ran completely dry before finally requesting new work. I think that there's universal agreement that this is the wrong sledgehammer, used to crack the wrong nut.

I'm trying to put forward the concept of "proportional work fetch". I tend to run a 0.25 day cache (partly because of the fast turnround required by this project). If you run on all four cores of a quad CPU, BOINC will interpret that as a 1.0 day CPU cache to keep all four cores busy for 0.25 days. A max_concurrent of 2 should limit that to 0.5 days, and so on. Unless anyone can suggest a better solution?

I was on a conference call last night, where I heard other developers, in no uncertain terms, urge David to start work on a better work fetch solution as soon as possible. And, in particular, not to release a new client in the gap where the max_concurrent bug has been fixed but work fetch is broken.

We're getting very close to the completion of the max_concurrent fix. Keith's logs from two nights ago revealed a small bug, which I've reported, but should be easy to fix. And then we can move on to phase 2.

Contrary to what Keith said, under the new "Community Governance" management of BOINC, no developer is allowed to merge their own code without independent scrutiny - not even David. I got approval last night that, if Keith and I can say that the max_concurrent code works within its design limits (i.e. putting aside the work fetch bug for the moment), and if Juha can confirm that the C++ code has been written correctly, then we can collectively approve and merge. David wants that to happen so he has a clean code base before he starts on the work fetch part of the problem.

I'll keep an eye on progress, and it's very easy in Windows to test as we go along (replacement binaries are available - automatically, within minutes - of any code change: And if the code won't compile, that's also reported back to the developer automatically). But I don't run the sort of heavy metal that you guys do, so any help with the monitoring and testing process that you can contribute will be greatly appreciated.

Good morning Richard, sorry about the location of this discussion. Jacob has provided some useful information to enhance my understanding.

My concerns were that David in the PR2918 comments said:

Not fetching work if max_concurrent is reached is the intended behavior.

I've never liked using those 2 commands when trying to limit a project to something like 50% of CPU threads awhile wanting another project to use the other 50%. I would many times end up with a full queue of the project using max # of tasks command and the other threads would be idle as the queue would be full. Another situation where the task run priority should be separate from the work download priority and another BOINC client instance ends up being the favorable way to finely tune BOINC management on a PC.

Are you guys sure that GTX 2080 GPUs get work from GPU Grid?
From my testing, it seems my PC hasn't gotten a single task from that project since installing that GPU.
http://www.gpugrid.net/results.php?hostid=326413

The only 2080s at this time that should get WUs are the ones in the same machine as a 1000 series card or below. The WU will not yet work on 2000 series cards

My machine has 2080, 980 Ti, 980 Ti. And I have a GPU Exclusion setup so GPUGrid work doesn't run work on the 2080.

Yet, GPUGrid work fetch never gives any work for my PC.

Any idea why?

My machine has 2080, 980 Ti, 980 Ti. And I have a GPU Exclusion setup so GPUGrid work doesn't run work on the 2080.

Yet, GPUGrid work fetch never gives any work for my PC.

Any idea why?

Yes, I have that set correctly.

However, this message appears to be related, and appears to show whenever a Short/Long task is available but not given to me.
I'll have to think about what it means, and if I have something misconfigured.

Edit: My cache settings are for 10 days buffer, plus 0.5 additional. Since I believe BOINC interprets that as "may be disconnected for 10 days", it may be limiting my ability to get work based on a calculation involving that 10 days.

Time for me to rethink my cache settings (which I have intentionally set for other valid reasons), and then retest.

2/7/2019 3:36:39 PM | GPUGRID | Tasks won't finish in time: BOINC runs 85.2% of the time; computation is enabled 95.1% of that

After I changed my cache settings, from 10.0d and 0.5d, to 2.0d and 0.5d .... The message went away, and I started getting GPUGrid work for the first time on this PC since getting the RTX 2080.

http://www.gpugrid.net/results.php?hostid=326413

Thanks.

Good to see. I don't think I've ever seen that message about not finishing in time.

2/7/2019 3:36:39 PM | GPUGRID | Tasks won't finish in time: BOINC runs 85.2% of the time; computation is enabled 95.1% of that

For now I have to remove the project_max_concurrent statement from cc_config and use the cpu limitation in Local Preferences to limit the number of cores to 16.

We're getting very close to the completion of the max_concurrent fix.

We're getting very close to the completion of the max_concurrent fix.

Richard, Not sure what you guys are up to but I sure hope you take the WCG MIP project into consideration before you roll it out. https://www.worldcommunitygrid.org/forums/wcg/viewpostinthread?post=569786
They coded the use of the L3 Cache wrong and it uses 4-5 MB per MIP WU. If you exceed that I've had BOINC performance cut in half. I have to use max_current in my WCG app_config or I cannot run MIP simulations.

<app_config>
<app>
<name>mip1</name>
<!-- needs 5 MB L3 cache per mip1 WU, use 5-10 -->
<!-- Xeon E5-2699v4, L3 Cache = 55 MB -->
<max_concurrent>10</max_concurrent>
<fraction_done_exact>1</fraction_done_exact>
</app>
</app_config>

We're getting very close to the completion of the max_concurrent fix.