GPU Developers:

I've been testing on running multiple tasks at the same time on my GPU, and just noticed something odd in my results.

If you look at these results, you'll see that they say "Completed and Validated", and I got the bonus credit, but... Look at those Run Times and CPU Times!

How can they be valid, if they only ran for less than 10 seconds? What happened?

http://www.gpugrid.net/result.php?resultid=6695537
http://www.gpugrid.net/result.php?resultid=6695975

Note 1: I didn't actually watch these units, and so I don't know how long they really took. Sorry.
Note 2: I'm not trying to cheat any system. This just happened, and I'm looking for an explanation.

Can you figure out what happened here?

Regards,
Jacob

http://www.gpugrid.net/result.php?resultid=6695537 Was a short WU and you got only 16,200 credit for it despite what that page says.

Check here: http://www.gpugrid.net/workunit.php?wuid=4315248

http://www.gpugrid.net/result.php?resultid=6695975 Was indeed a long WU and the credit of 70,800 was correct.

One was a short unit, one was a long unit. I already understood that.

My question is:
How could they have only used that little time, and yet been completed and validated, and granted bonus credit? What went wrong?

Well... I'll tell you my setup, to see if that helps you at all in determining the cause.

Basically, I have 2 devices:
GPU Device 0: eVGA GTX 660 Ti 3GB FTW
GPU Device 1: eVGA GTX 460 1GB

For each of the 4 GPUGrid applications, I had been running the following in app_config.xml, in an attempt to run 2 tasks on a single GPU device, without dedicating a CPU to each task (since, when they run on Device 1, they don't usually use CPU at all):
<gpu_usage>0.5</gpu_usage>
<cpu_usage>0.001</cpu_usage>

I'm also attached to POEM@Home, and in order to run up-to-6 at a time while allocating a CPU for each task, for POEM's app_config.xml, I have:
<gpu_usage>0.166</gpu_usage>
<cpu_usage>1</cpu_usage>

GPUGrid mainly runs on Device 0, and at one point, I believe I saw the following all running on GPU Device 0, all at the same time:
- GPUGrid Task (0.5 GPU)
- Poem Task (0.166 GPU)
- Poem Task (0.166 GPU)
- Poem Task (0.166 GPU)

Now, that GPUGrid task may have been one that completed normally (ie: took ~35,000 secs), or it may have been one of these low-run-time tasks... I can't be sure, because I wasn't watching it closely enough.

I'll try to keep a better eye out on this, but if you find anything more out on your end, please let me know. I'd like to figure out what's happening.

Running multiple jobs on the same GPU is somewhat unorthodox, and beyond what we can troubleshoot. Feel free to keep experimenting for now, but ultimately this raises two concerns for us:

1) Users could potentially cheat the credit system this way, if this in fact a bug (and the run times are real, and not a bug that is misleading us).
2) Equally concerning for us, it could affect the integrity of the returned data. Corrupted results might be returned without being flagged as such.

For now, it just looks like a one-off peculiarity. But if either of these things become true, we would have to do something to fix it. Thanks for letting us know, and keep us posted if something like this happens again.

Nate,

I understand that we want to prohibit cheating, and that we want to make sure the results are valid. I'll definitely keep watching for more of these, and if I get any more, I'll report them.

But, since it already happened, and it seems that there's no way the result could be valid, then you might consider seeing if you can somehow account for it now (in the validator?), instead of waiting for it to happen again.

I'll let you know when it happens again.

Thanks,
Jacob

Running multiple jobs on the same GPU is somewhat unorthodox, and beyond what we can troubleshoot. Feel free to keep experimenting for now, but ultimately this raises two concerns for us:

But, since it already happened, and it seems that there's no way the result could be valid, then you might consider seeing if you can somehow account for it now (in the validator?), instead of waiting for it to happen again.

Maybe you could add a validation criteria so that the WU has to run for at least X amount of time to Validate. Probably best based on the app types (Short, Long).

nate,

I had 2 more tasks that appear to have completed way prematurely, yet were granted full credit plus bonus credit.
I think I was suspending and resuming various GPU tasks, when these tasks decided they were completed. That's about all the information I have :-/

Can you please verify that the results are unusable?
And then, could you also build additional checks into the validator, that will not validate tasks like these, whose results are unusable?

Thanks,
Jacob

http://www.gpugrid.net/result.php?resultid=6738039
Name I3R50-NATHAN_dhfr36_3-31-32-RND3834_0
Workunit 4348765
Created 9 Apr 2013 | 22:06:35 UTC
Sent 10 Apr 2013 | 3:51:18 UTC
Received 10 Apr 2013 | 9:06:18 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 126725
Report deadline 15 Apr 2013 | 3:51:18 UTC
Run time 3,280.11
CPU time 3,247.89
Validate state Valid
Credit 70,800.00
Application version Long runs (8-12 hours on fastest card) v6.18 (cuda42)

http://www.gpugrid.net/result.php?resultid=6738043
Name I10R38-NATHAN_dhfr36_3-31-32-RND3600_0
Workunit 4348769
Created 9 Apr 2013 | 22:09:36 UTC
Sent 10 Apr 2013 | 3:51:18 UTC
Received 10 Apr 2013 | 9:06:18 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 126725
Report deadline 15 Apr 2013 | 3:51:18 UTC
Run time 3.12
CPU time 0.98
Validate state Valid
Credit 70,800.00
Application version Long runs (8-12 hours on fastest card) v6.18 (cuda42)

For that GPU, I usually have 4 GPUGrid tasks queued up, and as many POEM OpenCL tasks queued as possible (they rarely have them available). So, just because there was a 5-hour difference between download and reported, means nothing for this scenario.

I'm pretty sure the task started, then immediately died, then immediately reported the result, and cashed in on bonus credits.

I hope they can fix this. I don't want the credits, but even more importantly, I don't want error results mixed in with their other successful results.

Uhoh, Nate!

I just had a whole rash of these, 18 of them, all Nathan long-run dhfr, just happen! And, from the best I can tell, this happened in the middle of the night while I was away from the PC.

Each task took 3 seconds, was marked completed and validated, and then was granted the full 70,800.00 credit. It looks like I'm going to be getting the most credit today that I've ever gotten before.

Have you guys begun your investigation into the validator??
I urge you to take action ASAP.
If there's anything I can do to help test, please let me know.

Thanks,
Jacob




Task
click for details
Show names Work unit
click for details Computer Sent Time reported
or deadline
explain Status Run time
(sec) CPU time
(sec) Credit Application
6742602 4352606 126725 11 Apr 2013 | 11:31:23 UTC 11 Apr 2013 | 11:34:25 UTC Completed and validated 3.61 0.90 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742575 4352579 126725 11 Apr 2013 | 11:25:58 UTC 11 Apr 2013 | 11:28:41 UTC Completed and validated 3.23 0.92 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742574 4352578 126725 11 Apr 2013 | 11:25:58 UTC 11 Apr 2013 | 11:28:41 UTC Completed and validated 3.56 0.98 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742557 4352561 126725 11 Apr 2013 | 11:25:58 UTC 11 Apr 2013 | 11:28:41 UTC Completed and validated 3.60 0.86 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742544 4352548 126725 11 Apr 2013 | 11:28:41 UTC 11 Apr 2013 | 11:31:23 UTC Completed and validated 3.25 0.92 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742543 4352547 126725 11 Apr 2013 | 11:28:41 UTC 11 Apr 2013 | 11:31:23 UTC Completed and validated 3.71 0.95 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742528 4352532 126725 11 Apr 2013 | 11:23:07 UTC 11 Apr 2013 | 11:25:58 UTC Completed and validated 3.66 0.89 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742481 4352486 126725 11 Apr 2013 | 11:18:59 UTC 11 Apr 2013 | 11:20:56 UTC Completed and validated 3.24 0.86 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742441 4352446 126725 11 Apr 2013 | 11:31:23 UTC 11 Apr 2013 | 11:34:25 UTC Completed and validated 3.26 0.92 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742440 4352445 126725 11 Apr 2013 | 11:31:23 UTC 11 Apr 2013 | 11:34:25 UTC Completed and validated 3.64 0.94 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742431 4352436 126725 11 Apr 2013 | 11:20:56 UTC 11 Apr 2013 | 11:23:07 UTC Completed and validated 3.65 0.90 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742430 4352435 126725 11 Apr 2013 | 11:20:56 UTC 11 Apr 2013 | 11:23:07 UTC Completed and validated 3.21 0.98 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742409 4352415 126725 11 Apr 2013 | 11:16:58 UTC 11 Apr 2013 | 11:18:59 UTC Completed and validated 3.71 0.87 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742386 4352392 126725 11 Apr 2013 | 11:23:07 UTC 11 Apr 2013 | 11:25:58 UTC Completed and validated 3.19 1.01 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742385 4352391 126725 11 Apr 2013 | 11:23:07 UTC 11 Apr 2013 | 11:25:58 UTC Completed and validated 3.43 0.98 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742383 4352389 126725 11 Apr 2013 | 11:28:41 UTC 11 Apr 2013 | 11:31:23 UTC Completed and validated 3.74 0.86 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6742377 4352383 126725 11 Apr 2013 | 11:16:58 UTC 11 Apr 2013 | 11:20:56 UTC Completed and validated 3.22 0.86 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6741810 4351878 126725 11 Apr 2013 | 6:50:41 UTC 11 Apr 2013 | 11:16:58 UTC Completed and validated 3.27 0.90 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)

They should probably disable running multiple WUs here, at least until this issue is resolved.

I disagree.

They have a way to find the erroneously-validated results, and so they have a way to filter them out whenever they go to actually use the data.

Disabling multiple WUs would hinder performance for a lot of people, those that run 2-at-a-time as well as those that run 1-at-a-time (and may not be connected to the internet very often)

What they should do is 2 things:
Priority 1) Fix the validator to stop marking these results valid, and thus stop issuing credits for invalid results
Priority 2) Fix the workunits so they do not error under whatever conditions they are erroring

- Jacob

What they should do is 2 things:
Priority 1) Fix the validator to stop marking these results valid, and thus stop issuing credits for invalid results
Priority 2) Fix the workunits so they do not error under whatever conditions they are erroring - Jacob

I know this is about the science, silly!

Fixing priority 1, the validator, will assure that invalid results are not included within any list of valid results, thus preserving "the science". Fixing priority 2 will prevent network strain and user confusion.

I hope they are working on both.

But in the meantime why do something that's causing corrupted results?

Not all results are corrupted, and the admins still have a way of finding the corrupted ones after they've been erroneously validated.

So, to answer your question "why keep doing it", the answer is "the science" (Overall I do more science with a higher throughput, running 2-at-a-time, even if I get some that immediately error out).

If the admins make a request for me to change, I will of course honor it. But, so far, I've been encouraged to test (find problems) and report results (and problems).

I just hope they fix the problems I find.

Regards,
Jacob

JK is the only one affected by the problem, as far as I can tell, and it's a nasty one. My guess is that the client does not keep the two WUs separate. Disable as soon as possible.

(Unless there is something more odd like read-only filesystems, file permissions, or the like).

Toni,

There are times when GPUGrid-2-at-a-time works just fine for me. In fact, I'd say most of the time, it works correctly.

If it's a bug in the client, as you suggest, then let's get it fixed! I'm a beta tester, testing 7.0.62, and if you can isolate the reason you think it's a client bug, we can contact David Anderson and he can fix it. What makes you think it's a client bug?

I'm much more likely to believe that it's an application problem, though. I'm able to run x-at-a-time successfully on all of my other GPU projects.

If you determine it's an application problem, could you please fix it, as well as the validator?

I look forward to your reply,
Jacob

Toni,

There are times when GPUGrid-2-at-a-time works just fine for me. In fact, I'd say most of the time, it works correctly.

If it's a bug in the client, as you suggest, then let's get it fixed! I'm a beta tester, testing 7.0.62, and if you can isolate the reason you think it's a client bug, we can contact David Anderson and he can fix it. What makes you think it's a client bug?

I'm much more likely to believe that it's an application problem, though. I'm able to run x-at-a-time successfully on all of my other GPU projects.

If you determine it's an application problem, could you please fix it, as well as the validator?

I look forward to your reply,
Jacob

Toni / Nate / GDR:

I disabled x-at-a-time for GPUGrid tasks, per Toni's request, on 4/12/2013.

Yet, I had 2 more results that were marked "Completed and validated", with Run time less than 4 seconds, granted full+bonus credit of 70,800, on 4/14/2013!

This leads me to believe that the problem is unrelated to running x-at-a-time.
Again, I ask you, have you begun your investigation??

Task Work unit Computer Sent Time reported or deadline Status Run time (sec) CPU time (sec) Credit Application
6755331 4362872 126725 14 Apr 2013 | 7:01:18 UTC 14 Apr 2013 | 10:38:35 UTC Completed and validated 2.40 0.84 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)
6754567 4362305 126725 14 Apr 2013 | 2:29:48 UTC 14 Apr 2013 | 10:38:01 UTC Completed and validated 3.19 0.84 70,800.00 Long runs (8-12 hours on fastest card) v6.18 (cuda42)

Specs:
Windows 8 x64, BOINC v7.0.62 x64 Beta, nVidia GeForce 314.22 WHQL, eVGA GTX 660 TI 3GB FTW, eVGA GTX 460 1GB, GPUGrid app_config using the following settings for all 4 apps:
<max_concurrent>9999</max_concurrent>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>0.001</cpu_usage>
</gpu_versions>

<cpu_usage>.001</cpu_usage>

Maybe. Could you also try running WITH the app_config to see what results you get?
I'm quite frustrated -- it really feels like I'm the only one trying to solve this problem.

Maybe. Could you also try running WITH the app_config to see what results you get?
I'm quite frustrated -- it really feels like I'm the only one trying to solve this problem.

I don't think you understand the cpu limitation portion.
I removed the GPUGrid.net project, and re-added it, and started working on some long-run Nathan's (without any app_config override). By default, they use "0.73 CPUs + 1 NVIDIA GPU" on my machine. Having an app_config that sets it to 0.001, does not make any difference, if only 1 task is running. That number is used when to determine how many CPU jobs to add.

For instance:

Without app-config, my system runs:
1 GPUGrid.net task (0.73 CPUs + 1 NVIDIA GPU)
2 WCG HCC GPU tasks (each: 1 CPUs + 0.5 NVIDIA GPUs)
6 CPU jobs

With app_config, my system runs:
1 GPUGrid.net task (0.001 CPUs + 1 NVIDIA GPU)
2 WCG HCC GPU tasks (each: 1 CPUs + 0.5 NVIDIA GPUs)
6 CPU jobs

See? I don't see how the app_config could possibly make a difference, when running only 1 GPUGrid task, but I'm testing it anyways, since I'm losing hope that the admins care.

You may not see it as a problem, but it affects their science results. Tasks are failing immediately, their validator is erroneously marking those invalid results as valid... and if I can prove (to myself) that running x-at-a-time is not the cause, then I will likely switch back to running x-at-a-time, for increased performance.

The admins need to begin an investigation, if they care.

See? I don't see how the app_config could possibly make a difference, when running only 1 GPUGrid task, but I'm testing it anyways

You may not see it as a problem, but it affects their science results.

And how do you propose they disable those things? Seriously. I don't believe it's possible. <sigh>
Man, I am SICK of all of these "ifs" and "let's do this and that."

I'm the only one testing to find the root cause, so far as I know. It's frustrating, to say the least, it feels like I'm in a hole. But if I'm the only one having the issue, then sure, I can accept being the only one doing the testing. If anyone else truly cares, they should be trying to reproduce the problem, instead of guessing at causes/fixes.

These proposals, some of which are ludicrous (making validation based on CPU time??), some of which are impossible (ban app-config??), and some of which are detrimental (limit to 1 gpu task per person??)... they're all the wrong approaches, in my opinion.

I said it before, and I'll say it again:

What they should do is 2 things:
Priority 1) Fix the validator to stop marking these results valid, and thus stop issuing credits for invalid results
Priority 2) Fix the workunits so they do not error under whatever conditions they are erroring

If you want to help me test, then help me test.
Otherwise, please don't guess at causes/solutions.

Thanks,
Jacob

I'll try to relax.

If I knew that an investigation was proceeding, or that a fix was being worked on, then I could relax a lot easier.

It's been 2 full weeks since the problem was reported, and the admin responses thus far have been "oh wow that's odd and beyond me", "wait and see if it happens again", and "probably a client issue".

some of which are impossible (ban app-config??)

Toni / Nate / GDF:

I have been doing additional testing. Since removing/readding the project, I have not had the issue, even when using a custom app_config for 0.001 CPU and 1-at-a-time task processing.

In order to test whether the remove/readd fixed the problem, I was wondering...
Could I please turn on 2-at-a-time task processing, to continue my testing?

Awaiting your reply,
Jacob

Toni / Nate / GDF:

Just had another one of these problems. It was using an app_config.xml file that was set for 0.001 CPU and 1.000 GPU, with a cc_config.xml file setup such that GPUGrid could run on either the GTX 660 Ti or the GTX 460.

So, again, I don't think the problem is with running multiple tasks at the same time on the same GPU. But it might still have something to do with using an app_config.xml file. I am still testing that.

HAVE YOU DONE ANYTHING SERVER SIDE TO CORRECT THE VALIDATOR AND CORRECT THE APPLICATION FROM ERRONEOUSLY COMPLETING LIKE THIS? It would also help if we could see, in the Stderr output, what Device the work unit was running on.

Regards,
Jacob

Toni / Nate / GDF:

Just had another one of these problems. It was using an app_config.xml file that was set for 0.001 CPU and 1.000 GPU, with a cc_config.xml file setup such that GPUGrid could run on either the GTX 660 Ti or the GTX 460.

So, again, I don't think the problem is with running multiple tasks at the same time on the same GPU. But it might still have something to do with using an app_config.xml file. I am still testing that.

HAVE YOU DONE ANYTHING SERVER SIDE TO CORRECT THE VALIDATOR AND CORRECT THE APPLICATION FROM ERRONEOUSLY COMPLETING LIKE THIS? It would also help if we could see, in the Stderr output, what Device the work unit was running on.

Regards,
Jacob

Name I14R3-NATHAN_dhfr36_5-9-32-RND6302_0
Workunit 4380107
Created 20 Apr 2013 | 2:43:43 UTC
Sent 20 Apr 2013 | 4:01:07 UTC
Received 20 Apr 2013 | 6:50:11 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 149974
Report deadline 25 Apr 2013 | 4:01:07 UTC
Run time 3.28
CPU time 0.83
Validate state Valid
Credit 70,800.00
Application version Long runs (8-12 hours on fastest card) v6.18 (cuda42)
Stderr output

<core_client_version>7.0.64</core_client_version>

Simba123:
Did you read my post completely and carefully?
The most recent task that I reported, could have only run on 1 of the 2 GPUs, alone (ie: not 2-at-once-on-same-GPU), based on my settings.

Thus, again, I've been able to PROVE that these problems are happening even when NOT running 2-at-once-on-same-GPU.

I'm still trying to figure out exactly what causes the problem.
Want to help?

Did you read my post completely and carefully?
The most recent task that I reported, could have only run on 1 of the 2 GPUs, alone (ie: not 2-at-once-on-same-GPU), based on my settings.

Thus, again, I've been able to PROVE that these problems are happening even when NOT running 2-at-once-on-same-GPU.

I'm still trying to figure out exactly what causes the problem.
Want to help?

You must not have been following the thread very closely. I'm trying my best to be informative and provide accurate information.

But relax - :) - The whole project is not in jeopardy. I wish the admins were much more proactive (sigh), but they make it sound like they can dig themselves out of this problem, even if results are quick-returned with this problem.

Here's a summary of what my testing has shown thus far:
- I have seen this issue only on Nathan Long-runs and Nathan Short-runs
- I have seen this issue running <gpu_usage> of 0.5 (aka: 2-at-once-on-same-GPU)
- I have seen this issue running <gpu_usage> of 1 (aka: Still using an app_config.xml file, with a <cpu_usage> of 0.001, but not doing 2-at-once-on-same-GPU)
- I have not yet seen this issue when not running an app_config.xml file; am testing this currently (I removed the GPUGrid project, re-added the GPUGrid project, verified it doesn't have an app_config.xml, got some tasks, verified they say something like "0.728 CPUs + 1 NVIDIA GPU", and am letting them complete without adding an app_config.xml file)
- I have not yet tested using <exclude_gpu> in the cc_config.xml to limit GPUGrid.net to just 1 of my 2 GPUs.
- Toni said (several days ago) that I'm the only one affected by the problem; but I sure wish someone could help to reproduce it with me, to better understand it.

Right now, my gut tells me that this is an application problem, likely related to:
a) Using an app_config.xml file even when it has <gpu_usage> set to 1.000... or
b) Having 2 homogenous GPUs in the system (maybe dependent on whether GPUGrid is allowed to run on both, but not necessarily dependent)

You mention "what to look for"... Well, I'm just looking at my results, and seeing if the the "Run time" values look correct for the types of tasks that I've completed. That's all I'm doing.

Hope that helps to summarize. I'm still testing to resolve this issue.
Any help would be welcomed.
If you want to help, then try to provide evidence that one of my 2 "gut feelings" is correct.
- If you only have 1 GPU, then try using an app_config.xml file (use the one below, with <cpu_usage> set to 0.001, and <gpu_usage> set to 1, with <max_concurrent> set to 9999)... to see if it causes a problem.
- If you have 2 homogenous GPUs, try letting GPUGrid run on both and see if it causes a problem.

Sorry for being miffed about this whole thing. I care a ton.
- Jacob Klein

Reference app_config.xml file:

skgiven:

Thank you for reporting your results.

When crunching 1 task, <cpu_usage> of 0.001 should be no different than the normal setup where GPUGRid tasks use 0.072 CPU by default and app_config.xml is not used. In both of those cases, my computer will allocate the same number of jobs to fill the available CPU resources. So, I do not think using <cpu_usage> of 0.001 can be the problem.

It's possible that just USING the app_config.xml file could be a problem, but I also believe that to be very unlikely, and am still testing to prove it.

It sounds like you might be suggesting that using a BOINC setting of 100% CPU, or using "extreme" app_config settings, could be a cause. While I don't believe that to be the case... Let's try to get more evidence! Please help.

Are you running with a BOINC setting of 100% CPU (so that your CPU resources are slightly over-committed?) If not, would you please consider changing your BOINC setting to use 100% CPU?

Also, are you running using the exact same app_config.xml file I posted 2 posts ago? If not, would you please consider changing your app_config.xml file to be exactly the same as the one I posted 2 posts ago?

The goal should be for you to attempt to reproduce the problem, so we can identify its cause. And, if you believe it's a "setup" issue, due to either the CPU % setting or the app_config.xml file setting, then TEST IT! Use my CPU % setting, and my app_config.xml file, on your machine -- and see what your results are! Note: It may take a week to get the problem to surface, from my experience. So, even if you make the changes and restart BOINC, it's a waiting game to see the results.

I'm betting, and hoping, that:
- you change your settings to 100% CPU
- you change your app_config.xml file to use my exact same app_config.xml file
- you restart BOINC
- you run it for a week
- you still aren't able to reproduce the problem, even after that week.
.... I want this problem to be related to homogenous GPUs.

Will you please help, by performing those tests?

Most importantly, I want GPUGrid.net admins to further acknowledge the problem, to put forth effort into identifying its cause (including useful statements in stderr.txt), to fix the application, and to fix the validator. One can only hope that these things become a priority.

Thanks,
Jacob

Thanks Jacob for explaining everything for me all in one post, sometimes I'm a little slow, it makes much more sense to me now. I appreciate you're stubborn determination in getting to the bottom of these anomalies, I just hope some other contributors aren't taking advantage of this flaw in the software.

skgiven:

Thank you for reporting your results.

When crunching 1 task, <cpu_usage> of 0.001 should be no different than the normal setup where GPUGRid tasks use 0.072 CPU by default and app_config.xml is not used. In both of those cases, my computer will allocate the same number of jobs to fill the available CPU resources. So, I do not think using <cpu_usage> of 0.001 can be the problem.
Thanks,
Jacob

Thanks for the suggestion. I highly doubt that adjusting the <cpu_usage> within the app_config.xml file would trigger the problem, but it is on my list of things to test.

Right now, I'm trying to reproduce the issue without an app_config.xml file. For some reason, I haven't been able to get that to happen yet.

Other tests I will do include:
- Test with app_config set for 1.000 CPU
- Test with BOINC set to use less CPU %
- Test with only 1 GPU in the system

It's not good enough to test for an hour. Sometimes it takes several days for this issue to happen. So, I'd say each test should take about 2 weeks, to "be sure" of the results. I guess I have to be patient and diligent with my testing.

If anyone else can replicate the problem (or at least try very hard to), I would very much appreciate any information you find.

Thanks,
Jacob

It's more likely to happen when a workunit starts. CPU usage seems to peak at that point, with a setting limiting the cpu to .0001, that's probably where it is occasionally choking.

It may also depend on the type of GPUs. I run a 660Ti and a 560Ti. the 660Ti uses as much CPU time as GPU time, whereas the 560Ti uses about 1/10th.

Even though the cpu is not saturated by any means, I have to keep 2 cores free to keep GPU usage steady instead of bouncing around.
and that only works with nothing else using the rig. If were to start web browsing etc, GPU usage starts jumping up and down again.

Some of you still do not understand something very basic.

All of these 3 scenarios result in the exact same CPU saturation/allocation under my system's setup:
- Not using an app_config.xml file (so GPUGrid tasks use their default CPU usage usually around 0.73 CPU)
- Setting <CPU_Usage> to 0.999 via app_config.xml file
- Setting <CPU_Usage> to 0.001 via app_config.xml file

In all 3 of these scenarios, my system will run 1 GPUGrid task on the GTX 660 Ti, 2 WCG tasks on my GTX460, and 6 additional CPU tasks. And in all of these scenarios, the GPUGrid task is granted the exact same "amount of CPU"; ie: just because it says "0.001" CPU, that does not mean it is somehow "limiting" the CPU usage. It's just a number that's used to calculate how many total tasks to run right now. That's all. You can do research on how CPUs work, and how BOINC works, to confirm this.

Regarding CPU overloading, I disagree about the necessity to reserve cores. From my observations using ProcessExplorer (Google it and use it for yourself!), GPUGrid tasks are setup to run at a process priority of 6, with an active-thread priority of 6 or 7. These priorities are higher than regular CPU tasks, meaning that GPUGrid tasks are never starved of the CPU. Now, if you browse a webpage or move a window, then that requires GPU, and the GPU usage may fluctuate... but in terms of competing with other CPU tasks, GPUGrid tasks are never starved of CPU, because GPUGrid tasks have higher priority. You can do research into Windows priorities, and watch Task Manager's CPU Usage, and use ProcessExplorer, to confirm all of this information for yourself.

Regarding the differences between types of GPUs, the applications appear to be setup to treat different GPU architectures differently. A GTX 660 Ti appears to be setup to request to use a full core all of the time (I believe this is equivalent to the now-deprecated "Swan_Sync" "0" setting). A GTX 460, however, because it is a different architecture, appears to be setup to only request to use a small portion of a core. This is all normal behavior, and is in fact probably a reason that the "default CPU Usage" of the GPUGrid tasks is a value less than 1 (the GPUGrid admins do not want to reserve a whole core unnecessarily, which would waste CPU resources)

I have not ruled out that overloading the CPU could be a causal factor in my problem, but based on my understanding of CPUs and processes, I consider it quite unlikely. One of the tests to prove this would be to control it by decreasing BOINC's "use at most x% of the processors" setting, or to control it by setting <cpu_usage> to 1.000 in the app_config.xml file; both of these are tests on my to-do list from 2 posts ago. But currently I'm still trying to get the issue to occur without an app_config.xml file.

Finally, if you believe I'm wrong in any of this, or if you believe you have some other "answer", I encourage you to use the scientific method and PROVE it. Perform the setting changes necessary to recreate the problem. Seriously. Test, test, test! I challenge you to find the cause of the problem; I would appreciate any and all help testing. And, when you've tested for a sufficient enough time to prove something, report your results!

Guessing at the problem is no-longer helpful to me.

Thank you,
Jacob

I just had a task fail, without an app_config.xml file... but it actually failed (instead of being instant-done completed successfully with credit).

So, I'm a bit confused.
Nathan, did you change anything?

The details are below... they include an Exit Status of -1, a Client state of Compute error, and no credit granted. (ie: If it had to fail, it at least failed gracefully, without being marked successful and without granting credit).

Note: I was having internet connectivity issues during the time that it started and promptly failed. Don't know if that is a causal factor or not.

looking at that taskhttp://www.gpugrid.net/workunit.php?wuid=4368491 it has failed 8 times, I had 1 like that yesterday too, so I'd say it was a 'normal' failed task, rather than anything else,

Simba, I agree that the task that had the computational error, which failed for several other users, is unrelated to my research and testing. Thanks for pointing that out -- I should have spotted that earlier.

Also, just a note for the thread, I am now testing with nVidia Beta v320.00 drivers.

Jacob

I've had another workunit that was Completed Successfully in a really short time. Looking at the logs, it ran on my GTX 660 Ti, while the GTX 460 was busy with 2 WCG tasks. It happened while I was using an app_config.xml file that had:
- 9999 max_concurrent
- 0.001 cpu_usage
- 1.000 gpu_usage

This frustrates me, because so far, I've only been able to reproduce the problem when I use an app_config file with those settings. But I still really believe that using an app_config.xml file shouldn't be causing the problem.

So, again, to continue to test my theory, I've reset the project, and will be running without an app_config.xml file for a while. I'll chime in again, in about 2 more weeks, to report more results.

Task details:

I've been running it for a few days by now, no anomalies.

Except that the claimed CPU utilization of 0.001 results in my BOINC 7.0.44 starting 8 Einstein threads on my 8-threaded i7+HT. You said in your case you'd get 7 CPU tasks, whether you use the app_config or not.

Running 8 Einsteins still makes my GTX660Ti use about a full core, but GPU utilization is fluctuating between 86 and 90%, whereas it stays at a nice and steady 92% without any CPU tasks. From quickly experimenting it seems like 3 or 4 CPU tasks would be enough to get full GPU-grid performance.

MrS

WCG's HCC on GPU WU's began by using a full CPU, then used the GPU (high GPU utilization), and then used a full CPU thread again. When the CPU was being used the GPU was not and when the GPU was being used the CPU was not.

In this situation, you could therefore be running two HCC tasks without using any CPU (depending on how far along the tasks were) or be fully using two CPU threads.

In my opinion when using the app_config and specifying that GPUGrid tasks use 0.001 CPU's this meant that when some GPUGrid tasks started two CPU threads would be allocated and be fully used by HCC tasks. Along with 7 CPU tasks this meant that at that critical time for a GPUGrid task (the start), the CPU was already saturated and Boinc was being told (by the app_config file) that GPUGrid apps hardly needed any CPU (0.001). They failed as a result of CPU starvation.

They were incorrectly granted credit because the system didn't catch the error, but app_config is new and the server hasn't been equipped to catch such unknown errors. While this might be a problem for the researchers, perhaps future server updates will help protect against such problems.

WCG's HCC on GPU WU's began by using a full CPU, then used the GPU (high GPU utilization), and then used a full CPU thread again. When the CPU was being used the GPU was not and when the GPU was being used the CPU was not.

skgiven,

Your claim is that the CPU is somehow being "overloaded", and that using 0.001 CPU_Usage in an app_config.xml file could be the cause. I don't believe that's possible, but let's have you test it!

So, a suitable test (for you to do) would be:
- Tell BOINC to "use at most 100% of the processors"
- Close BOINC
- Add an app_config.xml file to your GPUGrid.net project directory, and make it look like the one referenced here: http://www.gpugrid.net/forum_thread.php?id=3332&nowrap=true#29520
- Restart BOINC
- Confirm that any running GPUGrid tasks show up as "Running (0.001 CPU + 1 NVIDIA GPU (device x))"
- Confirm that the CPU is either fully saturated, or over-saturated. The more saturation, the better for our test!
- Run that for 3 weeks, monitoring your GPUGrid.net task results
- Let me know if any results have really low run times but are still completed successfully, validated, and granted credit

Does that sound suitable?

skgiven,

I thought you were serious about wanting to help test. But it sounds like you don't want to perform the test I recommended. That's unfortunate.

But I'm trying to channel my frustrations into solving this. It's taken me 1 month thus far, with very little guidance/help from the admins. I feel I've made some progress, and if it takes 9 more weeks, or longer, to solve it, then so be it.

You're welcome to help out any time. But it will involve stepping outside your comfort zone.

Regards,
Jacob

PS: Thanks for the advice about the logging flags.

For reference, I believe the BOINC scheduling rule is (and has been for a long time):
- While CPU_Used <= (#CPU+1), schedule GPU tasks to run
- While CPU_Used < (#CPU+1), schedule CPU tasks to run
- If you have any high-priority tasks, they get scheduled first

So, if you're running 1 GPUGrid Task, on an 8-CPU machine, with BOINC set to use 100% CPUs, and the GPUGrid task's CPU_Usage is < 1, then I would expect you to also be running 8 CPU tasks, and it is normal.

Also, you should be running BOINC v7.0.64, which was recently publicly released.

Kind Regards,
Jacob

Fine so long as the project is happy with people using it to test and debug Boinc code.

My present concern is that there are important setup differences, I don't have a 3GB card or a GTX460 to test with, and it might be important to crunch the same WU's on the second GPU (and possibly the same CPU WU's). Can we agree on something here?

That's nice, but I got an app crash within 5min of running over committed. There is no point running with such a setup if you get lots of other failures, it defeats the purpose.

Is it not the case that without app_config you have 7CPU tasks and 1 GPU task running, but when using app_config it's 8?

Jacob, the version of Boinc you were using when you first encountered the credit rich 'errors' was pre-70.0.64, probably .58, .59 or .60.
The released date of .64 was 16-Apr, so the issue seems to span several release versions. While I agree it's best to test on a common Boinc version, I don't think it's essential. I've used 7.0.60 and 7.0.64 and possibly 7.0.62 (albeit for only a few days) - not that I've manages to replicate the problem. Do you think the operating system or GPU types in the system would make any difference in determining if the problem is CPU or app_config related?
If not I could test for a while on a different system (XP and a GTX470).

Running with Jacob's app_config.
I exited Boinc and removed the app_config file from the GPUGrid project folder.
Start Boinc, and click on the GPUGrid WU Properties. It says using 0.001 CPU's!
It should say what it's actually using which is 1 CPU.
Even after a system restart, it's still saying that the WU's using 0.001 CPU's.

The WU is clearly using a full CPU, but saying that it's using 0.001. That could only have come from the app_config file - which has now been removed. So why does Boinc still think it's using 0.001 CPU's?

When I don't use app_config and when I have the CPU set to be used 100% by Boinc apps I run 7 CPU WU's and one GPUGrid WU on the GTX660Ti system (ATI not in use).