Hi Matt,

I have set in my cc_config to not do any benchmarks, by a line posted by someone on the forum. Secondly I have all my rigs at 100% all the time, and do not use TTrottle to lower temperature. This is a nice program, it lowers CPU and GPU use as it rises above a set temperature value by the user.

So for me you might disable all features that suspend or hamper GPUGRID calculation.
Even the down clocking of the GPU clock as a WU has failed or terminated to prevent, if this is something in your power to do so off course.

I think that people who are a bit concerned about heat and warmth build up in the system and have most of the time expensive graphics cards, could have a different view of it.
____________
Greetings from TJ

I would, reluctantly, support that approach as a temporary stop-gap only.

I say reluctantly, because:

1) In principle, I support permanent diagnosis and robust fixes. Other GPU apps that I run, from other projects, don't suffer unrecoverable errors as a result of suspension. To be fair, that could simply be because other apps don't run as long as GPUGrid - you require longer-term stability than anyone else. And similarly, there are fewer reports of slow running at other projects, again perhaps because it isn't so noticeable on shorter tasks.

2) I suspect this has come to light because a stupid feature was, mistakenly in my view, included in the v7.0.64 client (sorry, we didn't catch it in time), and that client has remained 'recommended' for far longer than expected. Once that aberration has been corrected, I'd be happier if the requisite workround was removed too - so we end up with the design working as intended (CPU throttling allowed, without disrupting CPU work). We do have to watch out for that '0.5 CPU' definition of coprocessor work, though.

Even the down clocking of the GPU clock as a WU has failed or terminated to prevent, if this is something in your power to do so off course.

I agree with you Richard, however this down clocking emerged frequently at my rig with the 660 since the introduction of the "termination to prevent hang up". I am not sure that it has to do with that, but it seems so. It is now every other day I have to boot my rig to solve this, and I am not always there to do so. But we are going of topic, so I'll stop about it.
____________
Greetings from TJ

I understand that the downclocking that TJ is concerned about (which is permanent, until the next host reboot) is likely to occur if BOINC suspends the app without allowing time for what is known as a 'threadsafe' exit.

That's the reason for the note about Do GPU kernels within critical sections in the BOINC co-processor programming guide.

I understand that the downclocking that TJ is concerned about (which is permanent, until the next host reboot) is likely to occur if BOINC suspends the app without allowing time for what is known as a 'threadsafe' exit.

With the current application this should no longer be occuring. The application defers suspension and termination until a safe point. The Boinc library is no longer able to asynchronously suspend the process, since that was a significant cause of instability.

Matt

You should not ignore the request to suspend/resume, when benchmarking or when being CPU-Throttled.

If there was a client bug that was fixed, so be it.
If there still exists a client bug, we should find out about it and fix it

If any app, even a GPU app, is told to suspend, for whatever reason (including CPU throttling), it needs to suspend.
If the user wants to ensure no CPU throttling, then the user needs to change the setting to ensure no CPU throttling.
If the user specified CPU Throttling, then tasks will be and should be throttled.
Simple as that.

That's my opinion.

You should not ignore the request to suspend/resume, when benchmarking or when being CPU-Throttled.

If there was a client bug that was fixed, so be it.
If there still exists a client bug, we should find out about it and fix it

If any app, even a GPU app, is told to suspend, for whatever reason (including CPU throttling), it needs to suspend.
If the user wants to ensure no CPU throttling, then the user needs to change the setting to ensure no CPU throttling.
If the user specified CPU Throttling, then tasks will be and should be throttled.
Simple as that.

That's my opinion.

I'd also like to add another opinion:

I've heard of the following change to BOINC 7.2.1+:
client: don't apply CPU throttling to apps that use < .5 CPUs (like GPU, NCI).

Is it likely that GPUGrid tasks will show as using (0.5 CPUs) or more?
If so, then the application will receive suspend/resume events even in 7.2.x.

And if it has trouble responding to those events (trouble like task errors, or task hangs)... Then the problem is in the application, not in the client. And it should be solved (by making it not error/hang), not worked around (ie: ignore suspend/resume events).

Suspend/resume events need to be supported, even for GPU applications.

My TITAN uses 0.827 CPU and is subject to this issue, and I have others that exceed 0.50 CPU. So upgrading the BOINC client will not work here.

No other computer running GPUGRID is subject to this exception and I have always had CPU time usage at 100% and using on multiprocessor systems using 100% and I have never throttled back for any computer.

While I have employed a "work-around," I would prefer this issue be resolved as I have also recently experienced numerous computer and driver crashes. No other computer running GPUGRID has experienced this frequency of crashes, except this one. I am using driver 327.33 for this computer and the others are still using 320.49. The only third party utility I am using is the EVGA Precision X only to control the fan speeds, not for manual overclocking purposes.

Please note this situation does not occur with PrimeGrid or Collatz. Then again, their CPU resource requirement is below 0.10.

Thank you.

Raymond

My TITAN uses 0.827 CPU and is subject to this issue, and I have others that exceed 0.50 CPU. So upgrading the BOINC client will not work here.

I expect the 82.7% is just what's being reported in Boinc Task Properties

If a non-Noelia task uses significantly less than a full CPU (as in <95%) then it suggests there might be a problem - the CPU is being overused (by other apps), the CPU is not capable of supporting the GPU fully (architecture or setup), or some other hardware is limiting the GPU (PCIE, RAM speeds, clogged up drive)

There are four processor usage Boinc settings that could cause the GPU to suspend and resume frequently:

Maybe using some UPS's and having 'While computer is on batteries' unselected could cause issues, or if a laptop battery is misreporting its connection state (OS or OS driver bugs)

In my experience some default settings and recommended settings at other projects are unsuitable for crunching at GPUGrid. It would be useful if there was some sort of way to automatic select recommended settings from your online account (or a Project Button in BM).
It would also be useful if when you allow others to see your systems, they can also see your Boinc configurations, or at least an option for moderators to see these.

There are four processor usage Boinc settings that could cause the GPU to suspend and resume frequently:

Computing allowed,

• While computer is in use (when not ticked)
  
• Use GPU while computer is in use (when not Ticked)
  
• While processor usage is less than (anything other than zero)
  
• Use at most % CPU time (anything other than 100%)
  

Might Boinc 7.0.28 be an issue?

Raymond, I agree with the suggestions made by MrS and reducing the power target is certainly worth a go.

Your issue is most likely with the card, Power supply, cooling when running the GPUGRid app, but there are some other possibilities.

This behavior occurs with Asteriods@home, PrimaBonica or Fight Malaria@home are the CPU projects. This is a dedicated cruncher with no other work being processed.

Any ideas are appreciated. Thank you.

Does this 'start-stop' behavior occur when you are not using the CPU, or when you run different CPU projects (than those you mention)?

A 12 thread processor that's maxed out could conceivably struggle to feed a Titan. Have you tried setting Boinc to use 10 threads (83.4%)?

When your CPU is fully used, the draw on the PSU is probably another 100W or so.
If it's not the power draw itself then it could be the heat generated from it. Does the heat radiate into the case, or are you using an external water cooler?

To test if power/overuse of the CPU is the issue, stop running CPU tasks, and just run GPU tasks.

To test if heat is the issue, take the side of the case off and run a WU or two.

Might Boinc 7.0.28 be an issue?

Raymond, I agree with the suggestions made by MrS and reducing the power target is certainly worth a go.

Your issue is most likely with the card, Power supply, cooling when running the GPUGRid app, but there are some other possibilities.

When your CPU is fully used, the draw on the PSU is probably another 100W or so. If it's not the power draw itself then it could be the heat generated from it. Does the heat radiate into the case, or are you using an external water cooler?

Sounds you should be pretty safe, hardware-wise. But, reading a few posts back, I'm not sure which error we're actually talking about here. Could you summarize the "start and stop" problem? Are you seeing that "client suspended (user request)" message often in your tasks and they take quite long, about twice as long as the CPU time they required?

MrS

It's been discovered in a news thread that 331.40 beta fixes the access violations. Give it a try!

MrS

If you find nVidia bugs (like cannot use nVidia Control Panel when no monitor is connected, or performance degradation occurs on Adaptive Performance even when under full 3d load from a CUDA task)... Please report your issues in the nVidia driver feedback thread on their forums! Sometimes they are responsive.

Skgiven I think that what we see in the stderr file is the boost value of the GPU and not the actual value or the value it ran at.
My 660 is set to 1045MHz, while the stderr shows 1110MHz, even when it is down clocked.
I have now switched the 660 to LR only and it is running for 6 days and 9 hours without down clocking! Some of these WU´s ran even without interruption by "The simulation has become unstable. Terminating to avoid lock-up (1)".

I saw my down clocking especially with SR and beta´s, but I am not checking every WU that closely.
I have already downloaded the 331.4 beta driver and will install it next time I need to reboot the system (it runs fine now, so I leave it).
____________
Greetings from TJ

It always seems to be what GPU-Z displays as the Default Clock Boost. Not any over/underclock done with EVGA Precision or actual boost. The GPU-Z pop up description simply states: "Shows the default turbo boost clock frequency of the GPU without any overclocking."

Edit....This is for the Boost 680 for me.

The 460 card does not show the default clock under GPU-Z, but rather the GPU Clock.

The code to pull clock speed seems to work different between the two cards.

For me, for my eVGA GTX 660 Ti 3GB FTW (factory overclocked), I use a custom GPU Fan profile (in Precision-X), so while working on GPUGrid, the temperatures almost always drive the fan to maximum fan (80%). The clock is usually at 1241 MHz, and the Power % is usually around 90-100%. But there are times where the temperature still gets above 72*C, in which case the clock goes down to 1228 MHz. Also, there are times when a task really stresses the GPU (such that it would normally downclock to not exceed the 100% power target, even if temps are below 70*C), so in Precision-X, I have set the power target to its maximum, 140%. So, in times of considerable stress, I do sometimes see it running at 1241 MHz, at ~105% Power, while under 72*C. Regarding reported clock speeds, even though the GPU is usually at maximum boost (1241 MHz), I believe GPUGrid task results always reports 1124 MHz (which I believe may be the "default boost" for a "regular" GTX 660 Ti), which is fine by me.

Here are the GPU-Z values from my eVGA GTX 660 Ti:

GPU Clock: Shows the currently selected performance GPU clock speed of this device.
1046 MHz

Memory Clock: Shows the currently selected performance memory clock speed of this device.
1502 MHz

Boost Clock: Shows the typical turbo boost clock frequency of the GPU.
1124 MHz

Default GPU Clock: Shows the default GPU clock speed of this device without any overclocking.
1046 MHz

Default Memory Clock: Shows the default memory clock speed of this device without any overclocking.
1502 MHz

Default Boost Clock: Shows the default turbo boost clock frequency of the GPU without any overclocking.
1124 MHz

Below outputs from GPU-Z as compared to the stderr output.

GTX460
<stderr_txt>
# Device clock : 1760MHz
# Memory clock : 1850MHz

GPU-Z
Clock....Default / GPU: 760 / 880
Memory...Default / GPU: 900 / 925
Shader...Default / GPU: 1520 / 1760


GTX680
<stderr_txt>
# Device clock : 1124MHz
# Memory clock : 3200MHz

GPU-Z
Clock....Default / GPU: 1059 / 1085
Memory...Default / GPU: 1552 / 1600
Boost....Default / GPU: 1124 / 1150 (Boost is actually running 1201 ~90% of time)

Probably belongs in wish list, but having a reading added to the stderr output line of the GPU clock that GPU-Z pulls for boost added to each temp line. Then have Device clock be the average of those readings. Of course cross linked to the task results so it could be pulled in a view when trying to runs stats on the cards. :)