Saying none available :(

Short run queue is also dry.

8/27/2013 11:56:14 AM | GPUGRID | Scheduler request completed: got 0 new tasks
8/27/2013 11:56:14 AM | GPUGRID | No tasks sent
8/27/2013 11:56:14 AM | GPUGRID | No tasks are available for Short runs (2-3 hours on fastest card)
8/27/2013 11:56:14 AM | GPUGRID | No tasks are available for Long runs (8-12 hours on fastest card)

Unfortunately I do not get new work on my GTX 670 (2048MB) driver: 311.6, although you mention that the application will assign the Cuda version correspondingly.

I do get the following message on the boinc log:
27/08/2013 11:51:53 a.m. | GPUGRID | Scheduler request completed: got 0 new tasks
27/08/2013 11:51:53 a.m. | GPUGRID | NVIDIA GPU: Actualiza al último driver para usar todas las aplicaciones de GPU de este proyecto
27/08/2013 11:51:53 a.m. | GPUGRID | No tasks sent
27/08/2013 11:51:53 a.m. | GPUGRID | No tasks are available for Long runs (8-12 hours on fastest card)
Although the server status page shows, there are some long WUs available.
I personally do not like to change drivers as my systems are running without flaws for months, until it is indispensable, stable and makes the video cards faster.

Are you producing new WUs at this very moment? I am asking as there had been quite a lot (around 2000) a few days ago, but as somebody else noted they desapiered very fast some days ago.

I am beyond impressed

30k seconds for a long run nathan_kid on 680

# Time per step (avg over 5000000 steps): 6.104 ms
# Approximate elapsed time for entire WU: 30518.949 s

20k seconds for same task type on a 780

# Time per step (avg over 5000000 steps): 4.105 ms
# Approximate elapsed time for entire WU: 20526.378 s

Not getting any work units, installed lates drivers:
8/27/2013 1:25:58 PM | | CUDA: NVIDIA GPU 0: GeForce GTX 690 (driver version 320.78, CUDA version 5.50, compute capability 3.0, 2048MB, 1955MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | CUDA: NVIDIA GPU 1: GeForce GTX 690 (driver version 320.78, CUDA version 5.50, compute capability 3.0, 2048MB, 1926MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | CUDA: NVIDIA GPU 2: GeForce GTX 690 (driver version 320.78, CUDA version 5.50, compute capability 3.0, 2048MB, 1955MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | CUDA: NVIDIA GPU 3: GeForce GTX 690 (driver version 320.78, CUDA version 5.50, compute capability 3.0, 2048MB, 1955MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | OpenCL: NVIDIA GPU 0: GeForce GTX 690 (driver version 320.78, device version OpenCL 1.1 CUDA, 2048MB, 1955MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | OpenCL: NVIDIA GPU 1: GeForce GTX 690 (driver version 320.78, device version OpenCL 1.1 CUDA, 2048MB, 1926MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | OpenCL: NVIDIA GPU 2: GeForce GTX 690 (driver version 320.78, device version OpenCL 1.1 CUDA, 2048MB, 1955MB available, 3132 GFLOPS peak)
8/27/2013 1:25:58 PM | | OpenCL: NVIDIA GPU 3: GeForce GTX 690 (driver version 320.78, device version OpenCL 1.1 CUDA, 2048MB, 1955MB available, 3132 GFLOPS peak)


BUT AM GETTING THIS:
8/27/2013 10:10:19 PM | GPUGRID | Sending scheduler request: To fetch work.
8/27/2013 10:10:19 PM | GPUGRID | Requesting new tasks for NVIDIA
8/27/2013 10:10:21 PM | GPUGRID | Scheduler request completed: got 0 new tasks
8/27/2013 10:10:21 PM | GPUGRID | NVIDIA GPU: Upgrade to the latest driver to use all of this project's GPU applications
8/27/2013 10:10:21 PM | GPUGRID | No tasks sent
8/27/2013 10:10:21 PM | GPUGRID | No tasks are available for Long runs (8-12 hours on fastest card)
____________

Not getting any work units, installed lates drivers:
8/27/2013 1:25:58 PM | | CUDA: NVIDIA GPU 0: GeForce GTX 690 (driver version 320.78, CUDA version 5.50, compute capability 3.0, 2048MB, 1955MB available, 3132 GFLOPS peak)....


BUT AM GETTING THIS:

8/27/2013 10:10:21 PM | GPUGRID | NVIDIA GPU: Upgrade to the latest driver to use all of this project's GPU applications....

Operator, please post your completed time on the long run for you titan when finished.

Cheers

Just updated to 320.49, the latest Nvidia driver for my 680. Boinc is telling me I still need to update my drivers in order to receive work.

I see Operator posted a link to another driver. Is this the one we need in order to receive any of the new WUs? I don't have a lot of experience with these and am wondering how installing this will impact day to day uses of the computer (internet, movies, gaming).

Thanks for any info. I'd like to keep contributing as much as possible while still being to use my PC for other purposes.

Operator, please post your completed time on the long run for you titan when finished.

Cheers

Just updated to 320.49, the latest Nvidia driver for my 680. Boinc is telling me I still need to update my drivers in order to receive work.

I see Operator posted a link to another driver. Is this the one we need in order to receive any of the new WUs? I don't have a lot of experience with these and am wondering how installing this will impact day to day uses of the computer (internet, movies, gaming).

Thanks for any info. I'd like to keep contributing as much as possible while still being to use my PC for other purposes.

Just updated to 320.49, the latest Nvidia driver for my 680. Boinc is telling me I still need to update my drivers in order to receive work.

I see Operator posted a link to another driver. Is this the one we need in order to receive any of the new WUs? I don't have a lot of experience with these and am wondering how installing this will impact day to day uses of the computer (internet, movies, gaming).

Thanks for any info. I'd like to keep contributing as much as possible while still being to use my PC for other purposes.

First long unit GPUGRID completed by my Titan.

http://www.gpugrid.net/results.php?hostid=156585

The second is being,

https://www.dropbox.com/s/5s9xejf5ffit7o7/GpuGridUnitLong.jpg

@+
*_*
____________

We decided to go live with the new application on ACEMD-Long.

It is version 8.00, and the server will automatically assign you a CUDA 4.2 or 5.5 version, depending on your driver version.

Any problems on this thread, please.

MJH

There seems to be a small improvement in the new application but not in the Cuda version.
____________
Radio Caroline, the world's most famous offshore pirate radio station.
Great music since April 1964. Support Radio Caroline Team -
Radio Caroline

Hello guys and gals !
Still nothing to crunch for my GPUs ...

mer. 28 août 2013 12:49:09 CEST | GPUGRID | Sending scheduler request: To fetch work.
mer. 28 août 2013 12:49:09 CEST | GPUGRID | Requesting new tasks for NVIDIA
mer. 28 août 2013 12:49:11 CEST | GPUGRID | Scheduler request completed: got 0 new tasks
mer. 28 août 2013 12:49:11 CEST | GPUGRID | No tasks sent
mer. 28 août 2013 12:49:11 CEST | GPUGRID | No tasks are available for Short runs (2-3 hours on fastest card)
mer. 28 août 2013 12:49:11 CEST | GPUGRID | No tasks are available for Long runs (8-12 hours on fastest card)

____________
Lubuntu 16.04.1 LTS x64

There seems to be a small improvement in the new application but not in the Cuda version.

Now... I wish the server would give me more than just one WU at a time since I have TWO GPUs.

Operator

Odd error on a long run for my 780:

The simulation has become unstable. Terminating to avoid lock-up

All WUs are currently dry again as well.

Running dry of WUs in all machines. No long or short are beeing splited for sometime. As others say, it´s not a driver issue, despite the new message from server.

They are flowing now, thank you.

Ok, I'm getting wu's again, BUT I give up on what new app names to use in the app_config.xml file. Can anyone help me? Thanks in advance, Rick
____________

Operator, please post your completed time on the long run for you titan when finished.

Cheers

So about another 1k seconds shaved off.

Im really really impressed with these cards.

W7x64
Dell T3500 12GB
Titan x2 (both EVGA factory OC'd, and on air - limited to 80C)
1100hz (ballpark freq, not fixed) running 326.84 drivers from the developer site.

Hello Operator, you are the one I need to ask something.
What is the PSU in your Dell and how many GPU power plugs do you have? I have a T7400 with a 1000W PSU but only two 6 pins and 1 not usual 8 pin GPU power plug, so I am very limited with this big box.
Thanks for your answer highly appreciated.

There seems to be a small improvement in the new application but not in the Cuda version.

Well if you have a Titan or a 780 the apps now work! I think that was the point, at least it was for me (since March).

Now... I wish the server would give me more than just one WU at a time since I have TWO GPUs.

Operator

Getting the unknown error number crashes on kid WUs

More have crashed, with the exact same unknown error number. I dont know if this is from the 8.02 or whatever app that has been pushed out, but it only recently began doing this.

/sigh

Please can we go back to 8.00 (or maybe 8.01)?

8.02 makes all the Nathans error out now.

It got so messed up I just reset the project.

Was doing fine with the 8.00 on Titans.

Operator
____________

Getting the unknown error number crashes on kid WUs

I've not had any issues with 8.0, 8.1 or 8.2 on my 500 & 600 cards, either Linux or Win. The only problem now is getting fresh wu's from the server.

I've not had any issues with 8.0, 8.1 or 8.2 on my 500 & 600 cards, either Linux or Win. The only problem now is getting fresh wu's from the server.

Getting the unknown error number crashes on kid WUs

It isn't the error number which is unknown, it's the plain-English description for it.

Yours got a 0xffffffffc0000005: mine has just died with a 0xffffffffffffff9f, description equally unknown. Task 7221543.

Getting the unknown error number crashes on kid WUs

It isn't the error number which is unknown, it's the plain-English description for it.

Yours got a 0xffffffffc0000005: mine has just died with a 0xffffffffffffff9f, description equally unknown. Task 7221543.

Ty for the clarification.

Noelia has over 1000 WU to submit but we cannot as there are still these problems.

Now they are running on beta to test.

gdf

Using the 803-55 application now (on Titans) I received two NATHAN KIDKIX WUs

And both took almost 9k more seconds to complete than with previous 800-55 version.

Still using the 326.84 drivers.

I don't think this 803 is actually comparable to the 800. I think something changed, and not in a good way.

http://www.gpugrid.net/result.php?resultid=7224469

http://www.gpugrid.net/result.php?resultid=7224304

I've also received at least one Nathan baxbimx and a NOELIA KLEBE that I had to abort because they were going no where. Just stuck.

Operator
____________

803 and 800 are the exact same binary.

MJH

8.04 is a new beta app. Includes a bit more debugging to help me find the cause of the remaining common failure modes.

MJH

As Stderr output reports:
<core_client_version>7.0.28</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
]]>
I wanted to inform that the WU "I74R6-NATHAN_KIDKIXc22_6-6-50-RND6702_0" Version 8.03 caused a driver crash and several blue screens until I was able to get rid of it.

The following task: "I83R9-NATHAN_baxbimx-4-6-RND5261_0" short queue seems to go to nowhere, I am pretty sure it is also a blue screen tomorrow morning when I get to the office.

I would not say this video card does not crash at times, but this was the first time that I was not able to simply restart the computer and download a new task.

EDIT my previous post: Just not make false accusations: the task "I83R9-NATHAN_baxbimx-4-6-RND5261_0" did not run until today because I forgot to quite the suspension of BOINC MANAGER. It is running fine now.

Hi!
I got lot of this with 8.14 shutdown/restarts
This did not happend with 800 /803
http://www.gpugrid.net/result.php?resultid=7266298

# GPU [GeForce GTX 780] Platform [Windows] Rev [3203] VERSION [55]
# SWAN Device 0 :
# Name : GeForce GTX 780
# ECC : Disabled
# Global mem : 3072MB
# Capability : 3.5
# PCI ID : 0000:01:00.0
# Device clock : 954MHz
# Memory clock : 3024MHz
# Memory width : 384bit
# Driver version : r325_00 : 32680
# GPU 0 : 62C
# GPU 1 : 58C
# GPU 0 : 63C
# GPU 0 : 64C
# GPU 0 : 65C
# GPU 0 : 66C
# GPU 0 : 67C
# Access violation : progress made, try to restart]

While I wasnt getting them with 803, I havent had a crash yet on my 780s.

so while it does slow the task down a little bit. Thats better than crashing the task to me.

Hi, MJH:

I am getting short and long tasks for CUDA 4.2. My driver on the 650 Ti GPUs is 314.22 as I had too many failures with driver versions 320.57 and 320.18. What is the difference between CUDA 4.2 and 5.5 and do I need to do anything more to successfully process GPUGRID tasks in an optimum way? Despite wanting to, I cannot buy the higher performing video cards.

Thanks,

John

Hi Matt,

I had 50% error with your previous CRASH tests, the Santi´s on my 660 with the -97 error you wanted to see lots of. Well have a look if you have time.

Since the last 3 days I some times see this in the output file:

Since the last 3 days I some times see this in the output file:

# BOINC suspending at user request (exit)

However I have done nothing, I was asleep, so the rig was unattended.

Hi Matt,

I had 50% error with your previous CRASH tests, the Santi´s on my 660 with the -97 error you wanted to see lots of. Well have a look if you have time.

Since the last 3 days I some times see this in the output file:

# BOINC suspending at user request (exit)

However I have done nothing, I was asleep, so the rig was unattended.
It is there when a WU failed, but also with a success as in in the last Nathan LR. On the 770 everything works perfect.
Perhaps the latest derives are optimized for the 7xx series by nVidia and not so good for the 6xx series of cards.

Since the last 3 days I some times see this in the output file:

# BOINC suspending at user request (exit)

However I have done nothing, I was asleep, so the rig was unattended.

Sometimes benchmarks automatically run, and GPU work gets temporarily suspended. Also, sometimes high-priority CPU jobs step in, and GPU work gets temporarily suspended.

You can always look in Event Viewer, or the stdoutdae.txt file, for more information.

Hi, MJH:

I am getting short and long tasks for CUDA 4.2. My driver on the 650 Ti GPUs is 314.22 as I had too many failures with driver versions 320.57 and 320.18. What is the difference between CUDA 4.2 and 5.5 and do I need to do anything more to successfully process GPUGRID tasks in an optimum way? Despite wanting to, I cannot buy the higher performing video cards.

Thanks,

John

Many thanks, Operator!!

John

John,

As Operator suggests, the 326.80 drivers are working very well on my GTX 650 Ti also (1 GB). The only errors have been on the betas.
http://www.gpugrid.net/results.php?userid=90514&offset=0&show_names=1&state=0&appid=

I don't know what happened here but all of my tasks are not running. They start run for a second or two, get a fraction of a percentile done then just stop utilizing the video card. The timer in boinc is running but no progress is being done and I can see they are not hitting the cards for any work.

This just happened out of the blue that I can see, I do have the latest Nvidia drivers I loaded in a week or so ago and I don't recall any updates that might have caused any problems.

I tried restarting my computer / boinc but no help.

Any ideas?
Thank you
Aaron

Hi, Jim:

Good to know: I will update the drivers in the next few days.

Keep on crunching....

John

What is the difference between CUDA 4.2 and 5.5

Same code base, compiled with two different compiler versions.
4.2 is very old now and we need to move to 5.5 to stay current and be ready for future hardware.
(We had to skip 5.0 because of unresolved performance problems)

Matt

Then I will stay with driver: 311.6 and CUDA 4.2. Works great for me!

Is 5.5 supposed to be any faster? I didn´t notice any difference in crunching speed.

Nope.

M

TJ - that message probably doesn-t indicate that the client is running benchmarks unless you have a version or configuration I've not seen. For benchmarking the message is along the lines of '(thread suspended)' not '(exit)'.

Looks like something your computer is doing is causing the client to suspend the work. High priority tasks from other projects? Antivirus scans? Cat walking on the keyboard?

Matt

Tools, Computing Preferences, Processor usage tab,
While processor usage is less than - set it to Zero.

MJH;

Running 8.14 long on my Titan box the last few days has been weird.

Not anywhere near the temp limits I have set (running at around 60+ instead of 81-82), and tasks are taking a lot longer than before, almost twice the time.

And there a lot of this nonsense:

NC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)
# BOINC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)
# BOINC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)
# BOINC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)
# BOINC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)
# BOINC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)
# BOINC suspending at user request (thread suspend)
# BOINC resuming at user request (thread suspend)

It's no wonder it takes so long.

And believe me there is nothing else this system is doing besides GPUGrid.

I didn't change anything...so what I'm asking is what did you folks change that is causing this?

Is it just those NATHAN_KIDc tasks?

Operator
____________

Event Log may also give more insight into why the client is suspending/resuming. I don't think it's a client error, but rather user configuration that is making it do so.

Does Event Log give you any hint?

MJH;

I'm with you. I got it. We're on the same page.

Question is why this is doing this using 8.14 on NATHAN_KIDc and NOELIA longs on both my Titan and my GTX590 boxes. I don't see this behavior on my GTX 650Ti box though.

Both the Titan and GTX590 boxes are now seemingly taking almost if not exactly twice the normal amount of time to process a NATHAN/NOELIA (and possibly newer other) WUs.

Please take a look at:
http://www.gpugrid.net/results.php?hostid=147455

and;

http://www.gpugrid.net/results.php?hostid=152263

to see what I'm referring to.

Once I started processing with 8.14 instead of 8.03, that's when all the suspend/resume shenanigans started.

And we can't have shenanigans, now can we? ;-}

Operator
____________

Operator,

Your 590 machine looks fine - it seems to me that it's just your Titan machine that's affected.
What does the client GUI say about the task states?

Matt

And that's in addition to the "thread suspend"ing ? Can you say how frequently that is happening (watch the output to the stderr file as the app is running). Does the task state that the client reports (running, suspend, etc) match, or is it always showing the task as running ,even when you can see that it has suspended?

Could you try running just a single task, and see whether that behaves itself. Use the app_config setting Beyond advises, here:
[quote
http://www.gpugrid.net/forum_thread.php?id=3473&nowrap=true#32913

Operator, what setting do you have in place for, Switch Between Applications Every (time)?
- Boinc Manager (Advanced View), Tools, Computing Preferences, Other Options.

I use 990min, but I think the default is 60min (which means Boinc will run one app then another...).

My 780 completes them fine. But has constant access violations.

Operator, what setting do you have in place for, Switch Between Applications Every (time)?
- Boinc Manager (Advanced View), Tools, Computing Preferences, Other Options.

I use 990min, but I think the default is 60min (which means Boinc will run one app then another...).

I use a higher setting too but this should only apply when running more than 1 project. If it's switching between WUs of the same project it's a BOINC bug (heaven forbid) and should be reported.

When an internal problem occurs, the new v8.14 application exits completely, and as far as BOINC knows, the GPU is free and available to schedule another task. If you have another task - from this or any other project - ready to start, BOINC will start it.

On my system, with (still) a stupidly high DCF, the sequence is:
Task 1 errors and exits
Task 2 starts on the vacant GPU
Task 1 becomes ready ('waiting to run') again.
BOINC notices a deadline miss looming
BOINC pre-empts Task 2, and restarts Task 1, marking it 'high priority'

That results in Task 2 showing 'Waiting to run', with a minute or two of runtime completed, before Task 1 finally completes.

With more normal estimates and no EDF, Task 1 and Task 2 would swap places every time a fault and temporary exit occurred. Even if you run minimal cache and don't normally fetch the next task until shortly before it is needed, you may find that a work fetch is triggered by the first error and temporary exit, and the swapping behaviour starts after that.

If the tasks swap places more than a few times each run, your GPU is marginal and you should investigate the cause.

I'm not sure what your last statement means "your GPU is marginal".

Error -97s are a strong indication that the GPU is misbehaving. When I get around to it, I'll sort out a memory testing program for you all to test for this. The access violations are indicative of GPU hardware problems. There are a few hosts that are disproportionately affected by these (Operator, 5pot) and it's not clear why. I suspect there's a relationship with some third-party software, but don't yet have a handle on it.

Matt

Well my GTX660 has a lot of -97 errors, especially with the CRASH (Santi) tests. LR's and the Noelia's are handled most of the time error free on it. Einstein, Albert and Milkyway run almost error free on it too. So if it is something of the PC or other software (I don't have many installed), would also result in 50% error with the other 3 GPU projects. Or not?

The *only* other thing running besides GPUgrid, was WCG. And I mean no anti-virus, nothing at all. Still got the access violations. So I suspended the WCG app. Still the access violations occurred.

Although my GPU appears to be functioning correctly (just in general), I bumped the voltage to 1.175 from 1.163. Will report back if this has any impact on the access violations.

Thanks Zoltan, I will install them and try tomorrow (later today after sleep).
____________
Greetings from TJ

Error -97s are a strong indication that the GPU is misbehaving. When I get around to it, I'll sort out a memory testing program for you all to test for this. The access violations are indicative of GPU hardware problems. There are a few hosts that are disproportionately affected by these (Operator, 5pot) and it's not clear why. I suspect there's a relationship with some third-party software, but don't yet have a handle on it.

Matt

Yours appear to be happening much more frequently than mine. Even after I increased the voltage, still get access violations.

To reiterate, this is with absolutely no third party software running at all. In all honesty, I can only assume that something in the app is causing this behavior.

Why it's affecting Operator's Titans more than my 780s, I'm not sure.

I have now set this box to do betas only.

- Updated the betas (8.14) are doing the same swapping as the normal long WUs were doing. I have to manually suspend two of them to keep this from happening.

Hello Operator,

Perhaps a ridiculous idea and perhaps you have tried it already: what happens when you only use one Titan in that PC?

Have you set the minimum and maximum work buffer as low as possible, so that you don't get to fast new WU's?

It seems to be happening with Titan and 780 the two top cards. The 326.41 drivers are not the latest one at nVidia's site.
____________
Greetings from TJ