I had a bunch of compute errors on my 8800GT, but then the latest KASHIF_HIVPR completed OK over a couple of days.
Full task list: http://www.gpugrid.net/results.php?userid=9833
Latest KASHIF_HIVPR WU completed fine: http://www.gpugrid.net/workunit.php?wuid=449234
Comp specs: http://www.gpugrid.net/show_host_detail.php?hostid=17613

Doesn't seem much rhyme nor reason to the fails other than the recent probs with WUs in general (blackout).

2009-05-12 15:08:32 GPUGRID Starting task 4-KASHIF_HIVPRFE_dim_ba1-2-4-RND6858_0 using acemd version 664

hmmm it is now 39.8% after 6:40H, and it's says that it remains 10H....

Is it normal on GTX260 and 182.08 and 6.6.20 and XP 32 ?
____________
POLISH NATIONAL TEAM - Join! Crunch! Win!

Today the error rate for Kashif wus is lower, so it could have been a problem with drivers.

In the next few days we will perform a server update and application updates to use CUDA2.2.

It is as clear as crystal ...

But it usually crunch 7-8h a WU not 17!!!

And in this tread this type of WU is mentioned so maby it is relevant?
____________
POLISH NATIONAL TEAM - Join! Crunch! Win!

Alright.. could be the *usual* 6.6.20 bug.

MrS

Have 5-KASHIF_HIVPR_dim_ba1-4-100-RND6112_0 using acemd version 664 running since 21 hours on 9800gx2, 68% done, never had such long wu on gpugrid, usually i make like 3/4 wus in 21 hour. Hope credit will be as great as the time it takes to compute ;).

Well i downloaded as allways both the driver and the cuda toolkit from nvidia site.
After the initial pause on gpugrid i have to report that i did not have a failing unit for a few days now.
Not sure if anyone else does download the Cuda toolkit or just the driver.
I am almost done with the test unit which finishes in about an 1/2 hour or so
I hope the new received IBUCH ones will finish also without issues.
If they all finish without error i start to get the feeling the problems are solved .... i hope :D

Today the error rate for Kashif wus is lower, so it could have been a problem with drivers.

In the next few days we will perform a server update and application updates to use CUDA2.2.

gdf

2009-05-12 15:08:32 GPUGRID Starting task 4-KASHIF_HIVPRFE_dim_ba1-2-4-RND6858_0 using acemd version 664

hmmm it is now 39.8% after 6:40H, and it's says that it remains 10H....

Is it normal on GTX260 and 182.08 and 6.6.20 and XP 32 ?"

CUDA 2.2 libs will be distributed with the application, but you will need to upgrade the driver to the latest 185 version.

gdf

The test unit ended also without problem and new ibuchs on the way.
I haven't had any cancelled other then one being in queue for almost 2 days so nothing speical on that.
I am still running the 185.85 and boinc 6.6.28
Except the usual problems with boinc issues like fetch and such it runs stable for me, my slow 9600 Gt seems to do well.
But i had to lower my clock on my cpu since i had to disable my watercooling, the 9850 BE is a hothead because with the huge cooler on it becomes 55 C.
But its today extremly warm here i measured 31 C in the room ambient temp.

The test unit ended also without problem and new ibuchs on the way.
I haven't had any cancelled other then one being in queue for almost 2 days so nothing speical on that.
I am still running the 185.85 and boinc 6.6.28
Except the usual problems with boinc issues like fetch and such it runs stable for me, my slow 9600 Gt seems to do well.
But i had to lower my clock on my cpu since i had to disable my watercooling, the 9850 BE is a hothead because with the huge cooler on it becomes 55 C.
But its today extremly warm here i measured 31 C in the room ambient temp.

Today the error rate for Kashif wus is lower, so it could have been a problem with drivers.

In the next few days we will perform a server update and application updates to use CUDA2.2.

gdf

Just finished looking at a LOT of KASHIF_HIVPR WUs. The situation is not improving at all and is not a driver issue. What happens is these WUs are downloaded and either fail or are aborted repeatedly until they happen to be assigned to a GTX 260 or above, then they complete. The problem is not fixed and is not improving. IMO it needs to be dealt with ASAP.

Here's just a few examples:

http://www.gpugrid.net/workunit.php?wuid=440561
http://www.gpugrid.net/workunit.php?wuid=442250
http://www.gpugrid.net/workunit.php?wuid=454479
http://www.gpugrid.net/workunit.php?wuid=449101
http://www.gpugrid.net/workunit.php?wuid=457871
http://www.gpugrid.net/workunit.php?wuid=458509

2009-05-12 15:08:32 GPUGRID Starting task 4-KASHIF_HIVPRFE_dim_ba1-2-4-RND6858_0 using acemd version 664

hmmm it is now 39.8% after 6:40H, and it's says that it remains 10H....

Is it normal on GTX260 and 182.08 and 6.6.20 and XP 32 ?"

whell, now it crunch that WU 18H and it is 83%, it says 3h30min remaining...

Yep, the best driver for a 260 is Boinc 6.4.7 and driver 178.28. and cuda 2.
Working fine.........

Today the error rate for Kashif wus is lower, so it could have been a problem with drivers.

In the next few days we will perform a server update and application updates to use CUDA2.2.

gdf

Just finished looking at a LOT of KASHIF_HIVPR WUs. The situation is not improving at all and is not a driver issue. What happens is these WUs are downloaded and either fail or are aborted repeatedly until they happen to be assigned to a GTX 260 or above, then they complete. The problem is not fixed and is not improving. IMO it needs to be dealt with ASAP.

Here's just a few examples:

http://www.gpugrid.net/workunit.php?wuid=440561
http://www.gpugrid.net/workunit.php?wuid=442250
http://www.gpugrid.net/workunit.php?wuid=454479
http://www.gpugrid.net/workunit.php?wuid=449101
http://www.gpugrid.net/workunit.php?wuid=457871
http://www.gpugrid.net/workunit.php?wuid=458509

Here's a new KASHIF_HIVPR that was just downloaded to me (and I aborted). Notice that it just caused an error on a GTX 260 {after running a long time I might add).

http://www.gpugrid.net/workunit.php?wuid=459189

That same GTX 260 has only 3 recent failed WUs, all of them KASHIF_HIVPR. Take a look for yourself:

http://www.gpugrid.net/results.php?hostid=32169

It sure looks like the KASHIF_HIVPR problem also bites the faster cards, just not as often. Our team members have also been reporting the same problem on the GTX 260 and above. So it's documented. Any chance of getting this fixed?

Ubuntu 9.04 comes standard with driver version 180.44, which avoids so far to have to fiddle with manual interventions.

Wiil they follow before or after GPUGRID decides to require the 185 version drivers? If a manual update of the Linux community is required, please advise in advance.

Many thanks

Kind regards

Alain

That same GTX 260 has only 3 recent failed WUs, all of them KASHIF_HIVPR.

In light of the issues with the older GPU's and the KASHIR_HIVPR WU's, what is the best version of nvidia driver to use?

I have been aborting them when I see them, to get them over to a 200 series as quick as possible. I don't think it is beneficial for the project for me to let this sit in my queue for 12 hours, then run for another several before failing anyway.

I'd prefer not to babysit, so should I roll back my current 185.66 to the last WHQL approved non-185.xx driver, which is 182.50?

I guess I could just try this and report the results, but I wanted to know if anyone has already tried this 182.50 driver w/ an older (non-200-series) card.

That same GTX 260 has only 3 recent failed WUs, all of them KASHIF_HIVPR.

It looks like you have a GTX 260 and an 8800GT. All three tasks failed while running on the 8800GT (device 1), not on the GTX 260.

That same GTX 260 has only 3 recent failed WUs, all of them KASHIF_HIVPR.

It looks like you have a GTX 260 and an 8800GT. All three tasks failed while running on the 8800GT (device 1), not on the GTX 260.

You're right. Not my machine and I didn't see the 2 cards. But OK here's an example from a machine with only a GTX 260:

http://www.gpugrid.net/result.php?resultid=663665

CUDA 2.2 libs will be distributed with the application, but you will need to upgrade the driver to the latest 185 version.

gdf

Are you saying that without 185 version drivers we will not be able to successfully do GPU Grid work.

I have card/box combinations that will not accept that version and run properly.

If 185 version driver and above is "required" to crunch here, I will be taking my farm to FAH.

Task ID 665546 had been running well along with another task. As I was about to run a program that would "use" the GPU I decided to suspend all tasks and exit Boinc. Once I had completed my task I launched Boinc, all tasks appeared still suspended. So far so good.I then resumed all tasks, and task 665546 immediately went to "compute error". I also had another task 652947 that had been running for 29 out of about 30 hours and failed (different machine). When I get the time I will compile a list of the failures and successes over the past few days.

which card/machine combinations are not possible to use the 185.85 version may i ask mike047 ?

That same GTX 260 has only 3 recent failed WUs, all of them KASHIF_HIVPR.

It looks like you have a GTX 260 and an 8800GT. All three tasks failed while running on the 8800GT (device 1), not on the GTX 260.

You're right. Not my machine and I didn't see the 2 cards. But OK here's an example from a machine with only a GTX 260:

http://www.gpugrid.net/result.php?resultid=663665

:-) That one reports as "Aborted by user". So I don't think it errored out under normal circumstances -- it's was manually aborted.

That same GTX 260 has only 3 recent failed WUs, all of them KASHIF_HIVPR.

It looks like you have a GTX 260 and an 8800GT. All three tasks failed while running on the 8800GT (device 1), not on the GTX 260.

You're right. Not my machine and I didn't see the 2 cards. But OK here's an example from a machine with only a GTX 260:

http://www.gpugrid.net/result.php?resultid=663665

:-) That one reports as "Aborted by user". So I don't think it errored out under normal circumstances -- it's was manually aborted.

@Beyond - I didn't doubt you. :-)


@GDF/Admin:

Given these KASHIF_HIVPR seems to error out a lot, especially with "older, slower" cards, but also with new 200-series occasionally as well (as shown by Beyond), are no new ones going to be created?

I can understand cleaning out the queue, but I have gotten several today and with my cards I almost certainly expect them to error out. If I catch them in my queue, I try to abort them so they can move to a 200-series with a better change of finishing in a timely manner.

Is there any analysis from the project on why these particular WU's are an issue? I've read comments about the drivers possibly being an issue, but given the 2.2 CUDA software on the server will require these 185.xx drivers I expect to continue having issues with these WU's if they are still in queue. All others work fine.

As GPUGrid clearly does not want to put in much effort to support 8 and 9 series cards, I'm done here for now. I'd rather shut them down than to waste time and electricity in an endless circle jerk of BOINC versions and drivers. But hey, 3.7 million credits was a good run for me here. There will be a new GPU project out soon.
Sad really, as I think some of the science was worth doing here. :)
Ciao.
____________

which card/machine combinations are not possible to use the 185.85 version may i ask mike047 ?

I can confirm my BFG GTX-260 192 Shader card is also getting alot of these errors with 185.81.

One example
____________

Oh and SETI has nVidia support so there is another BOINC project.
____________

The user is one of my team members and he reported it as being stuck. It had processed for over twice as long as his other WUs and showed no progress. He was using BOINC client v6.6.28, not v6.6.20 so that wasn't the problem. :-)

We have managed to replicate the problem on one of our machines.
This should lead to a solution soon.

Be patient.

I worked out the numbers on my computers, they all run 182.50 drivers.

ID: 30829 (8800GT 256Mb) - 11% failure rate
ID: 33373 (9800GX2 512Mb) - 46% failure rate
ID: 26481 (9800GX2 & GTX260) - 29% failure rate
ID: 34636 (9800GX2 & 8800GT) - 18% failure rate

It seems strange that the 8800GT is the most reliable card give the issues. 26481, did have an issue that I know was my fault, so that's a little higher than expected.

Hmm i am not convinced its just the drivers i started under win xp with 182.50 driver and boinc 6.6.28 but again i see the ibuch unit hang on 64.688% for more then an hour after 13 hours of calculation.
So i start to believe this one is going to crash as well

My card is an 9800GTX whith 185.82 driver and Boinc 6.6.20.
I don't want to downgrade drivers, so, in the mean time, i suspended any WU's for GPUGRID.

I hope to see good news asap.

Sorry for my bad english...

Greetings, Matteo

Hmm i am not convinced its just the drivers

GDF said the problems appear with 185.xx and don't show up with some 180.xx, which apparently noone else is still using. This does not mean that 182.xx is fine and I think the usual "KASHIF_HIVPR" and "IBUCH_KID" problems definitely affect 182.50.
It seems to be a problem with the driver, triggered by some new WUs.

Sorry, not very specific post. Not all WUs with those names are affected, e.g. see here. "KASHIF_HIVPR_mon" and "KASHIF_HIVPR_dim" have been fine for me.

MrS

This tasks: p1480000-RAUL_pYEpYI1605-0-10-RND5295_0 started up and I have 5:10 or so on the clock ... so, unlike all the rest, finally got one running. It is running on the new MB, but the old GPU.

SO, this batch of tasks is so bad that most of them won't run on anything ... though my GTX 295s seem to be rolling on ...

{edit}

I was wrong ... it is on one of the new GTX 260 cards ...

GIANNI_FB's have come in for some flak lately , so thought I would post a successful one as comparator. The stop/starts in there were me, due to non-BOINC related stuff.

http://www.gpugrid.net/result.php?resultid=677172

Regards
Zy

GIANNI_FB's have come in for some flak lately , so thought I would post a successful one as comparator. The stop/starts in there were me, due to non-BOINC related stuff.

http://www.gpugrid.net/result.php?resultid=677172

Regards
Zy

Glad your new rig made it through one WU successfully! The oters don't look too well, though. They error on most other hosts as well, but 3 have been finished by other GT200 cards. One of them uses 185.85, but I can't see the others.

MrS

The similar for me,

http://www.gpugrid.net/result.php?resultid=678214
http://www.gpugrid.net/result.php?resultid=664263

The similar for me,

http://www.gpugrid.net/result.php?resultid=678214
http://www.gpugrid.net/result.php?resultid=664263

Hello,

oops
http://www.gpugrid.net/workunit.php?wuid=466073
http://www.gpugrid.net/workunit.php?wuid=458046

give me 5500 points for 17/24 hours of crunch (260GTX 216 SPU O/C stable
)

I maybe lucky but I am having very few problems. 185.85 drivers, XP64, 2 EVGA 260s, Boinc 6.6.28 I had 1 compute error yesterday but that was my fault for suspending right as it started and unsuspending a couple of seconds later, and a couple of others that everyone else in the quorum errored out on.
I have heard of hanging WUs but have never had one of those either.
but 99 percent of the time it runs great.
I always take great care to run driver sweeper in safe mode after uninstalling Nvidia drivers before updating. I do not know if this matters that much though.
I also never let the gpu temps get over 65c with moderate OC.
Also 4 cpu units of either seti astropulse, einstein or abc running along side at same time always.
I had a 9800gt in this computer for about a month that ran good as well. Replaced it with a 260 this week.

Now i have been able to save a few hanging units
It seems to work for me first make sure to disable keep units in memory under options.

I pause all other units available then i pause the unit which does not move in progress then push it to continue, i know it costs alot of time because it jumps back to some point in time.

Untill now i had 4 units which kept at a certain % and did not move in more then half an hour so i started messing with them.

When i woke up this morning i saw a 92-kashif_hivpr_dim unit reporting to have done 0.700 % in 7 hours so i paused it, ofcourse it jumped back to 0.426 % when it started over but now did in half an hour 1.5 %.

So the reason seems to be the units get stuck in the calculations and finally error out if this takes too long.

But i can tell you its a pain in the ass problem when they hang you hardly notice, we don't have time to watch them all day if the units progress or not.

Now I have the fourth error WU in a row. :-(((

http://www.gpugrid.net/result.php?resultid=678849
http://www.gpugrid.net/result.php?resultid=679319
http://www.gpugrid.net/result.php?resultid=680211
http://www.gpugrid.net/result.php?resultid=680860

It wastes a lot of GPU-time for scientific knowledge!
It costs a lot of credits...
It costs a lot of fun...

Is GPUGRID going to be used only with newer cards?

Attention! Sarcasm!
Is there a hidden deal with NVIDIA to push cards with G200...?

My System
Q9550 @ 3.4
8800 GT @ stock
4 GB
Windows 7 RC 64 Bit
185.85

Sadly i was not paying attention so the last one did error out again,but to be honest i was expecting it to fail also since i had to restart it 3 time in a row to start seeing progress.

I am on Win XP pro with 182.50 driver and boinc 6.6.28 , for me there was however indeed some gain with the 185.85 but i just wanted to make sure the drivers aren't the issue.

The newer driver gave a little faster finishing time the old was 20 - 27 hours and the 85 between 19 - 23 hours.

I have been trying to test the older 180.XX driver,
But it made my system unstable for some reason so i cleared out all nvidia stuff and reinstalled 182.50 whql version.

I am now going to change back the boinc to 6.5.0

Thank you for link. I'opened the Web page of my pc , as for GPU 260GTX of this pc's I am with boinc 6.6.20 who satisfied me and Nvidia 182.08 on Win Xp pro64.


http://www.gpugrid.net/hosts_user.php?userid=1695

On the contrary for points over 24h00:10.000 points on GPU 260 O/C:( all GPU 280/285GTX

@+

Drivers Nvidia 1XX.XX http://www.nvidia.fr/Download/Find.aspx?lang=fr

I really gotta call that electrician to change my old 3 phase 230 V UPS socket into a 30A 115 supply...

Do you think that's a good idea? I don't know your 230V, but at 115V the power supplies loose efficiency compared to 230V. 30A @ 115V is 3.5kW, quite massive :D
I know we can draw at least 2kW over the regular 230V, whereas I heard the US net may deliver something around 1.5kW at 110V. Our 3 phase plugs are 380V and I think you can get 5 - 6 kW from them.. but you're not talking about these, right?

Yes really 84000s instead of 42000s for 14-KASHIF_HIVPR_dim_ba3-8-100-RND7871_1

http://www.gpugrid.net/result.php?Resultid=680472

I am crossed has 6.6.28 boinc It is possible that Seti beta is responsble of this probleme. Thanks for your help, I keep posted PS3GRID about suite.

I rolled back my drivers from 185.85 to 182.50. With windows Vista 64 bit, Boinc client 6.6.28. Since which I have returned three successfull results, one of which had run for 30 hours on one core of my 9800 GX2 and gave me just over 10,000 credits :-)
So at present this role back on the driver is working for me (touch wood).

I also rolled back the driver on my other machine from 185.85 to 180.48.With windows Vista 32 bit, Boinc client 6.6.20 (Yes, I know :-) ). This has so far returned one result, plus another well on its way. I realise that neither of these is a large sample. But looks promising given the quantity of failures I had seen just prior to changeing drivers.

I will now leave alone for a few days and see how things turn out.

I am finding it hard to tell what is going on... I seem to be getting tasks our of order so that they don't sort well on the results pages. As I watch the computers they seem to be returning mostly good results ... with occasional errors.

Well, I guess I will have to wait till Monday when the staff comes back in and fixes the universe ... :)

I still cannot make heads nor tails of the pattern of errors. One of the problems of course is the difficulty of gathering data about the failures.

Some of the older tasks that failed on one of my systems passes on another system that is very much alike. I thought I was onto something about memory size where some of my cards have that 895 instead of 1G and the tasks passed on the 1G cards. Alas, I quickly found another case where it failed on mine and passed on someone else's card and they too had only 895 M VRAM.

Driver versions 182.50 on my systems failed, but the systems where the task passed also were running the same version.

The tasks are of all name classes...

Even my i7 with the pair of GTX295 cards finally had [url-http://www.gpugrid.net/result.php?resultid=685755]one fail[/url], the message is singularly unhelpful.

I am crossed has 6.6.28 boinc It is possible that Seti beta is responsble of this probleme. Thanks for your help, I keep posted PS3GRID about suite.

Have anyone tried to use updated dll's

The CPDN memory limit of 1.5Gb is set that way to allow for four running on a quad, and enough left over for op sys etc within the quoted figure of 1.5Gb. Each of the larger CPDN WUs takes up 210-220Mb in memory, therefore four of them will eat around 850Mb, with a comfortable margin for opsys etc, within the stated 1.5Gb. Its not 1.5Gb each WU, that figure they state as advisory, is total memory on the PC, not per WU.

Most CPDN models are much smaller - either side of 100Mb - albeit on the larger size than most BOINC WUs. I have happily run four of the biggest ones on my quad without issues and GPUGRID on the 9800GTX+. Usually I have two of the bigger CPDN ones running with two SETI Astropulse on the quad and GPUGRID on the 9800GTX+, they run fine with no issues.

Regards
Zy

Zydor did you actually select the big units on your account page since by default they are not loaded, you really should read what it is stated there.
Those units are minimal 1,5 Gb of memory nothing else the warning is clear >.<

Sadly i have forgotten to take a screenshot when i was running 4 of those biggest units at the same time.

Ofcourse i think the change that you get 1 or even more then 2 of those big ones is very small.

I have not seen recently any of those big units, so it could have been a freak moment that i received 4 of those big units at once. It also can be that these huge units are only send to x64 machines i have not been following up any news about them.

The only thing i can say with you running seti and gpugrid together that in my case it ended up several times with crashing my pc or the unit, but again i was having more problems with the seti beta then with the normal seti.

If it does not happen to you does not mean other people can be so lucky that all goes well.

I just had a case where a suspended CPDN task caused two GPU Grid tasks to go into waiting for memory state. I had to stop BOINC and restart it so that the CPDN task (only 300K) would be swapped out ...

As usual i reported it so that there is another bug for UCB to ignore ... :)

Task Manager reported the larger ones taking up 220Mb min and did go to 400Mb at times, and four did run fine. You can get four by setting preferences for only those units.

I thought the same as you re the 1.5Gb, but also thought it strange they would produce one that size even these days, it would cripple many PCs, not a good thing for general release.

I therefore checked it out with CPDN, the response was they take up at the most 500Mb, and four of them would fit on a PC with 3Gb with no issues. When I ran the four, Task Manager reported either side of 220-400Mb in use, may well have gone to 500Mb when I was not watching, didnt log it. The post and respones is :

http://climateprediction.net/board/viewtopic.php?f=21&t=8675

I had no doubts you had issues running the two. Its also the case that often responses on success of combinations can produce as much info to help debug as failures, as a comparitor can help isolate an issue. I've often cursed when something hasnt worked, then scratched my head when I discovered others were having some success - helped me.

Regards
Zy

So, it seems that there is a bug in the compiler/hardware which appears only on pre G200 cards.
We found a way to avoid it for now, but it limits what we can do, so it is not a solution.

gdf

I am running an 8800GT and 3 9800Gtx+ cards. I have had zero complete Kashif Wu's .
Could you make a way for me to "opt out" of those type of units?

I still get occasional errors with other units but the Kashif are 100% failure rates for me. (3 different machines)

I am really unable to babysit my machines as I am away from home for days at at time.....

we are running this set of workunits called

x-GIANNI_newFB-...

If they go on ok, then we have isolated the problem with G90 chips. It is not solved yet but still at least we would know where to look.

gdf

Just as a data point of reference, I have had 100% success on all work units using a GTX 260 Core 216 card, running CUDA 2.2 and 185.85 driver, even on work units that have had failures previously.

There is another way .....

Tap your well heeled benefactor you have tucked away for a mere $400,000 worth of vouchers to upgrade Crunchers pre-200's to 300GTXs - a snip at the price....

And ..... added Value!! ..... You'd also solve chruncher recruitment for while, they'd queue round to the next street, let alone next Block, for that one .....

*sigh* ........ aways nice to dream once in a while :)

Regards
Zy

There is another way .....

Tap your well heeled benefactor....

Regards
Zy

Just had a Kashif go bang

ERROR: c:\cygwin\home\speechserver\gpumd2\src\pme\CPME_cufft.cu, line 104: cufftExecC2R (gridcalc3)

http://www.gpugrid.net/result.php?resultid=699822

I have had a problem on that PC re Office, and got it back online 5 hours ago. However I dont think it was that issue, I think it looks like the old WU problem surfacing - maybe in one of the older WUs still in the system??

Regards
Zy