This one's killing me.....

I've been through the computation error thread....it's touching on so many different things these days...I really don't want to mix this in with that.

This "restart.coor" has not been a problem in the past, but the last couple of weeks--it's killing me....

If my PCs reboot--the WUs finish , then error out = restart.coor.

This started on it's own--out of nowhere. I've since upgraded the driver to 185.26--samething. I upgraded the Boinc version today from 6.6.3 to 6.6.17 and now I have cores that have been running for hours--the first of which has just finished and errored out = "restart.coor". I'm now expecting the next 3 WUs to do the same. Maybe the ones in queue again also.

Yesterday I rebooted a rig and the WUs completed, errored out restart.coor--and so did the WUs in queue....

There's something going on here. Really not happy about the errors, but these WUs continue to run for hours (5-6hrs today) then error out when they are finished. Major waste of resources. This is a new thing that developed without any changes to my rigs.

Today I did not reboot--just upgraded the boinc version. So, now every time I shut down boinc I can expect this? What if I just put it on snooze?

Something is wrong here and with all of the computation errors going around...I really don't believe the it's on my end or solely on my end.

Yes, all my stuff is OCed, but it's been that way for months. Everything is fine--unless there's a restart of boinc...plus I've tweaked down some OC's and this situation continues to go from bad to worse. My PCs are unhidden if anyone wants to take a look.

Getting very frustrated here. If they just errored out it wouldn't be so bad, but they run for hours....until they complete....just to error out....


HELP! ;)

Next one error our = restart.coor. Expecting at least 2 more from different rigs....

First of all, the real error message is "incorrect function - exit code 1". Disregard the restart.coor, it appears in all WUs, also the one that complete succesfully and validate.

Secondly, the errors occur on 2 of your 3 machines, but some of the WUs there do finish succesfully and validate. Which indicates that software is not the problem.

So remains a hardware problem. My guess here is aging of certain components in power supply and/or videocard. Even further reducing the OC on your videocards is likely the best option to try.

Good luck.

kind regards

Alain

First of all, the real error message is "incorrect function - exit code 1". Disregard the restart.coor, it appears in all WUs, also the one that complete succesfully and validate.

Secondly, the errors occur on 2 of your 3 machines, but some of the WUs there do finish succesfully and validate. Which indicates that software is not the problem.

So remains a hardware problem. My guess here is aging of certain components in power supply and/or videocard. Even further reducing the OC on your videocards is likely the best option to try.

Good luck.

kind regards

Alain

I may end up having to make a choice, but if so....GpuGrid will win out. It's good to know though that the problem(s) should be on my end. This has been driving me nutz....;)

Hmm i don't agree that it can't be a software problem.
But as i check other projects, i see that even though they are damn fast these GPU's.
They make a lot of errors on their jobs, they are simply made for raw performance not very demanding computations.
For a game loosing a pixel here or there won't do much, but into a computation thats some kinda different cake.
I also found units crash when reported to be done completely, and only have to be send to the project.
Can someone tell me where the hardware comes in when a unit has to be send.
I believe these GPU projects have a good future but we must not forget they still have some problems to be solved.
So i guess we will be stuck on what we have now and see units being destroyed sometimes for non good reasons.
Its not allways certain that underclocking the VC is a solution for not getting errors, i underclocked my VC on the lowest settings possible and even then units from seti and seti beta crashed.
We cant complain much since all these projects are basically still alpha or beta or in some kinda test.
So failures will keep coming for no good reasons i guess, or the makers must find a cheap/fast way to give a change to recalculate on the error given.
Sadly now even though it has been calculating the whole damn thing for 16 or more hours you end up getting nothing.

P.S talking about the devil my laptop crashed hard with a seti beta unit running

Hmm i don't agree that it can't be a software problem.
But as i check other projects, i see that even though they are damn fast these GPU's.
They make a lot of errors on their jobs, they are simply made for raw performance not very demanding computations.
For a game loosing a pixel here or there won't do much, but into a computation thats some kinda different cake.

... Which indicates that software is not the problem...

Michael my gpu temps are much below average.
I also have a EVGA card and must admit it to be a good brand.
My card full load only becomes 63 C with fan on 40%, and if i turn up that coolmonster to 70% it goes down to 53 C.
So i guess those temps should be fine ;)
Sometimes i wonder if the temp is because of the low shader count or that i simply clocked under specs which it should run at.
It should run with higher clocks provided by EVGA and i am sure it can do more.
But for the sake of the project i am going to keep it low.
I had a crash from windows and after restart, found a 12 hours running unit crashed immediate.
And again it show restart.coor as reason, so somehow i get the feeling that restarting the unit where it left at that moment is not properly working.
But i simply blame the admins thats easier ;)

The settings of my EVGA OC 9600 GT
Settings gpu 600 should be 675
shader 1600 should be 1800
memory 900 should be 900(1800)

Webbie,

the restart.coor file is only the checkpoint file... So in your case, the computer maybe just crashed when the app was writing to the file and it got corupted when the PC crashed. So it couldn't restart the computation of the WU... Just pure speculation of course... ;)
____________

pixelicious.at - my little photoblog

hihi yea lol i figured the same but then again pure speculation again ;)

Another potential source of errors is a power supply that is almost but not quite powerful enough to consistently drive the GPU. Since the GPU's power draw is dependent on how hard it's working, having a marginal power supply might work most of the time, but cause errors once in a while.

Actually, there's no end to the possible causes.

The question is how do you diagnose the problem.

One thing I would try is swapping hardware if you have more than one rig. If you have another GPU, see if that works in this computer. If you have another computer, see if your GPU works in that computer. Perhaps you can isolate whether it's the GPU or the computer that's the root of the problem.

Hey Everybody,

Just wanted to say Thanx for the input and a special Thanx to Paul for offering to help weed through this with a fine tooth comb. :)

2 possible issues--1 is heat. I have a unique mega water cooling setup for my rigs. Too complex and detailed to go into, but I've made some corrections/adjustments and things are running properly again. I think it's the core issue...maybe the only issue. I turned my OCs back up and I'll check out the last possible issue in a day or 2 to see if it is an issue or not. Ran all day today with Successful WUs, so hopefully I'm back on track!

Hi ETA,

Yes, I do push these cards hard....:). I don't use Furmark or any other "stability" program(s). Many have realized that they have limited value when it comes to crunching and doing it 24/7/365. I just OC until I get errors and then back down. Once I find the high end I back it down a notch further into the sweet spot. And by "errors" I mean driver errors, boot, etc...not necessaarily WU errors--those are usually the last step. But I only do the OC determinations when I have time to babysit. Otherwise I run "under" so-to-speak until I have an opportunity to do it right.

My stuff has basically been running fine this way for months. However, I had an issue with my mega WCing setup. I understand what was going on now, but was quite confused before. Ultimately, my cards are OCed to where they should work fine to +55C.....after that it's hit and miss. With the prior flow issues that developed from WCing changes and just other factors....temps were pushed upward in a somewhat erratic fashion. Again, it would take a book to try to explain everything....

Right now the cards are running ~35C, so I have quite a bit of head room....

Back to 1 successful WU after another, so it seems as though heat was the main issue....

Because of the likelihood of GPUs being overclocked to the point of producing non-crash errors, I would think that most projects would be well served by running with at least a quorum of 2 so that some flipped bits don't end up distorting the science results.

I'm sure very few people outside of the hardware manufacturers have done sufficient testing to really know at what point, for a given temperature, a particular GPU starts becoming unreliable in as much as the data starts becoming erroneous. It's easy to know when it crashes. It's not that easy to know if the calculations are correct.

To really insure that the card is working correctly, you need to repeat the tests at a range of temperatures, using software that will verify that the results are correct. Furthermore, the tests need to be created in such a way that all of the hardware is being exercised. In other words, it's not enough that the results were correct. The test must use every multi-processor and ever shader on the GPU. Then, given the expected MTBF of the hardware at a given temperature and clock rate, the test needs to run continuously for a multiple of that MTBF.

Determining the MTBF requires testing with multiple examples of the GPU.

To make things worse, as the card, power supply, and fans age, the GPU may not perform as well as it did when it was new. So the tests should be repeated periodically.

That's how you insure that your GPU is producing valid science data, as opposed to merely not crashing.

Clearly, this is not only onerous, but outright impossible for the average consumer. If you're running over-clocked, you really can't be certain that all the calculations are occurring correctly. To be fair, even if you're running at factory speeds, you can't be certain either. That's why, IMHO, projects should always use a quorum of at least 2.

Of course, there are some projects where validating 2 results against each other is impossible due to the nature of the work, but those projects are a small minority.

Frankly, given the current interest in GPU-computing, I'm surprised that Nvidia and ATI don't include tools that let you test the accuracy of your calculations so that consumers can make wise choices when overclocking. Right now, we're flying blind.

Mike

Glad it works again! And although I may not telling you naything new here: it was not "heat", it was the "combination of clock speed and heat" ;)
The higher the temperature the smaller the maximum stable frequency becomes. So at 50 or 100 MHz less you would likely still have been fine at those higher temps.

Do you think it's really in NV or ATIs interest to give us some tool to proof that hardware is defective?

I don't understand this error! See example below, it's the same WU!

me:

stderr out

<core_client_version>6.6.14</core_client_version>
<![CDATA[
<message>
Unzul�ssige Funktion. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
# Using CUDA device 0
# Device 0: "GeForce GTX 260"
# Clock rate: 1404000 kilohertz
# Total amount of global memory: 939196416 bytes
# Number of multiprocessors: 27
# Number of cores: 216
MDIO ERROR: cannot open file "restart.coor"
# Using CUDA device 0
# Device 0: "GeForce GTX 260"
# Clock rate: 1404000 kilohertz
# Total amount of global memory: 939196416 bytes
# Number of multiprocessors: 27
# Number of cores: 216

</stderr_txt>
]]>


User OnTheNet

stderr out

<core_client_version>6.6.17</core_client_version>
<![CDATA[
<stderr_txt>
# Using CUDA device 0
# Device 0: "GeForce GTX 260"
# Clock rate: 1515000 kilohertz
# Total amount of global memory: 939524096 bytes
# Number of multiprocessors: 24
# Number of cores: 192
MDIO ERROR: cannot open file "restart.coor"
# Using CUDA device 0
# Device 0: "GeForce GTX 260"
# Clock rate: 1515000 kilohertz
# Total amount of global memory: 939524096 bytes
# Number of multiprocessors: 24
# Number of cores: 192
# Time per step: 40.729 ms
# Approximate elapsed time for entire WU: 25455.677 s
called boinc_finish

</stderr_txt>
]]>


Almost the same GC (my gtx260-2), but my WU crashed = 0 credits!
There are 9 of 19 WUs crashed in my task history! I've lost 35k points that makes me sick.
The results of OnTheNet seems to be ok, every time. Moreover it seems to me that he has overclocked his GC.


I'll try to update boinc, NV forceware and try again.