Advanced search

Message boards : Graphics cards (GPUs) : Recent slower application

Author Message
Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 1561 - Posted: 20 Aug 2008 | 10:05:01 UTC

A very frequent checkpointing set in the boinc user preference will slow down the application (even by 50% more). New application version will fix this.

GDF

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 1562 - Posted: 20 Aug 2008 | 11:28:22 UTC - in response to Message 1561.

A very frequent checkpointing set in the boinc user preference will slow down the application (even by 50% more). New application version will fix this.

GDF


Fixed by version 6.34 for Linux.

GDF

Profile [XTBA>XTC] ZeuZ
Send message
Joined: 15 Jul 08
Posts: 60
Credit: 108,384
RAC: 0
Level

Scientific publications
wat
Message 1563 - Posted: 20 Aug 2008 | 11:40:40 UTC
Last modified: 20 Aug 2008 | 11:41:57 UTC

Hi

Where can we modify this (the frequency of the checkpoints) in the boinc user preference? I haven't found this option :/

I trie the new application (2.30) on windows and it seems to be similar as the 2.29 one

Thank you for your involvement and your presence for this project :)

Profile UBT - NaRyan
Avatar
Send message
Joined: 16 Jul 08
Posts: 68
Credit: 1,242,980
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 1564 - Posted: 20 Aug 2008 | 11:51:18 UTC - in response to Message 1563.
Last modified: 20 Aug 2008 | 11:52:00 UTC

Hi

Where can we modify this (the frequency of the checkpoints) in the boinc user preference? I haven't found this option :/

I trie the new application (2.30) on windows and it seems to be similar as the 2.29 one

Thank you for your involvement and your presence for this project :)


It's in Advanced, Preferences, Disk and Memory usage.
"Write to disk at most every" xx "seconds"

Mine is set to 60 seconds.
How often is "frequent checkpointing" that causes the slowdown?
____________

Down with the Kredit Kops!!!

Profile [XTBA>XTC] ZeuZ
Send message
Joined: 15 Jul 08
Posts: 60
Credit: 108,384
RAC: 0
Level

Scientific publications
wat
Message 1566 - Posted: 20 Aug 2008 | 12:00:38 UTC

Ah, thanks UBT - NaRyan, mine is set to 60 sec too

Profile Krunchin-Keith [USA]
Avatar
Send message
Joined: 17 May 07
Posts: 512
Credit: 111,288,061
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 1589 - Posted: 21 Aug 2008 | 13:52:28 UTC

I beleive 60 is the default. I had mine bumped up to 300s/5m or 600s/10m depending on my venue. Been that way for a year or more. I've noticed no ill effects with using this time with any project, except to note, if you turn off computer or boinc when any app restarts you loose up to that much time depending on when the last checkpoint was, whereas with 60s you loose only 1 minute, if app checkpoints per preferences that is. But shutdowns happen not that much so it is a good compromise to save on disk writes and/or speed up app processing.

---

GDF ?

I've noticed an increase of time to process work between 6.28 and 6.29. 6.29 is taking another 10,000s/2.75h or so. We still get exact same credit so I'm guessing something has slowed down the app processing, but it appears across all 3 of my hosts so I doubt it is something I have changed as any changes on my end are usually not across all 3. Is this the checkpoint issue ?

Temujin
Send message
Joined: 12 Jul 07
Posts: 100
Credit: 21,848,502
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwat
Message 1590 - Posted: 21 Aug 2008 | 14:10:48 UTC - in response to Message 1589.

I've noticed an increase of time to process work between 6.28 and 6.29. 6.29 is taking another 10,000s/2.75h or so. We still get exact same credit so I'm guessing something has slowed down the app processing, but it appears across all 3 of my hosts so I doubt it is something I have changed as any changes on my end are usually not across all 3. Is this the checkpoint issue ?
I've seen the same increase in processing time on Linux with apps 6.33 & 6.34,
8800GT ~45k -> ~57k seconds
8800GS ~49k -> ~78k seconds

Profile UBT - NaRyan
Avatar
Send message
Joined: 16 Jul 08
Posts: 68
Credit: 1,242,980
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 1591 - Posted: 21 Aug 2008 | 14:26:35 UTC - in response to Message 1590.
Last modified: 21 Aug 2008 | 14:46:02 UTC

Yeah noticing the same slow down here on Linux also.
It's only been going for 6 hours, however normaly 6 hours is 50% complete, at the moment it's at 40% complete.

At this rate it's going to workout taking nearly as long as the Windows version does :(

Also since the new windows app was released, all workunits have started to crash once again for me on windows :(
____________

Down with the Kredit Kops!!!

Profile koschi
Avatar
Send message
Joined: 14 Aug 08
Posts: 124
Credit: 792,979,198
RAC: 11,592
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 1593 - Posted: 21 Aug 2008 | 18:25:00 UTC

Hi, my first work unit with the new App is done, and I would like to have the old one back, please...

Instead of 48.000 seconds I need 75.000 seconds to finish one unit now. This is 27.000 seconds more! Nothing except the app changed on my side, system wasn't updated or rebooted.

How come that there is such a dramatic increase?

http://www.ps3grid.net/results.php?hostid=6998

Was that one maybe a special unit which is supposed to take that long?

Profile UBT - NaRyan
Avatar
Send message
Joined: 16 Jul 08
Posts: 68
Credit: 1,242,980
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 1597 - Posted: 21 Aug 2008 | 23:05:27 UTC - in response to Message 1593.
Last modified: 21 Aug 2008 | 23:15:24 UTC

Hi, my first work unit with the new App is done, and I would like to have the old one back, please...


Would be nice, as an increase from 11 hours 45 minutes, to just over 15 hours (for me), ain't nice from an app that was supposed to have sped things up (with checkpointing)

And I still cannot get a workunit to run for more than a few seconds on my Win XP x64 system, since the new windows app was released.....

All fail with the same error: 3 (0x3)
"<core_client_version>6.3.8</core_client_version>
<![CDATA[
<message>
The system cannot find the path specified. (0x3) - exit code 3 (0x3)
</message>
<stderr_txt>
# Device 0: "GeForce 8800 GTX"; Clock rate: 1350000 kilohertz
MDIO ERROR: cannot open file "restart.coor"
Cuda error: Kernel [kick_drift_kernel] failed in file 'step.cu' in line 48 : the launch timed out and was terminated.
Assertion failed: 0, file step.cu, line 48

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

</stderr_txt>
]]>"
____________

Down with the Kredit Kops!!!

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 1602 - Posted: 22 Aug 2008 | 4:18:44 UTC - in response to Message 1597.
Last modified: 22 Aug 2008 | 5:44:54 UTC

Clearly, the problem was not just checkpointing. Even my machine is slower. I know at least a couple of things that we have changed that could slow it down, but if it is the extra debugging code, you need to stand it for a little while.
New application is coming up.

gdf

[AF>HFR>RR] Laxou
Send message
Joined: 15 Aug 08
Posts: 9
Credit: 1,973,745
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 1604 - Posted: 22 Aug 2008 | 8:58:44 UTC - in response to Message 1597.

Hi, my first work unit with the new App is done, and I would like to have the old one back, please...


Would be nice, as an increase from 11 hours 45 minutes, to just over 15 hours (for me), ain't nice from an app that was supposed to have sped things up (with checkpointing)

And I still cannot get a workunit to run for more than a few seconds on my Win XP x64 system, since the new windows app was released.....

All fail with the same error: 3 (0x3)
"<core_client_version>6.3.8</core_client_version>
<![CDATA[
<message>
The system cannot find the path specified. (0x3) - exit code 3 (0x3)
</message>
<stderr_txt>
# Device 0: "GeForce 8800 GTX"; Clock rate: 1350000 kilohertz
MDIO ERROR: cannot open file "restart.coor"
Cuda error: Kernel [kick_drift_kernel] failed in file 'step.cu' in line 48 : the launch timed out and was terminated.
Assertion failed: 0, file step.cu, line 48

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

</stderr_txt>
]]>"

i get the same error.
i use drivers 177.35 with boinc 6.3.8 on win xp 32b.

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 1611 - Posted: 22 Aug 2008 | 11:23:52 UTC - in response to Message 1604.

Good news, the slowness problem is not in the debugging code but a flag left on while debugging which should not be on.
Application 6.36 for Linux will fix it. Coming up.

gdf

Profile koschi
Avatar
Send message
Joined: 14 Aug 08
Posts: 124
Credit: 792,979,198
RAC: 11,592
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 1612 - Posted: 22 Aug 2008 | 11:31:57 UTC

Very good, I'm really looking forward to that release :-D

Though it seems not all crunchers where effected by the slowness of 6.34, one user from our team reports that he crunched 2 WUs with 6.34 at the same speed as with the older apps...

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 1613 - Posted: 22 Aug 2008 | 11:48:10 UTC - in response to Message 1612.
Last modified: 22 Aug 2008 | 13:43:24 UTC

Very good, I'm really looking forward to that release :-D

Though it seems not all crunchers where effected by the slowness of 6.34, one user from our team reports that he crunched 2 WUs with 6.34 at the same speed as with the older apps...



Application 6.37 out. I expect that once we upload also the new Windows application, Linux and Windows will have the same speed. It may be convenient to abort new wu which you have already if they start with the old application.

gdf

Profile UBT - NaRyan
Avatar
Send message
Joined: 16 Jul 08
Posts: 68
Credit: 1,242,980
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 1635 - Posted: 22 Aug 2008 | 23:58:54 UTC - in response to Message 1613.
Last modified: 23 Aug 2008 | 0:05:10 UTC

Application 6.37 out. I expect that once we upload also the new Windows application, Linux and Windows will have the same speed. It may be convenient to abort new wu which you have already if they start with the old application.

gdf


The computer that has the 8800GTX and is running Linux is now crashing with in a few seconds.

Always with the error

"<core_client_version>6.3.8</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)
</message>
<stderr_txt>
# Using CUDA device 0
# Device 0: "GeForce 8800 GTX"
# Clock rate: 1350000 kilohertz
Cuda error: Kernel [reduce4_kernel] failed in file 'reduction.cu' in line 123 : the launch timed out and was terminated.

</stderr_txt>
]]>"

That's using the 173.14.12 drivers (installed using Envy), I have them installed on the system that has the 2 8800GT's in it and it is working fine (on 1 card for the moment)

I see that driver 177.67 is listed on the Nvidia get cuda page for linux x64, should I try them instead?
____________

Down with the Kredit Kops!!!

Temujin
Send message
Joined: 12 Jul 07
Posts: 100
Credit: 21,848,502
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwat
Message 1639 - Posted: 23 Aug 2008 | 9:10:35 UTC - in response to Message 1613.

Application 6.37 out.

6.37 is still slow for me :(

Profile koschi
Avatar
Send message
Joined: 14 Aug 08
Posts: 124
Credit: 792,979,198
RAC: 11,592
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 1641 - Posted: 23 Aug 2008 | 9:52:52 UTC

http://www.ps3grid.net/show_user.php?userid=5044

6.25 -> 48.000s
6.34 -> 74.500s
6.37 -> 64.500s

Where has the speed gone?

The computer of my team mate:

6.33 -> 46.767s - 48.000s
6.34 -> 49.000s
no Result for 6.37 yet

So I wonder how his 6.25 would have looked like, and also why he is 25.000 seconds faster on 6.34 with the only difference being 16 shaders and 125MHz shader clock more...

Profile UBT - NaRyan
Avatar
Send message
Joined: 16 Jul 08
Posts: 68
Credit: 1,242,980
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 1646 - Posted: 23 Aug 2008 | 11:06:30 UTC - in response to Message 1639.
Last modified: 23 Aug 2008 | 11:09:11 UTC

Application 6.37 out.

6.37 is still slow for me :(


Weird 6.37 is back to normal speed for me.

6.34 took around 54,434.09 Seconds
6.37 took 42,579.39 Seconds
____________

Down with the Kredit Kops!!!

Post to thread

Message boards : Graphics cards (GPUs) : Recent slower application

//