Advanced search

Message boards : Number crunching : Server is out of disk space

Post to thread Subscribe Sort
Author Message
Bedrich Hajek
Send message
Joined: 28 Mar 09
Posts: 484
Credit: 10,061,973,466
RAC: 13,861,392
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 61548 - Posted: 20 Jun 2024 | 21:56:05 UTC

Thu 20 Jun 2024 05:49:06 PM EDT | GPUGRID | [error] Error reported by file upload server: Server is out of disk space

ID: 61548 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
mrchips
Send message
Joined: 9 May 21
Posts: 16
Credit: 1,202,655,500
RAC: 5,536,685
Level
Met
Scientific publications
wat
Message 61550 - Posted: 21 Jun 2024 | 0:23:57 UTC

still out of space
____________

ID: 61550 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Bedrich Hajek
Send message
Joined: 28 Mar 09
Posts: 484
Credit: 10,061,973,466
RAC: 13,861,392
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 61554 - Posted: 21 Jun 2024 | 22:41:23 UTC

It's full again.

Fri 21 Jun 2024 06:40:41 PM EDT | GPUGRID | [error] Error reported by file upload server: Server is out of disk space

ID: 61554 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
pututu
Send message
Joined: 8 Oct 16
Posts: 23
Credit: 2,230,176,869
RAC: 27,915,611
Level
Phe
Scientific publications
watwatwatwat
Message 61719 - Posted: 25 Aug 2024 | 17:00:25 UTC
Last modified: 25 Aug 2024 | 17:25:46 UTC

Currently, I'm seeing disk full error message.

Edit1: seems to be clearing slowly now.

Edit2: uploading seems to be intermittent...

ID: 61719 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Bedrich Hajek
Send message
Joined: 28 Mar 09
Posts: 484
Credit: 10,061,973,466
RAC: 13,861,392
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 61720 - Posted: 25 Aug 2024 | 18:43:54 UTC

Confirmed. Server is out of disk space.

ID: 61720 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
pututu
Send message
Joined: 8 Oct 16
Posts: 23
Credit: 2,230,176,869
RAC: 27,915,611
Level
Phe
Scientific publications
watwatwatwat
Message 61722 - Posted: 25 Aug 2024 | 21:07:26 UTC
Last modified: 25 Aug 2024 | 21:09:12 UTC

Seems like it is intermittent, meaning need to perform network transfer retry occasionally to upload the completed tasks. Best to run with a script.

Fingers crossed that the server disk hasn't completely crashed yet as of this posting...

ID: 61722 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Freewill
Send message
Joined: 18 Mar 10
Posts: 17
Credit: 19,563,195,394
RAC: 124,731,903
Level
Trp
Scientific publications
watwatwatwatwat
Message 61728 - Posted: 26 Aug 2024 | 4:55:44 UTC

Still getting this error as of 26 Aug almost 7:00 AM Madrid time.

ID: 61728 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Richard Haselgrove
Send message
Joined: 11 Jul 09
Posts: 1605
Credit: 7,639,569,351
RAC: 22,136,470
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 61731 - Posted: 26 Aug 2024 | 8:00:37 UTC

26/08/2024 08:58:40 | GPUGRID | [error] Error reported by file upload server: Maintenance underway: file uploads are temporarily disabled.

ID: 61731 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Steve
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 21 Dec 23
Posts: 45
Credit: 0
RAC: 0
Level

Scientific publications
wat
Message 61734 - Posted: 26 Aug 2024 | 9:13:22 UTC - in response to Message 61731.

Thank you for reporting. It is now back and with more space.

ID: 61734 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Bedrich Hajek
Send message
Joined: 28 Mar 09
Posts: 484
Credit: 10,061,973,466
RAC: 13,861,392
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 61779 - Posted: 8 Sep 2024 | 21:07:19 UTC

Server is out of disc space again.


Sun 08 Sep 2024 05:05:46 PM EDT | GPUGRID | Started upload of p38_A31_A28_r0_4-QUICO_ATM_AF_04_Benchmark-16-20-RND3498_1_0
Sun 08 Sep 2024 05:05:47 PM EDT | GPUGRID | [error] Error reported by file upload server: Server is out of disk space
Sun 08 Sep 2024 05:05:47 PM EDT | GPUGRID | Temporarily failed upload of p38_A31_A28_r0_4-QUICO_ATM_AF_04_Benchmark-16-20-RND3498_1_0: transient upload error
Sun 08 Sep 2024 05:05:47 PM EDT | GPUGRID | Backing off 00:24:33 on upload of p38_A31_A28_r0_4-QUICO_ATM_AF_04_Benchmark-16-20-RND3498_1_0



ID: 61779 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Stacie
Send message
Joined: 29 Mar 20
Posts: 22
Credit: 669,047,371
RAC: 3,900,169
Level
Lys
Scientific publications
wat
Message 61780 - Posted: 9 Sep 2024 | 0:27:54 UTC - in response to Message 61779.

Is this why finished workunits are failing to upload? They are starting to pile up in my que.
____________

ID: 61780 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Stacie
Send message
Joined: 29 Mar 20
Posts: 22
Credit: 669,047,371
RAC: 3,900,169
Level
Lys
Scientific publications
wat
Message 61781 - Posted: 9 Sep 2024 | 1:51:46 UTC

...do we lose bonus turnaround credit because the upload server is locked up?
____________

ID: 61781 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Erich56
Send message
Joined: 1 Jan 15
Posts: 1113
Credit: 8,177,220,176
RAC: 19,401,847
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwat
Message 61782 - Posted: 9 Sep 2024 | 2:31:47 UTC - in response to Message 61781.

...do we lose bonus turnaround credit because the upload server is locked up?

yes, at least in the past this has happened. I doubt this has changed now.

ID: 61782 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Erich56
Send message
Joined: 1 Jan 15
Posts: 1113
Credit: 8,177,220,176
RAC: 19,401,847
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwat
Message 61784 - Posted: 9 Sep 2024 | 7:50:29 UTC - in response to Message 61734.

On August 26 Steve wrote:

Thank you for reporting. It is now back and with more space.

Steve, now again no finished tasks can be uploaded since last night :-(

How come that this happens that often? Is there not any kind of automated reporting early enough (say, once the disk is 80% full), or other measures that prevent this problem to happen every few weeks ? This shouldn't be that difficult to implement.

ID: 61784 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
WPrion
Send message
Joined: 30 Apr 13
Posts: 94
Credit: 1,389,434,111
RAC: 11,522,337
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 61785 - Posted: 9 Sep 2024 | 10:40:25 UTC - in response to Message 61784.
Last modified: 9 Sep 2024 | 10:41:04 UTC


How come that this happens that often? Is there not any kind of automated reporting early enough (say, once the disk is 80% full), or other measures that prevent this problem to happen every few weeks ? This shouldn't be that difficult to implement.


Kinda ironic. They are super anxious to get this project completed so they award 10X the points the tasks deserve, therefore attract an army of crunchers, but then don't keep their hardware up to speed to support the effort.

ID: 61785 | Rating: 0 | rate: Rate + / Rate - Report as offensive    Reply Quote
Post to thread

Message boards : Number crunching : Server is out of disk space

//